jvq runs your benchmarks on the base branch and on the change, taking turns on the same CI runner, and fails the pull request that is clearly over its budget. jvqq runs jq filters over big JSON Lines files on every core and prints what jq prints, byte for byte.
0 of 200
Runs where nothing changed that jvq called a regression
100 of 100
Runs with a 20% or 50% slowdown that jvq caught
19x to 31x
jvqq's speed over jq on seven filters, using 52 MiB of memory at most
518 filters
Run through jq and jvqq five ways each, with the same output every time
jvq: default settings and a 5% budget on a shared two-core cloud machine whose timings spread 8.9% from run to run. jvqq: 1 GiB of generated logs on a four-core GitHub-hosted runner, against jq 1.7.1. The test logs and results are on GitHub.
releases
The flaky test gate in 0.7 worked, but setting it up in GitHub Actions meant copying fifteen lines of YAML: restore a cache, pick the right flag for pull requests, run …
releases
The flaky test gate keeps one line per CI run, and through 0.8 that line spelled out every test’s name. Fine for a few hundred tests. For a suite of 2000 Go tests …
guides
jvq flaky needs two things from your CI: test results it can read, and a history file that survives from one run to the next. Everything else is optional. These are the …
releases
Every team has a test like this. It checks out a cart, or uploads a file, or waits on a socket, and about one run in ten it fails for no reason anyone can find. The pull …
case studies
The fastest way to find out what a flaky test gate does with real failures is to give it some. jvq has its own CI history on GitHub: 26 pushes to main between October 18 …
case studies
Logs written by real services aren’t always valid JSON Lines. A process killed mid-write leaves half a line. The restarted process appends its first event straight …
testing
The first round of tests ran jvq hundreds of times on a Go benchmark whose cost we set by hand. That showed how often it raises a false alarm and how often it catches a …
case studies
The alert says checkout is slow. You have the last six hours of application logs as one JSON Lines file, 1 GiB and 3,073,492 lines, and a handful of questions to answer …
releases
A budget gate has a blind spot, and it’s built in. jvq judges each change against its base. A change that makes a benchmark 1% slower passes a 5% budget, and it …
If jvq nearly fits your pipeline, or jvqq prints something jq doesn't, write in.