jvq Quarantines Flaky Tests So They Stop Blocking Merges
Every team has a test like this. It checks out a cart, or uploads a file, or waits on a socket, and about one run in ten it fails for no reason anyone can find. The pull request goes red. Somebody clicks “re-run” and it goes green. After a few weeks everyone clicks re-run without reading the failure, and one day the failure is real.
Version 0.7.0 adds a second gate to jvq for this. jvq flaky reads your test results, keeps a small history across CI runs, and tells two kinds of failure apart: a test that fails on code that also passed it, and a test that fails because the code broke.
How It Decides
The only thing jvq counts as flaky is a test that passed and failed on the same commit. That happens when a retry inside the run passes (go test -count=2, gotestsum’s --rerun-fails, Maven Surefire’s reruns), or when someone re-runs the job and it goes green. A test that passed yesterday and fails today on new code doesn’t count. The code changed, so the failure may well be right.
When a test has done that on two commits within its last 100 runs, it goes into quarantine. From then on its failures are still listed in the report and annotated in GitHub, but they don’t fail the build. Two rules keep this honest:
- A test leaves quarantine after 30 passing runs in a row. Its next failure blocks again.
- A test whose last three attempts all failed is broken, not flaky. It leaves quarantine at once and blocks. A test that fails every time never goes in.
A Short Example
go test -json ./... > go-test.json
jvq flaky -history jvq-tests.jsonl go-test.json junit/
It reads go test -json and JUnit XML, which nearly every test runner can write, and any number of files count as one run. Here is a real run, with one test broken by the change, one flaky test in quarantine, and one that failed and passed on retry:
TEST THIS RUN STATUS FLAKY COMMITS LAST RUNS
example.com/app.TestParseDate f failed, blocks 0 ppppppppppf
web.Checkout.pays by card f failed in quarantine, doesn't block 2 fpppppxpppf
example.com/app.TestUpload fp failed, passed on retry 1 ppppppppppx
FAIL: 1 test(s) failed (example.com/app.TestParseDate). 1 quarantined test(s) failed and didn't block. 1 test(s) failed and passed on retry. 1 test(s) in quarantine.
The history is one JSON line per run with each test’s attempts as letters. Nothing else is saved. jvq works out the quarantine again from the whole file on every run, so it can’t drift away from what actually happened. Keep the file in your CI cache. The quick start and the reference have the details, including a GitHub Actions setup.
How It Was Tested
The unit and end-to-end tests read real go test -json and pytest output and walk a flaky test through its whole life in the real binary. Then a simulation ran the binary through CI histories where the answer was known. Each run wrote real result files, half the tests as Go JSON and half as JUnit.
A suite of 200 stable tests and 5 flaky ones, failing 2 to 30% of the time, ran for 300 commits, ten times over, with no retries. When a run went red it was re-run once, the way people do. Without quarantine, 1,611 of the 3,000 first runs were red. With jvq flaky, 437 were, and in the last 100 commits of each trial it was 96 of 1,000. Most of what’s left comes early, before a flaky test has shown itself twice, and from the rarest one, which often never did in 300 commits. No stable test was ever quarantined.
The part that matters most is the other direction. In 50 trials a quarantined test broke for real. Without retries it blocked within three runs every time. With two retries it blocked in the run that broke it, all 50 times. That sharpness has a price: in 6 of 529 runs before the break, the flaky test happened to fail all three attempts and was taken for broken. A fixed test left quarantine after its 30 passing runs in all 10 trials. A history with junk in it and a line cut short, and later a deleted one, gave no errors. Each time jvq said so in a note and carried on.
All the numbers are in test/results/flaky.md, next to the code that produced them.
Real histories came next. jvq’s own CI, replayed through the gate, shows what it does with 26 pushes on three platforms and a Windows bug that failed four pushes in a row.
What to Know
Without retries, a quarantined test that really broke slips through for up to two runs, listed as a quarantined failure each time. Two retries for failed tests close that gap. On pull requests, run with -no-write so a branch’s runs don’t change the shared history. That also means a pull request’s own re-runs don’t count as evidence.
The history repeats every test’s name on every line, so a big suite makes a big file: 2,000 long Go test names over 200 runs came to 27 MB. Lower history_keep if that matters.
The use examples have tested setups for pytest, Go and GitHub Actions, with the two retries that close the gap.
Download it from the download page, or go install github.com/just-very-quickly/jvq/cmd/jvq@latest.