jvq Replays 26 of Its Own CI Runs and Blocks a Windows Test Bug Four Pushes in a Row
The fastest way to find out what a flaky test gate does with real failures is to give it some. jvq has its own CI history on GitHub: 26 pushes to main between October 18 and October 31, 2026, each tested on Linux, macOS and Windows, with a second run under the race detector on Linux. Eight of those 26 runs went red.
So we fed that history to jvq flaky 0.7.0, run by run, in the order it happened, with each run’s real commit. One history per platform and test step, as the reference asks. A test that fails on Windows and passes on Linux has a Windows problem, and mixing the two would hide it.
Nothing went into quarantine. Seven runs were blocked. Here is why that’s the right answer for this history, and what it would take to get a different one.
Module paths and file names shortened. The full output of every run is with the scripts.
Where the Data Comes From
GitHub doesn’t keep test reports for these runs. It keeps each job’s steps and their results, and jvq’s test workflow turns every --- FAIL line into an annotation. That’s enough to rebuild what happened to every test that ever failed: failed when an annotation names it, passed when its step passed or failed on other tests. Five tests failed somewhere in the 26 runs. The rest of the suite passed every time and is left out, since it can’t change the outcome.
Four steps are left out too, because the annotations don’t say which test failed: the first push’s macOS and Windows runs, from before the workflow made annotations, and two race steps on Linux: one that didn’t run because the plain tests had failed, and one that failed without naming a test. That leaves 100 replayed steps: 26 plain test runs on Linux, 24 race runs, and 25 each on macOS and Windows.
Windows Had a Real Bug, Four Pushes Running
The flaky gate’s own history tests check that jvq keeps a history file’s permission bits when it rewrites the file. Windows has no Unix file modes, so on the Windows runner TestCorruptedHistory and TestHistoryFile saw -rw-rw-rw- and failed. They failed on the push that added them and on the next three. The fix, skipping the mode checks on Windows, came 25 minutes after the first failure.
jvq blocked all four. In each run the tests failed with nothing to set against the failure: no retry that passed, no earlier run of the same commit. Its LAST RUNS column reads f, then ff, then fff, then ffff. A test that fails every time never builds a flaky commit, so it never comes near quarantine. That’s the case the gate was built not to get wrong, and on real data it didn’t.
The same thing happened once before, in October 19’s runs. TestReRunReplacesItsLine checked file modes on Windows too, failed on the push that added it, blocked, and was fixed four and a half minutes later.
A One-Off on Linux and Another on macOS
On Linux, TestInstructionsSteadyWithGoroutines failed on the 0.6.0 release commit. The test runs a Go program under valgrind and expects an unchanged program to count as unchanged. GitHub’s runner measured an interval of -0.52% to +0.75%, wider than the test allowed. Three minutes later a commit widened the tolerance to fit what the runner measured, and it passed from then on. jvq blocked that one run.
The macOS case is the interesting one. TestCommandAfterDashes passed on 20 pushes, failed on one, and passed again on the next. It runs three short benchmark rounds and checked the verdict, and on a busy runner three short rounds of an unchanged benchmark sometimes came out as a regression. That’s a flaky test by any reasonable reading. The 0.7.0 commit later changed it to accept any verdict.
jvq blocked it anyway, and from this history it had no way to do anything else. The workflow doesn’t retry failed tests, and nobody re-ran the job, so the test failed once on a commit where nothing else ran. A failure on new code may be the code’s fault. jvq only calls a test flaky when it has passed and failed on the same commit, and here there was never a second try.
What Would Change the Outcome
Retries. Suppose the workflow retried failed tests twice, as in the use examples post. If the macOS test had passed on a retry, jvq would have listed it as failed and passed on retry and let the run through. Had that happened on a second commit, the test would have gone into quarantine, still listed in every report.
The Windows tests can’t pass on a retry. Windows has no Unix file modes on the second try either, so they would have failed all three attempts and blocked exactly as they did.
Seven blocked runs out of 100 looks like a strict gate. It’s what a gate should do when CI gives it one try per commit. Six of the seven were fixed by a code change within half an hour.
The Same Suite on a Busy Machine
The flakes in jvq’s suite showed up on loaded machines, so we also ran it the way its CI does on Linux, plain and then with the race detector, at each of the 8 commits from the 0.6.0 recording to 0.7.0. The machine was a shared 2-core cloud VM with other jobs building on it. A failed test would have been retried twice.
None was needed. Each of the 16 steps reported 1,258 test results, every one a pass, so jvq printed PASS 16 times with nothing to list. That’s 16 runs, which is too few to say the suite has stopped flaking. It does show the gate staying out of the way when nothing goes wrong.
Run It Yourself
The API data, the scripts that rebuild the JUnit files and replay them, and every report jvq printed are in test/results/case-studies/flaky/jvq-github-ci, and the local runs are in jvq-suite next to it. python3 fetch.py saves the runs again, python3 rebuild.py writes one JUnit file per run, and sh replay.sh feeds them through jvq. The reports quoted here are from 0.7.0. Replayed with 0.7.1, every one came out the same. The reference has the rules, and the release post has the simulation behind them.