jvq Flaky Now Takes Three Lines of YAML and Shows the Tests a Crash Skipped

The flaky test gate in 0.7 worked, but setting it up in GitHub Actions meant copying fifteen lines of YAML: restore a cache, pick the right flag for pull requests, run the gate, save the cache, but only on the main branch. Most people who’d want it would read that block and decide to do it later.

Version 0.8.0 puts all of that in the action:

      - run: go test -json ./... > go-test.json || true
      - uses: just-very-quickly/jvq@v0.8.0
        with:
          flaky: go-test.json

The action installs jvq and keeps the test history in the Actions cache. Pushes to your default branch add to it, and so do manual and scheduled runs there. Pull requests are judged against it without writing, so a branch full of half-finished work can’t teach the gate that a broken test is flaky. Each job of a matrix keeps its own history, because a test that fails on one Go version and passes on another isn’t flaky, it’s broken on one of them.

The || true matters: the tests’ own step must not stop the job, or the gate never sees their results. The gate’s step then fails the job when a failure isn’t quarantined.

The Hole It Closes

There was one gap in 0.7 that the notes called out plainly. When a quarantined Go test crashed, with a panic or a timeout, Go stopped its whole package. The tests after it simply didn’t run. The gate saw a quarantined failure, didn’t block, and said nothing about the rest. A pull request that broke one of those tests could get through, because that test never got the chance to fail.

Now jvq notices. When a package fails, it looks at the package’s last passing run and lists every test that ran then and is missing now. This is the small demo module in jvq’s repository, after a passing run, two flaky runs and a planted crash:

TEST                             THIS RUN  STATUS                                                                                          FLAKY COMMITS  LAST RUNS
example.com/flakydemo.TestFlaky  f         failed in quarantine, doesn't block                                                             2              pxxf
example.com/flakydemo.TestB      -         not run: missing now that its package failed (a crash, a timeout or -failfast stops a package)  0              ppp

PASS: no test failed outside quarantine. 1 quarantined test(s) failed and didn't block. 1 test(s) didn't run in a package that failed. 1 test(s) in quarantine.

Those tests don’t block. The crash is in quarantine, and blocking on everything it stopped would undo the quarantine. But the report lists them, and a warning on the run names them, so you can see what went unchecked.

Blaming the Right Test

Getting this right meant reading Go’s test output more carefully. When one parallel test panics, the others running beside it never finish. In 0.7 they were counted as failures, so a quarantined crash could still fail the build through its siblings. Now a test interrupted by another test’s panic gets no result at all, and so does a parallel test still waiting for its turn. When a package hits its timeout, jvq blames only the tests Go names as running at the time. And when gotestsum’s --rerun-fails reruns a crashed test and it passes, the tests the first crash skipped are still listed.

All of these came out of two independent reviews of the new code, ten defects between them, each fixed with a test on real go test -json output before the release. The action itself is tested on Linux, macOS and Windows runners with a small module that has a planted flaky test and a planted crash. A second job checks that the history really does come back from the cache.

Also in 0.8

The README now warns that pytest-benchmark 5.3 doesn’t work with pytest 9. jvq’s own accuracy workflow ran into it and was red for twelve days. Until pytest-benchmark catches up, pin pytest<9.

Download 0.8.0 or use the action at @v0.8.0. The reference has the new rules and inputs.