jvq Shrinks Its Flaky Test History 80 Times and Lets You Pin Tests In or Out of Quarantine

The flaky test gate keeps one line per CI run, and through 0.8 that line spelled out every test’s name. Fine for a few hundred tests. For a suite of 2000 Go tests with long package paths, 200 runs came to 26.8 MB, and that file goes in and out of the Actions cache on every job.

Version 0.9.0 writes each name once. The file starts and ends with the list of test names, and a run’s line says which numbers passed, failed or were skipped:

{"kind":"tests","jvq":"0.9.0","commit":"4e1f0c2a…","names_id":"9c1d2e3f4a5b6c7d",
 "results":{"p":"0-41,43-1999","fp":"42"}}

In a healthy suite almost everything passes, so most runs come down to a range or two. The same 2000 tests over 200 runs now take 319 KB. Nothing changes in how quarantine is worked out: the whole history is still replayed every time. A 0.8 file is read as it is and converted the next time a run is added.

The name list is written twice, at the top and the bottom, for a reason. A cache cut short loses the end of the file, and with one copy at the end that would have cost every run. And an older jvq, which keeps the lines it can’t read and trims the oldest first, still has the copy at the bottom until no line needs it.

Pinned Tests

Quarantine normally waits for evidence: a test has to pass and fail on the same commit twice. Sometimes the team already knows. A test that talks to a sandbox payment provider fails one run in thirty, everyone knows it, and nobody wants to wait for two more red builds. Others you never want to let through on a coin flip.

[flaky]
always = ["web.Checkout.uploads*"]
never  = ["example.com/shop.TestPay*"]

A test in always is in quarantine from its first run, and passing runs don’t take it out. It still breaks the way any quarantined test does: three failed attempts in a row and it blocks, so a test that really stopped working can’t hide behind the pin. A test in never is never quarantined, however flaky it gets. A pattern that matches no test gets a note, since that’s usually a typo.

Submodules

jvq run checks out base and head into git worktrees, and git leaves submodules empty there. Repositories whose build needs one couldn’t use the performance gate. submodules = true in jvq.toml now checks them out in both worktrees, each at the commit its side records. Git and ssh are told not to ask for passwords, so a submodule that can’t be fetched is an error naming it rather than a job that sits until it times out.

What the Reviews Found

Two independent reviews of the new code found eight problems, all fixed with tests before the release. The worst: an always pattern broad enough to match a Go package’s own entries could quarantine a compile error. Pins now apply to tests only. Another was a crafted history of half a megabyte that expanded to 2 GB in memory, since a compact file is small on disk and big once read. jvq now reads at most 5 million test results, newest runs first, and leaves older runs out with a note. And a file cut short at the end used to lose every run, which is why the name list now comes twice.

Download 0.9.0 or use the action at @v0.9.0. The reference has the new settings. The 0.8 post covers the action’s flaky input.