Testing on JVQ.net

jvq Flaky Now Takes Three Lines of YAML and Shows the Tests a Crash Skipped

Wed, 04 Nov 2026 00:00:00 +0000

The flaky test gate in 0.7 worked, but setting it up in GitHub Actions meant copying fifteen lines of YAML: restore a cache, pick the right flag for pull requests, run the gate, save the cache, but only on the main branch. Most people who’d want it would read that block and decide to do it later.

Version 0.8.0 puts all of that in the action:

      - run: go test -json ./... > go-test.json || true
      - uses: just-very-quickly/jvq@v0.8.0
        with:
          flaky: go-test.json

The action installs jvq and keeps the test history in the Actions cache. Pushes to your default branch add to it, and so do manual and scheduled runs there. Pull requests are judged against it without writing, so a branch full of half-finished work can’t teach the gate that a broken test is flaky. Each job of a matrix keeps its own history, because a test that fails on one Go version and passes on another isn’t flaky, it’s broken on one of them.

jvq Shrinks Its Flaky Test History 80 Times and Lets You Pin Tests In or Out of Quarantine

Wed, 04 Nov 2026 00:00:00 +0000

The flaky test gate keeps one line per CI run, and through 0.8 that line spelled out every test’s name. Fine for a few hundred tests. For a suite of 2000 Go tests with long package paths, 200 runs came to 26.8 MB, and that file goes in and out of the Actions cache on every job.

Version 0.9.0 writes each name once. The file starts and ends with the list of test names, and a run’s line says which numbers passed, failed or were skipped:

How to Set Up jvq Flaky With GitHub Actions and Retries in pytest and Go

Sat, 31 Oct 2026 00:00:00 +0000

jvq flaky needs two things from your CI: test results it can read, and a history file that survives from one run to the next. Everything else is optional. These are the setups we’d start with, each one run for real before it went on this page.

They were tested with the released 0.7.0 binary on a small demo project: a pytest suite of four tests, one of which fails about one try in three, and a Go package with the same kind of test. That test fails at random, the way a test does that waits on a slow service. The outputs below are copied from those runs, and the script that made them is in test/results/case-studies/flaky/recipes.

jvq Quarantines Flaky Tests So They Stop Blocking Merges

Sat, 31 Oct 2026 00:00:00 +0000

Every team has a test like this. It checks out a cart, or uploads a file, or waits on a socket, and about one run in ten it fails for no reason anyone can find. The pull request goes red. Somebody clicks “re-run” and it goes green. After a few weeks everyone clicks re-run without reading the failure, and one day the failure is real.

Version 0.7.0 adds a second gate to jvq for this. jvq flaky reads your test results, keeps a small history across CI runs, and tells two kinds of failure apart: a test that fails on code that also passed it, and a test that fails because the code broke.

jvq Replays 26 of Its Own CI Runs and Blocks a Windows Test Bug Four Pushes in a Row

Sat, 31 Oct 2026 00:00:00 +0000

The fastest way to find out what a flaky test gate does with real failures is to give it some. jvq has its own CI history on GitHub: 26 pushes to main between October 18 and October 31, 2026, each tested on Linux, macOS and Windows, with a second run under the race detector on Linux. Eight of those 26 runs went red.

So we fed that history to jvq flaky 0.7.0, run by run, in the order it happened, with each run’s real commit. One history per platform and test step, as the reference asks. A test that fails on Windows and passes on Linux has a Windows problem, and mixing the two would hide it.

How jvq Holds Up Beside a Noisy Neighbour, on a Drifting Machine and With Python Scripts

Mon, 19 Oct 2026 00:00:00 +0000

The first round of tests ran jvq hundreds of times on a Go benchmark whose cost we set by hand. That showed how often it raises a false alarm and how often it catches a real slowdown on an ordinary machine. CI machines aren’t always ordinary, though. Version 0.2 adds three harder settings, each with a known right answer, recorded on a shared two-core cloud VM and on a four-core GitHub-hosted runner.

How jvq Was Tested With Hundreds of Runs Where the Right Answer Was Known

Sun, 18 Oct 2026 00:00:00 +0000

A gate that blocks merges has two ways to fail. It can cry wolf, flagging changes that did nothing, until people stop trusting it. Or it can sleep through a real slowdown. The only way to know how often jvq does either is to run it many times on changes where you already know the answer, and count.

So that’s what its test suite does. A fixture benchmark runs a CPU loop whose length is set by a constant. One commit uses the base length. The next adds a set amount of work, 2% more, 10% more, 50% more, or changes only a comment. Then the real jvq binary runs on that repository, start to finish, with its default settings: two git worktrees, a build for each side, a warm-up, 10 rounds taking turns, more rounds while undecided, and the 5% budget.