testing

How jvq Was Tested With Hundreds of Runs Where the Right Answer Was Known

A gate that blocks merges has two ways to fail. It can cry wolf, flagging changes that did nothing, until people stop trusting it. Or it can sleep through a real slowdown. The only way to know how often jvq does either is to run it many times on changes where you already know the answer, and count.

So that’s what its test suite does. A fixture benchmark runs a CPU loop whose length is set by a constant. One commit uses the base length. The next adds a set amount of work, 2% more, 10% more, 50% more, or changes only a comment. Then the real jvq binary runs on that repository, start to finish, with its default settings: two git worktrees, a build for each side, a warm-up, 10 rounds taking turns, more rounds while undecided, and the 5% budget.

The machine was a shared two-core cloud VM. Before the runs, the same benchmark binary ran 40 times in a row on it, and its timings spread by 8.9% from run to run. That’s noisier than a typical CI runner, and more than the budget jvq had to enforce.

Stacked bars of jvq’s verdicts for each injected slowdown: no change mostly pass and never regression, +2% and +5% mostly inconclusive, +10% mostly regression, +20% and +50% always regression.

The Results

Head Runs Regression Pass Inconclusive Median measured change
no change 200 0 151 49 -0.2%
2% more work 50 0 24 26 +2.2%
5% more work 50 2 2 46 +5.0%
10% more work 50 40 0 10 +10.6%
20% more work 50 50 0 0 +19.9%
50% more work 50 50 0 0 +49.8%

Default settings, 5% time budget, 95% intervals. A regression exits 1; pass and inconclusive exit 0.

No false alarm in 200 runs where nothing changed. Every slowdown of 20% or more caught, almost all within the first 10 rounds. At 10%, twice the budget, 40 runs out of 50 caught and the other 10 undecided. That number moves with the machine’s noise: other recordings on the same kind of VM caught 43 and 42 at 8.9% noise, 38 at 7.1%, 34 at 7.2% and 48 at 4.6%. Never once did a real slowdown come back as a pass.

The measured change lands where it should, too. Across each set of runs, the median estimate was within about half a point of the work that was added.

Where It Got It Wrong

Two runs flagged a change that was within the budget: 5% more work, exactly on the line. A change sitting right at the budget can tip either way, and jvq looks again each time it adds rounds, which gives a borderline result a few more chances to cross. Other recordings flagged between 1 and 4 of 50 there. Nothing at 2% was flagged, and nothing that didn’t change. A few in 50 at the very edge is the price of a gate that decides at all.

The other side of the coin is inconclusive. At exactly 5%, jvq said it couldn’t tell in 44 runs out of 50, and passed with a note. At 2% it passed in 17 and couldn’t tell in 33. That’s the design: with timings wandering several percent from run to run, the data couldn’t put those changes firmly on one side of the line, and jvq said so instead of guessing.

On a Quiet Machine

The same suite ran again on a GitHub-hosted runner, the kind most pipelines use. Its timings spread by only 0.5% from run to run, and every clear case was settled in the first 10 rounds. All 200 runs with no change passed, and so did all 50 at 2% more work. Every run at 10%, 20% and 50% more was caught. At exactly 5%, on the line, 2 runs out of 50 were flagged and 45 came back inconclusive. With so little noise, a change sitting right on the budget is the only thing left to argue about.

Allocations

The same fixture can add a heap allocation to each operation. With allocs = "0" as the budget, jvq caught the extra allocation in 100 runs out of 100, and in 100 runs of an unchanged control it never flagged one. Allocation counts don’t wander like times do, so there’s no statistics involved, only the medians of each side.

Everything Else the Tests Check

The accuracy runs measure judgement. Most of the suite checks behaviour, again with the real binary on throwaway git repositories:

  • a 50% slowdown fails with exit code 1 and a correct JSON report, and runs in base, head, head, base order
  • a renamed benchmark shows as new and removed and doesn’t fail the run
  • a base that doesn’t compile is an error, unless -base-may-fail says to pass
  • a shallow clone gets a message saying to fetch the history
  • the GitHub job summary and error annotations are written
  • an interrupt mid-run removes both worktrees at once, and a run killed outright is cleaned up by the next one
  • the statistics match published tables of Mann-Whitney critical values and keep their stated coverage in simulation

Every release binary is also run on its own kind of machine before it goes out: it must catch a change three times slower and pass a change to a comment.

All of it, the raw runs included, is in test/results on GitHub, recorded by one script that anyone can run again.