Performance Budgets in CI Only Work When Base and Head Run Side by Side

A performance budget is a promise written into the pipeline: this change may make the code at most 5% slower, and if it’s worse, the build fails. Tests already guard what the code does. A budget guards how fast it does it.

The idea is old and simple. Making it work on ordinary CI machines is the hard part, because those machines are noisy. On the shared two-core cloud machine where jvq was tested, the very same benchmark binary, run 40 times in a row, spread by 8.9% from one run to the next. A budget of 5% sits well inside that noise, so a naive check can’t tell a real slowdown from the runner having a bad minute.

There are three ways out. Buy quiet hardware, keep a history and compare against it, or measure both versions in the same job. jvq takes the third, and this post is about why.

Two rows of benchmark runs on a runner that slows down during the job. In the first row all base runs come first and all head runs last, so head looks slower. In the second row base and head take turns, and the slow-down hits both.

Quiet Hardware Is the Gold Standard and Most Teams Don’t Have It

Large companies run benchmarks on dedicated machines with frequency scaling off, nothing else running, sometimes with the CPU cores pinned. Noise drops below 1% and a 5% budget becomes easy to enforce. The catch is cost and upkeep. Someone has to own those machines, and pull requests wait in their queue.

Shared CI runners are the opposite: free or cheap, fast to start, and noisy, because your job shares the hardware with other people’s.

Comparing Against History Cries Wolf

The next idea is to store each run’s numbers and compare a pull request with last week’s. The trouble is that last week’s numbers came from a different machine, at a different time of day, with different neighbours. When the gap between two machines is bigger than the budget, the check fails changes that did nothing and passes changes that did harm. People learn that the red light means little, and they stop looking at it.

Measure Both, in the Same Job, Taking Turns

jvq checks out the base commit and the change into two git worktrees on the same runner and runs the benchmarks on both, in alternating rounds: base, head, head, base, base, head. If the machine gets slower halfway through the job, both sides feel it. Nothing is stored between runs, so there’s no history to drift and no server to keep.

Running both sides costs twice the benchmark time. In exchange every comparison is fair, and that’s worth more than speed for a gate that blocks merges.

Ranks Instead of Averages

Benchmark timings aren’t bell-shaped. Most runs cluster, and a few are much slower because of an interrupt, a page fault or a neighbour’s burst. An average gets dragged around by those few. A rank statistic doesn’t: one wild value moves it by one place in the order, however wild it is.

So jvq pairs every base run with every head run, takes the ratio of their times, and reports the median ratio, which is the Hodges-Lehmann estimate. A 95% confidence interval comes from the exact distribution of the Mann-Whitney rank statistic, the same test Go’s benchstat uses, with no assumption about the shape of the timings.

Three Answers Instead of Two

A gate that only says pass or fail has to guess when the data is unclear. jvq has a third answer.

Three confidence intervals against a 5% budget line. One sits entirely below it: pass. One straddles it: inconclusive. One sits entirely above it: regression.

Regression when the whole interval is above the budget. Even the kindest reading of the data says the change is too slow.
Pass when the whole interval is at or below it.
Inconclusive when the interval straddles the budget. jvq then adds rounds, two at a time, up to 30 per side, and decides again.

Inconclusive passes by default with a note in the report, because blocking a merge on a coin toss teaches people to ignore the gate. Teams that would rather wait can set fail_on_inconclusive.

The result is a gate that’s strict about clear slowdowns and honest about close ones. On that noisy machine, 200 runs where nothing changed produced no false alarm, while slowdowns of 20% and 50% were caught every time and a 10% slowdown in 42 runs out of 50.

Allocations Need No Statistics

Time is noisy. The number of heap allocations per operation usually isn’t: the same code allocates the same way every run. jvq compares those, and bytes per operation, on their medians, so a budget of allocs = "0" fails a change that adds even one allocation to a hot path. In testing it caught one extra allocation per op in all 100 runs and never flagged the unchanged control.

Where This Approach Stops

Side-by-side measurement doubles the benchmark time, so it suits benchmarks of seconds rather than minutes. It can’t see a slowdown that only shows up under production load or on other hardware. And on a very noisy runner, a tight budget will often come back inconclusive. That’s the right answer, and the fix is a looser budget, more rounds, or a quieter machine.

The quick start sets it up in a GitHub workflow, and how jvq was tested has the numbers behind the claims above.