jvq 0.6 Counts Instructions Under Valgrind to Catch a 2 Percent Slowdown on a Noisy Runner

Time is a noisy thing to gate on. On our 2-core test machine the same benchmark spreads 7 to 9% from run to run, so jvq’s default budget is 5%, and a 2% slowdown walks straight through. It should, given the noise. But 2% is real, and a dozen of them add up.

Version 0.6 has a second way to measure. The instructions adapter runs the benchmark under valgrind’s cachegrind and counts every instruction it executes. A count doesn’t care how busy the machine is. Run the same program twice and you get nearly the same number, so the budget can drop to 1%.

Here’s one real commit from the history post, a 2% step in BenchmarkParse, measured both ways on the same noisy machine:

Two intervals for the same 2 percent slowdown: timed, the change reads +0.1% with an interval from -4.5% to +4.9% and passes a 5% budget; counted, it reads +1.9% with an interval from +1.8% to +1.9% and fails a 1% budget.

adapter = "instructions"
build   = "go test -c -o x.test"
command = "./x.test -test.run=NONE -test.bench=Parse -test.benchtime=2000x"
rounds  = 5

That config caught it in 5 rounds and 9 seconds. In the recorded scenarios, 20 runs at +2% were all caught, 20 runs at +0.5% all passed, and a no-change run never failed. Next to a neighbour process eating half the CPUs, where the time gate caught only half of the runs at +10%, the count still read +1.9% and caught all 20.

Why Valgrind

The obvious tool is perf stat and the CPU’s own counters. CI runners don’t expose them. GitHub’s hosted machines don’t, and neither did our VM. Cachegrind simulates the CPU instead, so it works anywhere Linux runs. The price is speed: the command runs 20 to 50 times slower. So keep it short and give it a fixed amount of work. A benchmark that runs for “one second” does more work on a faster machine, and the count would see that as a change.

The Catch the Review Found

The first version looked perfect on our test program. The independent review wrote a different one: eight goroutines, allocating. Two runs of it counted 96.6 and then 78.9 billion instructions. Valgrind runs one thread at a time, and Go’s idle threads spin while they wait, so the spinning got counted. A lighter program of the same kind, now in jvq’s tests, ranged from 0.66 to 23.7 billion. With GOMAXPROCS=1 it counted 215.0 to 215.1 million, every time. jvq now sets it for you unless you’ve set it yourself, and that program is in the scenarios.

Two more things to know. A process that forks without exec carries its parent’s count into the copy, so run the program itself rather than through a subshell. And fewer instructions usually means faster, but not always: a cache miss costs more than an add, and a count can’t see it. It’s a sharper gate. It doesn’t replace the time gate.

The reference has the details. It’s Linux only and needs apt install valgrind.