How jvq Holds Up Beside a Noisy Neighbour, on a Drifting Machine and With Python Scripts

The first round of tests ran jvq hundreds of times on a Go benchmark whose cost we set by hand. That showed how often it raises a false alarm and how often it catches a real slowdown on an ordinary machine. CI machines aren’t always ordinary, though. Version 0.2 adds three harder settings, each with a known right answer, recorded on a shared two-core cloud VM and on a four-core GitHub-hosted runner.

These numbers come from the jvq 0.2.0 recording. Later recordings, with the same tests and more, are in the repository, and the counts move a little from one to the next with the machine’s noise that day.

Every run is a full jvq run with the default 5% budget, except the drift test, which drives jvq’s runner directly so it can also try the order jvq never uses.

A Machine That Slows Down

A runner can get slower during a job: it warms up, a neighbour starts, the clock drops. Whatever runs second then looks slower. We made every run 2% slower than the one before, with base and head identical, and ran it two ways.

Run all of base and then all of head, and that drift faked a regression in 37 of 50 runs on the VM and 50 of 50 on the GitHub runner. Run them taking turns, base, head, head, base, as jvq does, and there were no false alarms on either. The drift still costs something: the timings spread wider, and almost all of the turn-taking runs came back inconclusive. That’s the right answer for a machine drifting that hard. jvq can’t tell, and it says so.

A Noisy Neighbour

Next, a second process kept half the CPUs busy and walked through 512 MB of memory over and over, while jvq compared a Go benchmark with itself and then with 10% more work.

• No change, 100 runs: no false alarm on either machine.
• 10% more work, 50 runs: all 50 caught on the GitHub runner. On the VM, 43 caught and 7 inconclusive.

Python Through the New Adapters

The last setting is a Python script with a work loop, gated two ways. The time adapter times the whole command, Python’s start-up included. The json adapter has the script time five batches of its own work and print them.

At 10% more work the two machines part ways. The GitHub runner caught all 30 runs with both adapters. The VM caught 11 with the time adapter and 12 with the json adapter, and left the rest inconclusive, after 30 rounds a side. On that machine a Python run’s time wanders by more than the effect, and 30 rounds weren’t enough to settle it.

What It Adds Up To

Across all three settings and both machines, jvq raised no false alarm in its normal mode. When it missed a slowdown, it said inconclusive rather than pass, and the noisier the machine, the more often that happened. The full tables are in the repository, in test/results/scenarios.md for the VM and test/results/github/scenarios.md for the GitHub runner, with every run in the JSON files beside them.