jvq 0.5 Keeps a History to Catch the Slowdown That Arrives One Small Merge at a Time

A budget gate has a blind spot, and it’s built in. jvq judges each change against its base. A change that makes a benchmark 1% slower passes a 5% budget, and it should. So does the next one. Ten merges later the code is about 10% slower, and every check said pass.

Version 0.5 keeps a history so that kind of creep can show. Each run on the default branch adds a line to a small file. jvq history reads it back: how far each benchmark has moved in all, with an interval, how many runs called it a regression, and a row of blocks with its change run by run.

Here’s a real one. Twelve commits, each making one benchmark 2% slower, run on our 2-core test machine, which is noisy (7 to 9% run to run in recent recordings):

A chart of 12 real runs: the true slowdown climbs to about +27%, jvq’s chained estimate for the slowed benchmark ends at +12.5% with an interval from -6.2% to +31.7%, and an unchanged benchmark ends at -8.6% with an interval from -23.1% to +7.1%.

BENCHMARK       METRIC  RUNS  IN ALL                   REGRESSIONS  RUN BY RUN
BenchmarkParse  time    12    +12.5% (-6.2% … +31.7%)  0            ▁▃▁▅▁▂▄▂▃█▁▂
BenchmarkWrite  time    12    -8.6% (-23.1% … +7.1%)   0            ▁▁▃▁▁▁▁▁▂▃▁▄

Every single run passed or said inconclusive, which is right for a 2% step under a 5% budget. The history points the right way and its interval covers the real +27%. On a machine this noisy it doesn’t prove it, though, and the unchanged benchmark wandered to -8.6%. That’s why the interval is there.

Changes Chain, Medians Don’t

The obvious way to track a trend is to plot the time run after run. In CI that fails: today’s runner may be a different machine from yesterday’s. What is comparable is each run’s own change, base against head, measured on one machine in alternating rounds. jvq multiplies those. Ten runs at +1% each make about +10.5%, whatever machines they ran on.

Each run’s interval comes along too. The spreads add up, so twelve noisy runs give a wide interval and twelve tight ones a narrow one. My first version chained only the point estimates. In a test of 12 merges at +1% each it ended at -4%, the wrong way entirely, with nothing to say how little that number meant. A limit on numbers like that fails builds on noise.

In CI

In GitHub Actions it’s one more input:

- uses: just-very-quickly/jvq@v0.5.0
  with:
    history: jvq-history.jsonl
    history-fail-over: 10%

The action keeps the file in the Actions cache, one per workflow and job. Pushes to the default branch add to it, pull requests only read it, and every run puts the trend in the job summary. history-fail-over fails a push when the low end of something’s interval is over the limit, so a creep measured clearly trips it and noise doesn’t. It only applies where the history grows, so a pull request isn’t failed for a slowdown it didn’t cause.

A push of several commits is measured from where the last run ended, so nothing in between goes unmeasured, and a re-run of the same job replaces its line. GitLab gets the same with JVQ_HISTORY: "true" in the template. Anywhere else, jvq run -history FILE and jvq history FILE do it by hand.

The file is plain JSON, one line per run, and keeps the newest 1000. One catch: GitHub drops a cache that nobody reads for a week, so the history needs a run at least weekly. The reference has the rest.