First jvq Release Fails Pull Requests That Make Go Benchmarks Slower Than Their Budget

Most slowdowns get in one pull request at a time. An extra allocation in a loop, a string copied where a slice would do, a log call on the hot path. Each costs a few percent, each passes review, and a month later the service is 30% slower and nobody can say which change did it.

jvq 0.1 is a check you put in CI to stop that. You give it a benchmark command and a budget, such as 5% slower at most. On every pull request it runs your benchmarks on the base branch and on the change, on the same runner, taking turns, and fails the build when the change is clearly over budget.

When the runner is too noisy to tell, it says so. That part matters more than it sounds. A gate that cries wolf gets switched off within a week.

What You Get

• One binary, nothing stored. No server, no account, no history. Each run measures both commits fresh, in two git worktrees it removes afterwards.
• Fair comparisons. Base and head run in alternating rounds after a warm-up each, so a runner that drifts during the job hits both sides alike.
• Statistics that suit timings. The change is the median over all pairs of runs, with an exact rank-based confidence interval. Long tails and odd outliers don’t drag it around.
• Three verdicts. Regression when the whole interval is over budget, pass when it’s all within, inconclusive in between. While it’s inconclusive, jvq adds rounds, up to 30 a side.
• Allocation gates. Bytes and allocations per op are compared directly, so allocs = "0" fails a change that adds one allocation to a hot path.
• A GitHub Action. One step in a workflow file. It downloads the binary, checks its checksum, writes the table to the job summary and annotates each regression.

How Well It Works

The test suite runs the real jvq hundreds of times on a benchmark where the right answer is known, on a shared two-core cloud machine whose timings wander 8.9% from run to run. With the default 5% budget:

• 200 runs with no change: no false alarm.
• 50 runs each at 20% and 50% slower: every one caught.
• 50 runs at 10% slower: 42 caught, the other 8 inconclusive.
• 100 runs with one extra allocation per op: all caught.

The test write-up has the full table, including the cases it got wrong.

Try It

jvq run -- go test -run='^$' -bench=. -benchmem ./...

Or add it to pull requests:

- uses: actions/checkout@v7
  with:
    fetch-depth: 0
- uses: just-very-quickly/jvq@v0.1.0

The quick start walks through both. Binaries for Linux, macOS and Windows are on the download page, and the code is on GitHub under the Apache License 2.0.

What’s Next

Version 0.1 reads Go’s benchmark output. Next is any language: timing a whole command directly, and a small JSON format any harness can print. After that, Rust criterion, Google Benchmark, pytest-benchmark and vitest.