The jvq Performance Gate

jvq is a performance budget gate for CI. You give it a benchmark command and a budget, such as 5% slower at most. On every pull request it measures the base branch and the change on the same machine, taking turns, and fails the build when the change is clearly over budget. When the noise is too high to tell, it says so instead of guessing.

Why It Exists

Speed regressions rarely arrive as one big mistake. They come one small pull request at a time: an extra allocation in a loop, a map where a slice would do, a log call on the hot path. Each one is a few percent. Nobody runs benchmarks by hand on every change, and the ones who try in CI find that shared runners wander by more than the budget from one run to the next.

The usual tools solve this with dedicated machines or a server that stores results over time. Both work, and both cost money and upkeep. jvq is for teams that have neither: one binary, nothing stored, both versions measured fresh in the same job.

How It’s Built

• Two worktrees. jvq checks out the base commit and the change with git worktree, builds each once if you ask, and removes them afterwards, even after Ctrl-C. A run killed outright is cleaned up by the next one.
• Taking turns. Base and head run in alternating rounds (base, head, head, base…), after a warm-up run each, so a machine that drifts during the job hits both sides alike.
• Rank statistics. The change is the Hodges-Lehmann estimate of the time ratio, with an exact rank-based confidence interval. No assumption that timings follow a bell curve.
• Three verdicts. Regression, pass, or inconclusive, with extra rounds added while it’s inconclusive.
• Exact gates. Bytes and allocations per operation are compared directly, so one extra allocation can fail a build.
• A flaky test gate. jvq flaky reads JUnit XML or go test -json results across CI runs and quarantines tests that pass and fail on the same commit, so they stop blocking merges. Every quarantined failure is still listed, a test that is stable again leaves on its own, and a test that keeps failing always blocks.
• Tested against known answers. The test suite runs jvq hundreds of times on benchmarks where the right answer is known and publishes how often it was right. The results are on GitHub with the code.

Case Studies

Worked examples, each with real commits run through jvq:

• A Python cleanup that made URL parsing nearly five times slower, and the three tries it took to fix it within budget.
• A Go logging change that added two allocations to every request, caught by an allocation budget of 0.
• An update check that made a command-line tool nearly four times bigger, caught by a size budget.
• jvq’s own CI history replayed through the flaky test gate: 26 pushes on three platforms, a Windows bug blocked four times and never quarantined.

All of them are under case studies. For setting up the flaky test gate, the use examples cover pytest, Go and GitHub Actions.

License

All the code is open source under the Apache License 2.0, on GitHub. It uses only the Go standard library.

Questions People Ask

Is it really free?

Yes. jvq is free to use, and the code is free to take under the Apache License 2.0. The GitHub Action needs no account or token.

Which languages does it support?

Any language. jvq reads Go’s go test -bench, Rust’s criterion, Google Benchmark, pytest-benchmark and vitest as they are, times any other command from start to exit, or reads a few lines of JSON that a benchmark in any language can print. It can also gate the size of the files your build makes.

Why does it run my benchmarks twice?

Because comparing against numbers from another machine or another day is where false alarms come from. Measuring both versions in the same job doubles the benchmark time and makes every comparison fair.

What does inconclusive mean?

That the confidence interval for the slowdown contains the budget, so the data can’t say which side of it the change is on. jvq adds rounds, up to 30 per side, before it gives that answer. By default an inconclusive run passes with a note. fail_on_inconclusive = true makes it fail.

Does it need a quiet machine?

No, but it does better on one. On a noisy runner it calls fewer close cases and more inconclusives. The test results show how it behaves on a noisy cloud machine.

Can I suggest something?

Please do. The contact page has the address.

Is jvq related to jvqq?

They come from the same place and share the name’s promise, and that’s all. jvqq is a separate tool for querying JSON logs with jq filters, with its own page: the jvqq query tool. Neither needs the other.