releases

jvq Now Reads criterion, Google Benchmark, pytest-benchmark and vitest as They Are, and Gates File Sizes

Version 0.2 let jvq gate a benchmark in any language, by timing the whole command or reading a few lines of JSON. That still meant a little glue for projects that already have a benchmark harness. Version 0.3 removes the glue for four of the most used ones. Point jvq at the harness and it reads the harness’s own output.

It also adds a second kind of budget. Besides time, bytes and allocations, jvq can now fail a pull request that makes a file it builds bigger than it should be.

Four harnesses feeding jvq: cargo bench with criterion’s time lines, Google Benchmark’s JSON, pytest-benchmark’s JSON file and vitest bench’s JSON file, each going to jvq, which runs base and head taking turns.

Four New Adapters

  • criterion, for Rust. cargo bench as it is. jvq reads the time: [low estimate high] line for each benchmark and takes the estimate. It skips criterion’s own comparison with its last run, since jvq measures base and head itself.
  • google-benchmark, for C++ and Python. Run with --benchmark_format=json. Each repetition counts; a skipped benchmark is left out.
  • pytest-benchmark, for Python. pytest --benchmark-json=bench.json with output = "bench.json". Each test’s median is its time for the run.
  • vitest, for JavaScript and TypeScript, versions 1 to 4. vitest bench --run --outputJson bench.json, also with output.

A Rust crate needs three lines in jvq.toml:

adapter = "criterion"
build   = "cargo bench --no-run"
command = "cargo bench"

The tests for these adapters run on output captured from the real tools. That’s how the review before release caught a case a hand-written sample would have missed: a criterion benchmark with a throughput prints its change since the last run as a block with a time: line of its own, in percent. Read naively, that would have failed every run after the first.

A Budget on File Sizes

size = ["target/release/app", "dist/*.js"]

[budget]
size = "1%"

jvq measures each file once per side, after the build, and fails the run when one grows by more than its budget. A percentage or an amount in bytes both work; 1% is the default, and an amount suits small files better.

Tested End to End

The scenario recordings now run each harness through jvq run on a shared two-core cloud VM, 10 runs with no change and 10 with 20% more work in the benchmark’s loop. criterion and vitest caught all 10 slowdowns and raised no false alarm. Google Benchmark caught 8 and left 2 inconclusive. pytest-benchmark caught all 10, but it also raised one false alarm in its 10 no-change runs: with a short time per test, its medians wandered by more than the budget on that machine. The results file has every row, and the reference has the details of each adapter.

The GitHub Action is just-very-quickly/jvq@v0.3.0, and binaries for Linux, macOS and Windows are on the download page.