releases

jvq Now Gates Benchmarks in Any Language, by Timing a Command or Reading a Few Lines of JSON

The first jvq release read one thing: the output of Go’s go test -bench. That covered Go projects and nothing else. Version 0.2 drops the limit. If your benchmark is a Python script, a Node program, a shell loop or a compiled binary in any language, jvq can now gate it.

There are two new ways in. The time adapter times the whole command, from start to exit, so there’s nothing to change in your code. The json adapter reads results your benchmark prints, one per line, with a name, a unit and a value. Pick one with a line in jvq.toml or -adapter on the command line.

Three ways into jvq: the go adapter reads go test output, the time adapter times a command such as python3 bench.py, and the json adapter reads lines such as name parse, unit ns, value 912.

The Two New Adapters

  • time. jvq run -adapter time -- python3 bench.py. Each run gives one time. Start-up is part of what’s measured, so it suits commands that run for a tenth of a second or more. The clock stops when the command exits, even if something it started in the background keeps running.
  • json. Print {"name": "parse", "unit": "ns", "value": 912} lines, in any language that can print. Units ns, us, ms and s are time, B is bytes, allocs is allocations. Other lines are skipped, so the benchmark can log as it likes, and structured JSON log lines are skipped too. A whole {"results": [...]} document works, and so does a file the benchmark writes, with output = "results.json".

When a run prints several values for one benchmark, jvq now takes their median as that run’s value. Values from one run share that run’s machine state, so counting each as a separate observation would make jvq more sure of itself than it should be. Each run counts once.

Two Saved Outputs

jvq compare old.txt new.txt judges two saved outputs with the same statistics, budgets and exit codes as jvq run. It takes Go output or the JSON format. The two files weren’t measured in alternating rounds, so a machine that changed between them shows up as a change; the report says so. It’s for a quick check on a quiet machine. jvq run stays the gate for CI.

Tested Where It Gets Hard

This release adds recorded tests for three harder settings: a noisy neighbour taking half the CPUs, a machine that slows down during the job, and Python scripts through both new adapters. On a GitHub runner, every Python run at 20% more work was caught by both adapters, and none of the 120 no-change runs raised an alarm. The write-up on those tests has all the numbers, including the ones that came out inconclusive on a noisy two-core VM.

Get It

The GitHub Action is just-very-quickly/jvq@v0.2.0. Binaries for Linux, macOS and Windows are on the download page, the reference has the JSON format in full, and the code is on GitHub under the Apache License 2.0.

Next are adapters that read popular harnesses as they are: Rust’s criterion, Google Benchmark, pytest-benchmark and vitest. A gate on binary size comes with them.