Reference

Everything both tools do, in short: jvq, the performance gate, and jvqq, the query tool. Each has a full version on GitHub.

jvq

This is jvq 0.9.0. The full version, with the JSON report’s fields, is docs/reference.md in the jvq repository.

Commands

jvq run [flags] [-- command]   compare base and head; fail on a regression
jvq compare [flags] OLD NEW    compare two saved benchmark outputs
jvq history [flags] FILE       show each benchmark's trend from a history file
jvq flaky [flags] RESULTS...   gate on test results; quarantine flaky tests
jvq check [-config FILE]       read jvq.toml and show the settings
jvq version                    print the version

A command after -- replaces the one in jvq.toml. With no command anywhere and a go.mod in the current folder, jvq runs go test -run=^$ -bench=. -benchmem ./....

Exit Codes

jvq flaky exits 0 when no test failed outside quarantine, 1 when one did, and 2 for an error such as a results file it can’t read.

Which Commits

Head is HEAD unless you pass -head. Base is -base if you pass it. Otherwise it’s where head left the base branch: GITHUB_BASE_REF in a GitHub pull request, or else origin’s default branch, main or master. When head is on the base branch itself, as on a push to main, base is head’s parent.

jvq checks both out with git worktree into a temp folder and removes them when it’s done, after an error or Ctrl-C too. If a run is killed outright, the next run cleans up what it left. A shallow clone that lacks the base commit is an error telling you to fetch the history (fetch-depth: 0 in actions/checkout).

Submodules stay empty in the worktrees unless submodules = true is set in jvq.toml (or -submodules is passed). Then jvq runs git submodule update --init --recursive in both, so each side gets the submodule commits it records. Git and ssh don’t prompt for passwords there, so a submodule jvq can’t fetch is an error naming it, not a stuck run. A repository with submodules and the setting off gets a note.

Run Flags

jvq.toml

Optional. jvq looks for it in the current folder and its parents, up to the repository root. Unknown keys are errors, so a typo can’t fall back to a default without you noticing.

command = "./bench.test -test.run=NONE -test.bench=. -test.benchmem"
build   = "go test -c -o bench.test ./parser"
adapter = "go"                # or json, time, criterion, google-benchmark, pytest-benchmark, vitest
# output = "results.json"     # a file the command writes its results to
# size = ["bin/app"]          # files whose size is gated, after the build
rounds     = 10
max_rounds = 30
warmup     = true
confidence = 0.95
timeout    = "10m"
fail_on_inconclusive = false
submodules = false    # true checks out submodules in both worktrees
# history = "jvq-history.jsonl"  # add a line per run to this file
history_keep = 1000     # how many runs the history file keeps

[budget]
time   = "5%"
bytes  = "10%"
allocs = "0"
size   = "1%"
instructions = "1%"   # the instructions adapter

[budget.bench."BenchmarkParse*"]
time = "2%"

[ignore]
bench = ["BenchmarkNetwork*"]

[flaky]                       # jvq flaky, the test gate
history = "jvq-tests.jsonl"
history_keep = 200            # runs kept
window = 100                  # a test's latest runs searched for flaky commits
quarantine_after = 2          # flaky commits that put a test in quarantine
release_after = 30            # passing runs in a row that take it out
broken_after = 3              # failed attempts in a row: broken, blocks
always = ["web.Checkout.uploads*"]     # in quarantine from the start, unless broken
never = ["example.com/shop.TestPay*"]  # never quarantined

Budgets say how much worse a metric may get. "5%" allows 5% more than base. An amount such as "0" allows that much more in the metric’s own unit, bytes or allocations per op. "off" reports the metric without gating it. Time budgets are always percentages, because an absolute time means something different on every machine. By default time has a 5% budget, file sizes 1%, and bytes and allocations are reported only.

In benchmark patterns * matches anything, slashes included, so BenchmarkParse* covers its sub-benchmarks.

Verdicts

Time. For every pair of a base run and a head run, jvq takes the log of head time over base time. The median of all those pairs is the change it reports, such as +7.2%. The confidence interval comes from the exact distribution of the Mann-Whitney rank statistic, so it assumes nothing about the shape of the timings. Regression means the whole 95% interval is above the budget. Pass means it’s all at or below. Anything in between is inconclusive, and jvq adds rounds, two at a time, while that’s the case.

Bytes and allocations. These hardly move between runs, so jvq compares the medians directly. Head is a regression when its median is over base’s plus the budget.

Adapters

The adapter says how jvq gets numbers out of a run.

go, the default, reads the output of go test -bench. ns/op is time, B/op is bytes and allocs/op is allocations, which need -benchmem or b.ReportAllocs(). Other units, such as MB/s or your own b.ReportMetric, are reported and not gated. pkg: lines keep equal names in different packages apart.

time times the whole command, from start to exit, and calls the result command. Whatever it prints is ignored. Start-up is part of the time, so it suits commands that run for a tenth of a second or more.

instructions (Linux, with valgrind installed) runs the command under valgrind’s cachegrind and counts every instruction it and its child processes run, as one result named command. It’s judged like time, against [budget] instructions, 1% by default. The command must do a fixed amount of work: for Go, go test -c -o x.test as the build and ./x.test -test.run=NONE -test.bench=. -test.benchtime=1000x as the command. It runs 20 to 50 times slower, with no warm-up, so 5 rounds is plenty. jvq sets GOMAXPROCS=1 unless you have, since Go’s spinning idle threads would otherwise be counted. A process that forks without exec counts its parent’s work again, so run the program itself; background work left running isn’t counted.

json reads results the command prints, one per line, or as one {"results": [...]} document, or as an array:

{"name": "parse/small", "unit": "ns", "value": 912.3}
{"name": "parse/small", "unit": "B", "value": 128}
{"name": "parse/large", "group": "parser", "unit": "ms", "values": [1.9, 2.1, 2.0]}

name and unit are required, with value or values; group keeps equal names apart. ns, us, ms and s (or spelled out) are time, B is bytes, allocs is allocations, instructions is an instruction count (judged like the instructions adapter’s), and /op may follow any of them. Other units are reported and not gated. Lines that don’t start with { are skipped, and so are JSON objects with none of a result’s fields, such as structured log lines.

Harnesses, read as they are. Each gives one time per benchmark per run:

The harness’s own comparison with its last run is ignored; jvq measures base and head itself.

With output = "FILE", jvq reads results from a file the command writes, instead of its standard output. The file is deleted before each run, so an old one can’t count.

When a run gives several values for one benchmark (Go’s -count, several JSON lines), jvq takes their median as that run’s value. Each run counts as one observation.

File Sizes

size = ["target/release/app", "dist/*.js"] lists files whose size jvq gates. It measures them once per side after the build (before the first run if there’s no build), in the folder the command runs in, and calls a file a regression when it grows by more than its budget: 1% by default, or set size under [budget], as a percentage or an amount in bytes. 1% of a small file is only a few bytes, so give small files an amount. * matches within one folder name. A file on one side only is reported as new or removed. A pattern that matches nothing is an error, and so is a folder: name the files.

jvq compare

jvq compare old.txt new.txt judges two saved outputs, in any format but time, with the same statistics, budgets and exit codes as jvq run. Files ending in .json are read as json; -adapter names the format otherwise. Each needs several values per benchmark. The two weren’t measured in alternating rounds, so a machine that changed between them shows up as a change. jvq run is the gate for CI.

History

jvq run -history FILE (or history in jvq.toml) adds one JSON line per run to FILE: each result’s change, interval and verdict. It keeps the newest history_keep lines, and a re-run with the same base and head replaces its line. On the default branch, a run starts from the last line’s head when that is an ancestor, so a push of several commits measures all of them. A relative path is relative to where jvq runs.

jvq history FILE shows, for each benchmark and metric, how many runs measured it, how far it has moved in all with a 95% interval, how many runs called it a regression, and its change run by run. In all multiplies the runs’ changes, since each was measured on one machine; the interval adds up the runs’ own spreads, so noisy runs give a wide one. Flags: -n N for the newest N runs, -bench and -metric to narrow it, -markdown FILE, and -fail-over 10% to exit 1 when a gated result’s interval is over the limit even at its low end.

In the action, history: jvq-history.jsonl keeps the file in the Actions cache, one per workflow, job and runner OS (history-key to choose). history-write decides which runs add to it: auto (pushes to the default branch), true or false. history-fail-over applies on the runs that write. GitHub drops a cache unread for 7 days, so run at least weekly; a concurrency group keeps two pushes from saving over each other. In GitLab, JVQ_HISTORY: "true" keeps it in the job’s cache, saved by default-branch pipelines and only read by merge requests, with JVQ_HISTORY_FAIL_OVER as the limit.

Flaky Tests

jvq flaky [flags] RESULTS... reads one CI run’s test results, JUnit XML or go test -json, from files or folders (shards and mixed formats add up), judges them against a test history and adds the run to it. It exits 1 only when a test failed outside quarantine.

• A flaky commit is a commit on which a test both passed and failed, through a retry in the run (go test -count=2, gotestsum --rerun-fails, Surefire reruns) or a re-run of the job. Runs on different commits never count together.
• A test goes into quarantine after 2 flaky commits in its last 100 runs. Its failures are listed and annotated, and don’t block.
• It leaves after 30 passing runs in a row, and its next failure blocks.
• A test whose last 3 attempts all failed is broken: it leaves quarantine and blocks. A test that always fails never goes in.
• A Go package that doesn’t build is its own entry, package (build), which only ever fails, so a compile error always blocks.
• Tests named in always under [flaky] are in quarantine from their first run, with no evidence needed, and passing runs don’t take them out; they still break and block after 3 failed attempts in a row. Tests named in never are never quarantined, so every failure blocks. Names take *. A Go package’s own entries can’t be pinned, and a pattern that matches no test gets a note.
• When a Go package fails, the tests that ran in its last passing run and are missing now are listed as not run: a crash (a panic, a timeout) or -failfast stopped the package before them. They don’t block; a warning names them. A test running beside one that panicked was interrupted, not failed, and a test still paused for t.Parallel never ran, so neither is blamed. Under a timeout, only the tests Go names as running are.

Flags: -history FILE (or history under [flaky]; - for none), -commit SHA (default git HEAD, then GITHUB_SHA, CI_COMMIT_SHA), -no-write to judge without adding the run, as on a pull request, -json FILE, -markdown FILE and -q. With no results it shows what is in quarantine. The history is one JSON line per run, each test’s attempts as letters (p, f, s), and quarantine is worked out again from it every time. Since 0.9 the test names are written once, in a list at the top and bottom of the file, and each run refers to them by number: 2000 tests over 200 runs take 319 KB instead of 26.8 MB. Older files are read and converted on the next write. A damaged line is skipped with a note; a missing file starts a new history. jvq reads at most 5 million test results, newest runs first, so a crafted file can’t fill the runner’s memory.

With two retries for a failed test, a quarantined test that really broke blocks in the run that broke it. Without retries it is listed as a quarantined failure until its third failure in a row, which blocks. A pull request run with -no-write counts alone each time, so there only retries can catch it.

GitLab CI

include:
  - remote: https://raw.githubusercontent.com/just-very-quickly/jvq/v0.9.0/ci/gitlab/jvq.yml

benchmarks:
  extends: .jvq
  image: golang:1.26
  variables:
    JVQ_COMMAND: go test -run=NONE -bench=. -benchmem ./...

The .jvq job runs in merge request pipelines and on the default branch, with a full clone. It downloads the release and checks its SHA256SUMS, runs jvq run, and keeps a JUnit report for the merge request’s test summary, plus jvq.md and jvq.json. Variables: JVQ_COMMAND, JVQ_ARGS, JVQ_DIR, JVQ_VERSION. The image needs curl, git, tar and sha256sum; on Alpine, apk add --no-cache curl git.

The base comes from GitLab’s variables: in a merged results pipeline or merge train, the target branch the change was merged into or rebased onto; otherwise the merge request’s diff base; on the default branch, the commit before.

JUnit Report

-junit FILE writes each gated result as a test case: a regression fails it, inconclusive is skipped with the reason, pass and improved pass. Results that aren’t gated aren’t listed.

The GitHub Action

uses: just-very-quickly/jvq@v0.9.0 downloads the release binary for the runner (Linux, macOS or Windows, x86-64 or ARM64), checks it against the release’s SHA256SUMS and runs jvq run. Inputs: command, base, config, args (more flags), working-directory and version, and history, history-write, history-fail-over and history-key for a history. With flaky (the results files or folders) it runs the flaky test gate instead, and keeps the test history in the Actions cache: flaky-history (default jvq-tests.jsonl), flaky-write (auto: pushes, manual and scheduled runs on the default branch add to it; pull requests are judged without writing), flaky-key (default one per workflow, job, matrix leg and runner OS) and flaky-args. It needs no token. The report goes to the job summary, and each regression becomes an error annotation.

jvqq

This is jvqq 0.1. The full version, with the input rules, the error messages and the jq builtins, is docs/reference.md in the query repository.

Command Line

jvqq [options] <filter> [file...]
jvqq [options] -f <file> [file...]

The filter is a jq program. Files are read in the order given, as if joined. With no file, or with -, jvqq reads standard input. gzip and zstd input is recognized by its first bytes, whatever the name.

Options

jq’s options work as in jq: -c, -r, -j, -n, -s, -R, -e, -S, --tab, --indent n, --arg, --argjson, --rawfile, --slurpfile, --args, --jsonargs, -f, --unbuffered, -M, -h and -V. --stream, --seq, --raw-output0, -a, -C and -L are refused with exit code 2.

jvqq adds these:

Input and Output

Each line holds one JSON value. Blank lines are skipped and \r\n endings are fine. When the first megabyte holds a value that spans lines, such as pretty-printed JSON, jvqq reads value by value instead, still on every core. A filter that calls input or inputs runs in order on one core, as in jq.

Output is in input order whatever the number of workers, written the way jq 1.7.1 writes it: number literals keep their digits, strings use jq’s escapes, objects keep their key order.

A line that isn’t valid JSON is skipped. At the end jvqq says how many it skipped and where the first one was:

jvqq: skipped 2 lines that aren't valid JSON (first at app.jsonl:812: Unfinished JSON term at column 41)

Exit Codes

Aggregations

Aggregations go in one step of the filter’s top-level pipe, A | B | C. A runs on every input, on all cores. B holds the aggregations and runs once at the end, with null as its input, so it can combine them: {n: count, mean: (sum(.ms) / count)}. C runs on what B gives. sum and avg add without rounding and round once, so the result doesn’t depend on how the input was split. group_by(f) with one argument is still jq’s array function.

Not Supported

Function definitions (def), label and break, destructuring (. as [$a, $b] and ?//), modules (import, include), input_line_number and $__prog_args. jvqq refuses them with exit code 3 rather than doing something different. Regular expressions use Go’s RE2, so there are no backreferences or lookaround.