jvqq Runs jq Filters on Every Core and Is Tested Against jq Byte for Byte

Application logs tend to end up in the same shape: a file of a few gigabytes with one JSON event per line, and a question about it. How many 500s per service since the deploy? Which requests hit the timeout? jq answers that kind of question well. On a big file it takes its time, because it reads on one core.

jvqq 0.1 is a command-line tool that takes the same filters and runs them on every core. The output comes out in input order and is what jq would print, down to the byte. On a four-core GitHub-hosted runner, seven filters over 1 GiB of logs took 0.7 to 1.8 seconds each with jvqq and 17 to 35 seconds with jq.

1 GiB of generated application logs, 3,073,492 lines. Median of 3 runs after a warm-up run. Every output was checked against jq’s.

What You Type

The command line is jq’s: a filter, then files, or standard input.

jvqq -c 'select(.level == "error") | {ts, service, msg}' app.jsonl
jvqq -r '.msg' app.jsonl.gz
kubectl logs deploy/api | jvqq -c 'select(.status >= 500)'

The options are jq’s too, from -c and -r to --arg and --slurpfile. jvqq adds two. --workers n sets how many cores to use, and --strict stops at the first line that isn’t valid JSON. Without it, such a line is skipped and counted, and a note at the end says how many there were and where the first one was. jq stops at the first one, which on a real log is often hours into the file.

gzip and zstd files are read as they are, recognized by their first bytes.

Counting Without Slurping

Counting is where jq hurts most on big files. The choices are reduce inputs, which reads in order on one core, and -s, which loads the whole file into memory first. jvqq adds aggregations that each core works out on its share of the file, merged at the end:

jvqq 'count_by(.service)' app.jsonl
jvqq 'select(.status >= 500) | group_by(.service; {n: count, ms: avg(.latency_ms)})' app.jsonl

Lines per service took jq 29.7 seconds with reduce inputs and jvqq 1.0 second with count_by. Sums are exact: jvqq adds without rounding and rounds once at the end, so the answer is the same however the file was split between cores.

Checked Against jq

“Prints what jq prints” is easy to claim and tedious to keep, so the test suite checks it on every change. It runs 518 filters through jq 1.7.1 and through jvqq, each five ways: from standard input and from a file, with the input split between workers in different places. Output and exit codes have to match. The benchmark does the same with its own output, on 3 million lines per filter.

Where jvqq differs, the README says so. Lines that aren’t valid JSON are skipped rather than fatal. Regular expressions run on Go’s RE2 engine, which has no backreferences or lookaround. A few math functions can differ in the last digit. Six features of the language aren’t supported yet, among them def and label, and jvqq refuses them with exit code 3 instead of doing something different.

Where Others Are Faster

jvqq isn’t the only fast jq. qj, built on simdjson, was a little faster in the same run on the four filters that work line by line: 0.8 seconds against 1.0 for select, for example. On counting and grouping, where jvqq’s aggregations run in parallel and qj runs reduce inputs, qj was 7 to 10 times slower. jaq ran at 0.9 to 2 times jq’s speed and used about 1 GiB of memory on every filter.

jvqq used at most 52 MiB on the 1 GiB file, because it holds only a few chunks per core at a time. On a 6 GiB file of 18,440,091 lines it counted every line in 4.4 seconds at 68 MiB.

Get It

jvqq is one static binary for Linux, macOS and Windows, on x86-64 and ARM64, from the download page. With Go:

go install github.com/just-very-quickly/query/cmd/jvqq@latest

The code is open source under the Apache License 2.0 on GitHub, with the recorded benchmark and the test results. The quick start has the first commands.

jvqq is an independent tool, not a fork of jq and not affiliated with the jq project. It reads the jq language and is tested against jq 1.7.1’s output.