The jvqq Query Tool

jvqq runs jq filters over big JSON Lines files on every CPU core. You write the filter you’d write for jq, and you get what jq would print, byte for byte, in the same order. On 1 GiB of logs it took 0.7 to 1.8 seconds per filter, where jq took 17 to 35.

jvqq -c 'select(.level == "error") | {ts, service, msg}' app.jsonl

Independent of jq. jvqq is an independent tool, not a fork of jq and not affiliated with the jq project. It reads the jq language and is tested against jq 1.7.1’s output.

Why It Exists

jq is the tool everybody knows for JSON, and it runs on one core. On a log of a few gigabytes, a plain select keeps you waiting for a minute, and people end up writing a throwaway Python script instead. jvqq keeps the filter language and makes the big file fast.

It’s not the only fast jq. qj, built on simdjson, is a little faster on filters that work line by line, and jaq is a careful reimplementation. jvqq’s angle is exact jq output from one portable binary, counting and grouping that run in parallel, and calm handling of messy input: a line that isn’t valid JSON is skipped and counted instead of ending the run.

Counting and Grouping

jq answers “how many errors per service” with reduce inputs, which reads in order on one core, or with -s, which loads the whole file into memory. jvqq adds aggregations that each core works out on its share of the file, merged at the end:

jvqq 'count_by(.service)' app.jsonl
jvqq 'select(.status >= 500) | group_by(.service; {n: count, ms: avg(.latency_ms)})' app.jsonl

The aggregations are count, count_by(f), sum(f), avg(f), min(f), max(f) and group_by(f; agg). Sums are exact, so the answer doesn’t depend on how the file was split.

How Fast

1 GiB of generated application logs, 3,073,492 lines, on a 4-core GitHub-hosted runner. Median of 3 runs after a warm-up run.

For count, count by and sum by, jq and qj run reduce inputs and jvqq runs its aggregations.

Every output was checked against jq’s: the same bytes for the line-by-line filters, the same values for the aggregations. jvqq used at most 52 MiB. jaq, also in the run, used about 1 GiB on every filter. On a 6 GiB file of 18,440,091 lines, jvqq counted every line in 4.4 seconds at 68 MiB.

The full tables, with gojq and jaq, the exact commands and the versions, are in the recorded results on GitHub.

Compared With jq

For the filters jvqq supports, standard output is jq 1.7.1’s: number formatting, string escapes, key order, the text of most error messages, and the exit codes. The test suite runs 518 filters through jq and through jvqq, each five ways, and compares them.

The differences, all documented:

• Bad lines. jq stops at the first line that isn’t valid JSON. jvqq skips it and reports how many there were at the end. --strict stops instead.
• Not supported. Function definitions (def), label and break, destructuring, modules, input_line_number and $__prog_args. jvqq refuses these with a message and exit code 3 rather than doing something different.
• Regular expressions use Go’s RE2 engine, so there are no backreferences or lookaround.
• Math functions use Go’s math library and can differ from jq in the last digit.
• Messages about invalid JSON and compile errors are worded differently.

Case Studies

Worked examples on generated logs, run with jvqq and jq 1.7.1 side by side:

• Tracing a 12-minute payments outage through 3 million log lines, one question at a time.
• Three broken lines in a 1 GiB log, where jq stopped at the first one.
• A week of gzipped logs, read faster than zcat can unzip them.

All of them are under case studies.

License

The code is open source under the Apache License 2.0, on GitHub. It’s one static binary for Linux, macOS and Windows, with gzip and zstd input read as it is. The download page has the files, and the quick start the first commands.