jvqq Reads a Week of Gzipped Logs Faster Than zcat Can Unzip Them

Most log files older than a day are compressed. Logrotate gzips them, object storage keeps them as .gz, and the habit for querying them is a pipe: zcat app-*.jsonl.gz | jq .... It works, and it’s slow twice over. gzip unpacks on one core, and jq parses on one core, each waiting on the other.

jvqq reads gzip and zstd files directly, recognized by their first bytes, and parses on every core. This worked example is a week of rotated application logs: seven daily files from jvqq’s benchmark generator, 150 MiB each uncompressed and about 23 MB each compressed, 161 MB in all and 3,151,367 lines. The commands below ran with jvqq 0.1.0 and jq 1.7.1 on a two-core cloud VM, with the files already in the page cache.

jvqq counts users by country across seven gzip files in 4.3 seconds. zcat piped into jq gives the same counts in 55.7 seconds. zcat piped into wc -l, which only unzips and counts lines, takes 6.3 seconds.

Timings are wall-clock on two cores.

Users by Country, Over the Week

jvqq -c 'count_by(.user.country)' app-2026-10-*.jsonl.gz

{"null":787606,"JP":296087,"IN":295780,"DE":295735,"GB":295694,"BR":295577,"US":295340,"IL":294887,"FR":294661}

4.3 seconds. null is the anonymous requests, which have no user. count_by puts the most frequent first.

The usual pipe, with jq’s reduce doing the counting:

zcat app-2026-10-*.jsonl.gz | jq -cn 'reduce inputs as $x ({}; .[$x.user.country | tostring] += 1)'

Same counts, in the order jq first saw the keys, in 55.7 seconds.

Faster Than the Unzip Alone

To see where the time goes, we timed the pipe with nothing but a line count at the end:

zcat app-2026-10-*.jsonl.gz | wc -l

3,151,367 lines in 6.3 seconds. So jvqq unzipped, parsed and counted the whole week faster than zcat could unzip it. jvqq unpacks with the gzip decoder from klauspost/compress, and its workers parse each chunk while the reader unpacks the next. We didn’t time its unpacking on its own, but the whole job finished two seconds before zcat did. On the jq pipe, nearly all of the 55.7 seconds is jq.

The files are read one after another, as if they were one file, and the output keeps that order. A file that can’t be opened is reported, and the others are still read.

Errors per Day

The same files, with one row per day and a count of the timeouts among the errors:

jvqq -c 'select(.level == "error") | group_by(.ts[:10]; {n: count, timeouts: sum(if .error.type == "TimeoutError" then 1 else 0 end)}) | .[]' app-2026-10-*.jsonl.gz

{"key":"2026-10-13","n":56136,"timeouts":13921}
{"key":"2026-10-14","n":56242,"timeouts":14123}
...
{"key":"2026-10-19","n":56140,"timeouts":14229}

4.1 seconds at 41 MiB. The generated week is steady, so the rows look alike. On real logs this is the query that shows the bad day.

zstd

zstd files work the same way, recognized by their first bytes whatever the name. A .jsonl.zst, a .jsonl.gz and a plain file can go in the same command, and jvqq reads each one the right way.

The download page has the binaries and the quick start the first commands.

jvqq is an independent tool, not a fork of jq and not affiliated with the jq project. It reads the jq language and is tested against jq 1.7.1’s output.