How to Set Up jvq Flaky With GitHub Actions and Retries in pytest and Go

jvq flaky needs two things from your CI: test results it can read, and a history file that survives from one run to the next. Everything else is optional. These are the setups we’d start with, each one run for real before it went on this page.

They were tested with the released 0.7.0 binary on a small demo project: a pytest suite of four tests, one of which fails about one try in three, and a Go package with the same kind of test. That test fails at random, the way a test does that waits on a slow service. The outputs below are copied from those runs, and the script that made them is in test/results/case-studies/flaky/recipes.

From the “why two retries” example below. The legend lines are left out.

Retry Failed pytest Tests Twice

pytest writes JUnit XML with --junitxml, and --lf runs only the tests that failed last time. Chain three runs, one file each:

mkdir -p junit
pytest --junitxml=junit/attempt1.xml ||
  pytest --lf --junitxml=junit/attempt2.xml ||
  pytest --lf --junitxml=junit/attempt3.xml || true

jvq flaky -history jvq-tests.jsonl junit/

A retry only runs when the run before it failed, so a green run makes one file. jvq reads the whole folder as one CI run, and a test that’s in two files has two attempts. The final || true hands the verdict to jvq.

In the demo, the flaky test went into quarantine on the 12th CI run, each on a new commit. It had failed and passed on retry once before, on the 6th:

TEST                                     THIS RUN  STATUS                                    FLAKY COMMITS  LAST RUNS
tests.test_cart.test_checkout_times_out  fp        failed, passed on retry; into quarantine  2              pppppxpppfpx

PASS: no test failed outside quarantine. 1 test(s) failed and passed on retry. 1 test(s) in quarantine.

Run 10 shows the other side. The test failed all three attempts, and with only one flaky commit behind it, it wasn’t in quarantine yet. That run blocked.

Skip pytest-rerunfailures for This

The pytest-rerunfailures plugin retries inside one run with --reruns 2, which looks like the easier route. We tried it with a test that fails once and then passes. Version 16.7 wrote that test into the JUnit file twice, and neither entry had a failure in it. jvq read two passes and had nothing to judge. Use the --lf chain above, which keeps each failed attempt where jvq can see it.

Go Tests With gotestsum Reruns

gotestsum reruns failed tests and writes every attempt into one go test -json file:

gotestsum --rerun-fails=2 --packages=./... --jsonfile=go-test.json -- -count=1
jvq flaky -history jvq-tests.jsonl go-test.json

We couldn’t install gotestsum on the test machine, so this one was tested with the stream it writes: go test -json for the first run, then go test -json -run for the failed test, added to the same file. jvq counted both attempts:

TEST                                   THIS RUN  STATUS                   FLAKY COMMITS  LAST RUNS
example.com/shop.TestCheckoutTimesOut  fp        failed, passed on retry  1              x

If your CI runs the tests twice, once plain and once with -race, give each step its own history: jvq-tests.jsonl for one and jvq-tests-race.jsonl for the other. In one history, a test that fails only under the race detector would have passed and failed on the same commit, and jvq would call it flaky. A data race isn’t something to quarantine.

Keep the History in GitHub Actions

The history is one JSON line per run. Keep it in the Actions cache, let pull requests read it without writing, and save it from pushes to main:

      - name: Tests, with up to 2 retries of what failed
        run: |
          mkdir -p junit
          pytest --junitxml=junit/attempt1.xml ||
            pytest --lf --junitxml=junit/attempt2.xml ||
            pytest --lf --junitxml=junit/attempt3.xml || true
      - name: Install jvq
        run: |
          curl -sSLO https://github.com/just-very-quickly/jvq/releases/download/v0.7.0/jvq_linux_amd64.tar.gz
          curl -sSLO https://github.com/just-very-quickly/jvq/releases/download/v0.7.0/SHA256SUMS
          grep linux_amd64 SHA256SUMS | sha256sum -c
          tar xzf jvq_linux_amd64.tar.gz
      - uses: actions/cache/restore@v5
        with:
          path: jvq-tests.jsonl
          key: jvq-tests-${{ github.run_id }}-${{ github.run_attempt }}
          restore-keys: jvq-tests-
      - name: Flaky test gate
        run: ./jvq flaky -history jvq-tests.jsonl ${{ github.ref != 'refs/heads/main' && '-no-write' || '' }} junit/
      - if: always() && github.ref == 'refs/heads/main'
        uses: actions/cache/save@v5
        with:
          path: jvq-tests.jsonl
          key: jvq-tests-${{ github.run_id }}-${{ github.run_attempt }}

A cache entry can’t be overwritten, so every save gets a new key, and restore-keys finds the newest one. The save runs even when the gate fails, so a red run on main still counts. The whole workflow is in the folder above.

We didn’t run this on GitHub itself. The jvq step was checked outside it, with the variables GitHub sets, and the YAML was parsed to catch typos. On a pull request ref, with -no-write, the history stayed at 12 lines. On main it grew to 13. Both runs wrote the report to the job summary file.

See What Is in Quarantine

Run it with the history and no results:

$ jvq flaky -history jvq-tests.jsonl

TEST                                     THIS RUN  STATUS                                FLAKY COMMITS  LAST RUNS
tests.test_cart.test_checkout_times_out  -         quarantined, 0 of 30 passes to leave  2              pppppxpppfpx

1 test(s) in quarantine.

For a script or a dashboard, write the JSON report and pick out the quarantined tests:

jvq flaky -history jvq-tests.jsonl -no-write -json report.json junit/
jq -r '.result.tests_of_note[] | select(.quarantined) | "\(.test)  since \(.quarantined_since)  flaky commits \(.flaky_commits)"' report.json

tests.test_cart.test_checkout_times_out  since 2026-10-31T10:54:59Z  flaky commits 2

Release a Fixed Test

Usually there’s nothing to do. Fix the test, and after 30 passing runs in a row it leaves on its own. In the demo, the run after the fix said passed; in quarantine, 1 of 30 passes to leave, and the 30th said passed; left quarantine (30 passing runs in a row). From then on, a failure blocks again.

If you’d rather not wait, take the test out of the history. The next run builds the quarantine again from what’s left:

jq -c --arg id 'tests.test_cart.test_checkout_times_out' \
  'if .tests then del(.tests[$id]) else . end' before.jsonl > jvq-tests.jsonl
jvq flaky -history jvq-tests.jsonl

0 test(s) in quarantine.

This forgets the test’s whole record. If it’s still flaky, it needs two new flaky commits to go back in, and until then its failures block. In GitHub Actions, run the jq step once on main, between restoring the cache and the gate, so the edited file is the one saved.

Why Two Retries

A test in quarantine can still break for real, and jvq takes it out as broken when its last three attempts all failed. With two retries, three attempts happen in the run that broke it. To see it, the demo’s quarantined test was made to fail every time, and one run was judged against the same history three ways:

Without retries, the break gets through two runs and blocks on the third failing run in a row. Each of the two green runs lists it as a quarantined failure and makes a warning annotation, but the run is green.

The price is the case from run 10 above. Now and then a flaky test fails all three attempts by chance, and jvq takes it for broken and blocks once. In the 0.7.0 simulation, that happened in 6 of 529 runs. The release post has the rest of those numbers, and the reference has every flag and setting.