flakemap — inspection report

Test reliability across CI history, computed from JUnit XML.

runs 222 tests 12 run range run-0000 → run-0221 window 2026-06-01 → 2026-06-04
1broken
3flaky
8healthy
0insufficient data
0test-runs passed on retry
0test-runs excluded

Test × run heatmap

Tests under tests.test_suite

test_regression_after_commit DEFECT 32%
Final failure rate (95% CI)31.5% [25.8%, 37.9%]
Usable outcomesruns: 222; skipped: 0; excluded: 0
Passed on retry0 (explicit failed attempts before a pass)
Retry exhaustion0 (every recorded attempt failed)
Flip rate0.5% of 221 usable consecutive pairs
Same-context rerun disagreement0/32 commit/branch/runner/OS groups
Flakiness score0.128
Change point0% → 100% failure rate at run run-0152 (commit 00082aaaaaaa)
Duration trend (without retries)+0.0 ms/observation
Duration × failure correlation+0.10
Failure rate by runnermacos-latest: 32% (n=47), ubuntu-latest: 31% (n=175)
Failure rate by OSmacos: 32% (n=47), linux: 31% (n=175)
Failure messages
  • Failed: off-by-one regression introduced at commit 130 ×70
test_reads_cache_depends_on_order INTERMITTENT 48%
Final failure rate (95% CI)47.7% [41.3%, 54.3%]
Usable outcomesruns: 222; skipped: 0; excluded: 0
Passed on retry0 (explicit failed attempts before a pass)
Retry exhaustion0 (every recorded attempt failed)
Flip rate48.0% of 221 usable consecutive pairs
Same-context rerun disagreement19/32 commit/branch/runner/OS groups
Flakiness score0.620
Duration trend (without retries)-0.0 ms/observation
Duration × failure correlation+0.12
Failure rate by runnermacos-latest: 51% (n=47), ubuntu-latest: 47% (n=175)
Failure rate by OSmacos: 51% (n=47), linux: 47% (n=175)
Failure messages
  • AssertionError: cache was not warmed before this test ran ×106
test_tight_timeout_race INTERMITTENT 29%
Final failure rate (95% CI)29.3% [23.7%, 35.6%]
Usable outcomesruns: 222; skipped: 0; excluded: 0
Passed on retry0 (explicit failed attempts before a pass)
Retry exhaustion0 (every recorded attempt failed)
Flip rate44.3% of 221 usable consecutive pairs
Same-context rerun disagreement10/32 commit/branch/runner/OS groups
Flakiness score0.419
Duration trend (without retries)+0.0 ms/observation
Duration × failure correlation+0.79
Failure rate by runnermacos-latest: 51% (n=47), ubuntu-latest: 23% (n=175)
Failure rate by OSmacos: 51% (n=47), linux: 23% (n=175)
Failure messages
  • AssertionError: took 0.0135s, budget 0.0125s ×65
test_probabilistic_threshold INTERMITTENT 12%
Final failure rate (95% CI)11.7% [8.1%, 16.6%]
Usable outcomesruns: 222; skipped: 0; excluded: 0
Passed on retry0 (explicit failed attempts before a pass)
Retry exhaustion0 (every recorded attempt failed)
Flip rate19.9% of 221 usable consecutive pairs
Same-context rerun disagreement8/32 commit/branch/runner/OS groups
Flakiness score0.226
Duration trend (without retries)-0.0 ms/observation
Duration × failure correlation+0.18
Failure rate by runnerubuntu-latest: 13% (n=175), macos-latest: 9% (n=47)
Failure rate by OSlinux: 13% (n=175), macos: 9% (n=47)
Failure messages
  • assert 0.10490622573466313 >= 0.12 ×26
test_sleep_a_bit 0%
Final failure rate (95% CI)0.0% [0.0%, 1.7%]
Usable outcomesruns: 222; skipped: 0; excluded: 0
Passed on retry0 (explicit failed attempts before a pass)
Retry exhaustion0 (every recorded attempt failed)
Flip rate0.0% of 221 usable consecutive pairs
Same-context rerun disagreement0/32 commit/branch/runner/OS groups
Flakiness score0.000
Duration trend (without retries)+0.0 ms/observation
Failure rate by runnermacos-latest: 0% (n=47), ubuntu-latest: 0% (n=175)
Failure rate by OSlinux: 0% (n=175), macos: 0% (n=47)
test_string_formatting 0%
Final failure rate (95% CI)0.0% [0.0%, 1.7%]
Usable outcomesruns: 222; skipped: 0; excluded: 0
Passed on retry0 (explicit failed attempts before a pass)
Retry exhaustion0 (every recorded attempt failed)
Flip rate0.0% of 221 usable consecutive pairs
Same-context rerun disagreement0/32 commit/branch/runner/OS groups
Flakiness score0.000
Duration trend (without retries)+0.0 ms/observation
Failure rate by runnermacos-latest: 0% (n=47), ubuntu-latest: 0% (n=175)
Failure rate by OSlinux: 0% (n=175), macos: 0% (n=47)
test_list_comprehension 0%
Final failure rate (95% CI)0.0% [0.0%, 1.7%]
Usable outcomesruns: 222; skipped: 0; excluded: 0
Passed on retry0 (explicit failed attempts before a pass)
Retry exhaustion0 (every recorded attempt failed)
Flip rate0.0% of 221 usable consecutive pairs
Same-context rerun disagreement0/32 commit/branch/runner/OS groups
Flakiness score0.000
Duration trend (without retries)+0.0 ms/observation
Failure rate by runnermacos-latest: 0% (n=47), ubuntu-latest: 0% (n=175)
Failure rate by OSlinux: 0% (n=175), macos: 0% (n=47)
test_string_split 0%
Final failure rate (95% CI)0.0% [0.0%, 1.7%]
Usable outcomesruns: 222; skipped: 0; excluded: 0
Passed on retry0 (explicit failed attempts before a pass)
Retry exhaustion0 (every recorded attempt failed)
Flip rate0.0% of 221 usable consecutive pairs
Same-context rerun disagreement0/32 commit/branch/runner/OS groups
Flakiness score0.000
Duration trend (without retries)-0.0 ms/observation
Failure rate by runnermacos-latest: 0% (n=47), ubuntu-latest: 0% (n=175)
Failure rate by OSlinux: 0% (n=175), macos: 0% (n=47)
test_populate_cache_first 0%
Final failure rate (95% CI)0.0% [0.0%, 1.7%]
Usable outcomesruns: 222; skipped: 0; excluded: 0
Passed on retry0 (explicit failed attempts before a pass)
Retry exhaustion0 (every recorded attempt failed)
Flip rate0.0% of 221 usable consecutive pairs
Same-context rerun disagreement0/32 commit/branch/runner/OS groups
Flakiness score0.000
Duration trend (without retries)+0.0 ms/observation
Failure rate by runnermacos-latest: 0% (n=47), ubuntu-latest: 0% (n=175)
Failure rate by OSlinux: 0% (n=175), macos: 0% (n=47)
test_dict_merge 0%
Final failure rate (95% CI)0.0% [0.0%, 1.7%]
Usable outcomesruns: 222; skipped: 0; excluded: 0
Passed on retry0 (explicit failed attempts before a pass)
Retry exhaustion0 (every recorded attempt failed)
Flip rate0.0% of 221 usable consecutive pairs
Same-context rerun disagreement0/32 commit/branch/runner/OS groups
Flakiness score0.000
Duration trend (without retries)+0.0 ms/observation
Failure rate by runnermacos-latest: 0% (n=47), ubuntu-latest: 0% (n=175)
Failure rate by OSlinux: 0% (n=175), macos: 0% (n=47)
test_sorted_is_stable 0%
Final failure rate (95% CI)0.0% [0.0%, 1.7%]
Usable outcomesruns: 222; skipped: 0; excluded: 0
Passed on retry0 (explicit failed attempts before a pass)
Retry exhaustion0 (every recorded attempt failed)
Flip rate0.0% of 221 usable consecutive pairs
Same-context rerun disagreement0/32 commit/branch/runner/OS groups
Flakiness score0.000
Duration trend (without retries)+0.0 ms/observation
Failure rate by runnermacos-latest: 0% (n=47), ubuntu-latest: 0% (n=175)
Failure rate by OSlinux: 0% (n=175), macos: 0% (n=47)
test_addition 0%
Final failure rate (95% CI)0.0% [0.0%, 1.7%]
Usable outcomesruns: 222; skipped: 0; excluded: 0
Passed on retry0 (explicit failed attempts before a pass)
Retry exhaustion0 (every recorded attempt failed)
Flip rate0.0% of 221 usable consecutive pairs
Same-context rerun disagreement0/32 commit/branch/runner/OS groups
Flakiness score0.000
Duration trend (without retries)+0.0 ms/observation
Failure rate by runnermacos-latest: 0% (n=47), ubuntu-latest: 0% (n=175)
Failure rate by OSlinux: 0% (n=175), macos: 0% (n=47)
pass passed on retry fail error skip not run unknown / excluded Rates use the final outcome of each run; retry attempts do not increase the sample size. Rows sorted broken → flaky → insufficient data → healthy, then by score. Columns are runs in chronological order. Click a row for detail. Hover a cell for the run.