flakemap — inspection report

Test reliability across CI history, computed from JUnit, NUnit XML and TRX.

runs 1 tests 15 run range report → report window 2026-10-21 → 2026-10-21
0broken
4flaky
0healthy
11insufficient data
4test-runs passed on retry
0test-runs excluded

Tests that fail together

9 tests in 2 exact outcome cohorts. Each group has the same normalized outcomes, retry states, skips and missing runs. A shared history is a lead to investigate, not proof of a common cause.

Download all shared-outcome evidence

Showing 2 of 2 cohorts, ordered by affected test-runs. Final failures and passed-on-retry results remain separate.

5 tests share 1 final-failure runs and 0 recovered runs

Matching run history

Retries exhausted (1)
report

Members (5)

Showing 5 of 5 affected test-runs. Source positions are 1-based result ordinals. Long text and retry lists are previewed; the JSON download retains every record.

Scroll the evidence table horizontally to inspect every column.

Original failures and recovered retries
Run / environmentTestReported outcomeSource and message
report
Environment not recorded		test_exhausted_callfail; exhausted; 3 attemptsreport.xml :: testcase 1

rerunFailure: call: Failed: service never ready

rerunFailure: call: Failed: service never ready

failure: call: Failed: service never ready

report
Environment not recorded		test_exhausted_call_and_teardownerror; exhausted; 3 attemptsreport.xml :: testcase 2

rerunError: call: Failed: call never ready either; teardown: Failed: teardown never ready

rerunError: call: Failed: call never ready either; teardown: Failed: teardown never ready

error: call: Failed: call never ready either; teardown: Failed: teardown never ready

report
Environment not recorded		test_exhausted_setuperror; exhausted; 3 attemptsreport.xml :: testcase 3

rerunError: setup: Failed: setup never ready

rerunError: setup: Failed: setup never ready

error: setup: Failed: setup never ready

report
Environment not recorded		test_exhausted_teardownerror; exhausted; 3 attemptsreport.xml :: testcase 4

rerunError: teardown: Failed: teardown never ready

rerunError: teardown: Failed: teardown never ready

error: teardown: Failed: teardown never ready

report
Environment not recorded		test_strict_xpassfail; exhausted; 3 attemptsreport.xml :: testcase 13

rerunFailure: call: [XPASS(strict)] strict unexpected pass

rerunFailure: call: [XPASS(strict)] strict unexpected pass

failure: call: [XPASS(strict)] strict unexpected pass

4 tests share 0 final-failure runs and 1 recovered runs

Matching run history

Passed on retry (1)
report

Members (4)

Showing 4 of 4 affected test-runs. Source positions are 1-based result ordinals. Long text and retry lists are previewed; the JSON download retains every record.

Scroll the evidence table horizontally to inspect every column.

Original failures and recovered retries
Run / environmentTestReported outcomeSource and message
report
Environment not recorded		test_recovered_callpass; recovered; 3 attemptsreport.xml :: testcase 8

flakyFailure: call: AssertionError: connection not ready

flakyFailure: call: AssertionError: connection not ready

report
Environment not recorded		test_recovered_call_and_teardownpass; recovered; 2 attemptsreport.xml :: testcase 9

flakyError: call: AssertionError: call failed first; teardown: AssertionError: teardown failed too

report
Environment not recorded		test_recovered_setuppass; recovered; 2 attemptsreport.xml :: testcase 10

flakyError: setup: AssertionError: setup not ready

report
Environment not recorded		test_recovered_teardownpass; recovered; 2 attemptsreport.xml :: testcase 11

flakyError: teardown: AssertionError: teardown not ready

Test × run heatmap

Tests under pytest retry capture.test_retries.py::

test_recovered_call INTERMITTENT 0%

pytest retry capture.test_retries.py::test_recovered_call

Final failure rate (95% CI)0% [0%–79%] · 0 of 1 runs
Usable outcomesruns: 1; skipped: 0; excluded: 0
Passed on retry1 (explicit failed attempts before a pass)
Retry exhaustion0 (every recorded attempt failed)
Flip rateunavailable: no usable consecutive pairs
Same-context rerun disagreementno commits with multiple runs observed
Flakiness score0.020
Activity1 run with observed failures (1 passed on retry) in 1 separate stretch: too few to place.
Timing observations (without retries)0
Failure messages
  • call: AssertionError: connection not ready ×2

Recorded evidence

Sources identify a file and its 1-based testcase position. Failure records follow XML order. Attempt timing and total retry cost are not inferred.

report — recovered

report.xml :: testcase 8 — reported pass

3 recorded attempts; 2 failed.

  • <flakyFailure> call: AssertionError: connection not ready
  • <flakyFailure> call: AssertionError: connection not ready
test_recovered_call_and_teardown INTERMITTENT 0%

pytest retry capture.test_retries.py::test_recovered_call_and_teardown

Final failure rate (95% CI)0% [0%–79%] · 0 of 1 runs
Usable outcomesruns: 1; skipped: 0; excluded: 0
Passed on retry1 (explicit failed attempts before a pass)
Retry exhaustion0 (every recorded attempt failed)
Flip rateunavailable: no usable consecutive pairs
Same-context rerun disagreementno commits with multiple runs observed
Flakiness score0.020
Activity1 run with observed failures (1 passed on retry) in 1 separate stretch: too few to place.
Timing observations (without retries)0
Failure messages
  • call: AssertionError: call failed first; teardown: AssertionError: teardown failed too ×1

Recorded evidence

Sources identify a file and its 1-based testcase position. Failure records follow XML order. Attempt timing and total retry cost are not inferred.

report — recovered

report.xml :: testcase 9 — reported pass

2 recorded attempts; 1 failed.

  • <flakyError> call: AssertionError: call failed first; teardown: AssertionError: teardown failed too
test_recovered_setup INTERMITTENT 0%

pytest retry capture.test_retries.py::test_recovered_setup

Final failure rate (95% CI)0% [0%–79%] · 0 of 1 runs
Usable outcomesruns: 1; skipped: 0; excluded: 0
Passed on retry1 (explicit failed attempts before a pass)
Retry exhaustion0 (every recorded attempt failed)
Flip rateunavailable: no usable consecutive pairs
Same-context rerun disagreementno commits with multiple runs observed
Flakiness score0.020
Activity1 run with observed failures (1 passed on retry) in 1 separate stretch: too few to place.
Timing observations (without retries)0
Failure messages
  • setup: AssertionError: setup not ready ×1

Recorded evidence

Sources identify a file and its 1-based testcase position. Failure records follow XML order. Attempt timing and total retry cost are not inferred.

report — recovered

report.xml :: testcase 10 — reported pass

2 recorded attempts; 1 failed.

  • <flakyError> setup: AssertionError: setup not ready
test_recovered_teardown INTERMITTENT 0%

pytest retry capture.test_retries.py::test_recovered_teardown

Final failure rate (95% CI)0% [0%–79%] · 0 of 1 runs
Usable outcomesruns: 1; skipped: 0; excluded: 0
Passed on retry1 (explicit failed attempts before a pass)
Retry exhaustion0 (every recorded attempt failed)
Flip rateunavailable: no usable consecutive pairs
Same-context rerun disagreementno commits with multiple runs observed
Flakiness score0.020
Activity1 run with observed failures (1 passed on retry) in 1 separate stretch: too few to place.
Timing observations (without retries)0
Failure messages
  • teardown: AssertionError: teardown not ready ×1

Recorded evidence

Sources identify a file and its 1-based testcase position. Failure records follow XML order. Attempt timing and total retry cost are not inferred.

report — recovered

report.xml :: testcase 11 — reported pass

2 recorded attempts; 1 failed.

  • <flakyError> teardown: AssertionError: teardown not ready
test_exhausted_call 100%

pytest retry capture.test_retries.py::test_exhausted_call

Final failure rate (95% CI)100% [21%–100%] · 1 of 1 runs
Usable outcomesruns: 1; skipped: 0; excluded: 0
Passed on retry0 (explicit failed attempts before a pass)
Retry exhaustion1 (every recorded attempt failed)
Flip rateunavailable: no usable consecutive pairs
Same-context rerun disagreementno commits with multiple runs observed
Flakiness score0.000
Activity1 failures: too few to place.
Timing observations (without retries)0
Failure messages
  • call: Failed: service never ready ×3

Failures, newest first

RunSame commit, other runsOther tests failedNamed in output, or message
report
2026-10-21 12:36		commit not recorded4call: Failed: service never ready

Recorded evidence

Sources identify a file and its 1-based testcase position. Failure records follow XML order. Attempt timing and total retry cost are not inferred.

report — exhausted

report.xml :: testcase 1 — reported fail

3 recorded attempts; 3 failed.

  • <rerunFailure> call: Failed: service never ready
  • <rerunFailure> call: Failed: service never ready
  • <failure> call: Failed: service never ready
test_exhausted_call_and_teardown 100%

pytest retry capture.test_retries.py::test_exhausted_call_and_teardown

Final failure rate (95% CI)100% [21%–100%] · 1 of 1 runs
Usable outcomesruns: 1; skipped: 0; excluded: 0
Passed on retry0 (explicit failed attempts before a pass)
Retry exhaustion1 (every recorded attempt failed)
Flip rateunavailable: no usable consecutive pairs
Same-context rerun disagreementno commits with multiple runs observed
Flakiness score0.000
Activity1 failures: too few to place.
Timing observations (without retries)0
Failure messages
  • call: Failed: call never ready either; teardown: Failed: teardown never ready ×3

Failures, newest first

RunSame commit, other runsOther tests failedNamed in output, or message
report
2026-10-21 12:36		commit not recorded4call: Failed: call never ready either; teardown: Failed: teardown never ready

Recorded evidence

Sources identify a file and its 1-based testcase position. Failure records follow XML order. Attempt timing and total retry cost are not inferred.

report — exhausted

report.xml :: testcase 2 — reported error

3 recorded attempts; 3 failed.

  • <rerunError> call: Failed: call never ready either; teardown: Failed: teardown never ready
  • <rerunError> call: Failed: call never ready either; teardown: Failed: teardown never ready
  • <error> call: Failed: call never ready either; teardown: Failed: teardown never ready
test_exhausted_setup 100%

pytest retry capture.test_retries.py::test_exhausted_setup

Final failure rate (95% CI)100% [21%–100%] · 1 of 1 runs
Usable outcomesruns: 1; skipped: 0; excluded: 0
Passed on retry0 (explicit failed attempts before a pass)
Retry exhaustion1 (every recorded attempt failed)
Flip rateunavailable: no usable consecutive pairs
Same-context rerun disagreementno commits with multiple runs observed
Flakiness score0.000
Activity1 failures: too few to place.
Timing observations (without retries)0
Failure messages
  • setup: Failed: setup never ready ×3

Failures, newest first

RunSame commit, other runsOther tests failedNamed in output, or message
report
2026-10-21 12:36		commit not recorded4setup: Failed: setup never ready

Recorded evidence

Sources identify a file and its 1-based testcase position. Failure records follow XML order. Attempt timing and total retry cost are not inferred.

report — exhausted

report.xml :: testcase 3 — reported error

3 recorded attempts; 3 failed.

  • <rerunError> setup: Failed: setup never ready
  • <rerunError> setup: Failed: setup never ready
  • <error> setup: Failed: setup never ready
test_exhausted_teardown 100%

pytest retry capture.test_retries.py::test_exhausted_teardown

Final failure rate (95% CI)100% [21%–100%] · 1 of 1 runs
Usable outcomesruns: 1; skipped: 0; excluded: 0
Passed on retry0 (explicit failed attempts before a pass)
Retry exhaustion1 (every recorded attempt failed)
Flip rateunavailable: no usable consecutive pairs
Same-context rerun disagreementno commits with multiple runs observed
Flakiness score0.000
Activity1 failures: too few to place.
Timing observations (without retries)0
Failure messages
  • teardown: Failed: teardown never ready ×3

Failures, newest first

RunSame commit, other runsOther tests failedNamed in output, or message
report
2026-10-21 12:36		commit not recorded4teardown: Failed: teardown never ready

Recorded evidence

Sources identify a file and its 1-based testcase position. Failure records follow XML order. Attempt timing and total retry cost are not inferred.

report — exhausted

report.xml :: testcase 4 — reported error

3 recorded attempts; 3 failed.

  • <rerunError> teardown: Failed: teardown never ready
  • <rerunError> teardown: Failed: teardown never ready
  • <error> teardown: Failed: teardown never ready
test_parameter_identity[<xml>&[name]] 0%

pytest retry capture.test_retries.py::test_parameter_identity[<xml>&[name]]

Final failure rate (95% CI)0% [0%–79%] · 0 of 1 runs
Usable outcomesruns: 1; skipped: 0; excluded: 0
Passed on retry0 (explicit failed attempts before a pass)
Retry exhaustion0 (every recorded attempt failed)
Flip rateunavailable: no usable consecutive pairs
Same-context rerun disagreementno commits with multiple runs observed
Flakiness score0.000
Timing observations (without retries)1
test_parameter_identity[ordinary] 0%

pytest retry capture.test_retries.py::test_parameter_identity[ordinary]

Final failure rate (95% CI)0% [0%–79%] · 0 of 1 runs
Usable outcomesruns: 1; skipped: 0; excluded: 0
Passed on retry0 (explicit failed attempts before a pass)
Retry exhaustion0 (every recorded attempt failed)
Flip rateunavailable: no usable consecutive pairs
Same-context rerun disagreementno commits with multiple runs observed
Flakiness score0.000
Timing observations (without retries)1
test_pass 0%

pytest retry capture.test_retries.py::test_pass

Final failure rate (95% CI)0% [0%–79%] · 0 of 1 runs
Usable outcomesruns: 1; skipped: 0; excluded: 0
Passed on retry0 (explicit failed attempts before a pass)
Retry exhaustion0 (every recorded attempt failed)
Flip rateunavailable: no usable consecutive pairs
Same-context rerun disagreementno commits with multiple runs observed
Flakiness score0.000
Timing observations (without retries)1
test_skipped n/a

pytest retry capture.test_retries.py::test_skipped

Final failure rate (95% CI)unknown: no usable outcomes
Usable outcomesruns: 0; skipped: 1; excluded: 0
Passed on retry0 (explicit failed attempts before a pass)
Retry exhaustion0 (every recorded attempt failed)
Flip rateunavailable: no usable consecutive pairs
Same-context rerun disagreementno commits with multiple runs observed
Flakiness score0.000
Timing observations (without retries)0
test_strict_xpass 100%

pytest retry capture.test_retries.py::test_strict_xpass

Final failure rate (95% CI)100% [21%–100%] · 1 of 1 runs
Usable outcomesruns: 1; skipped: 0; excluded: 0
Passed on retry0 (explicit failed attempts before a pass)
Retry exhaustion1 (every recorded attempt failed)
Flip rateunavailable: no usable consecutive pairs
Same-context rerun disagreementno commits with multiple runs observed
Flakiness score0.000
Activity1 failures: too few to place.
Timing observations (without retries)0
Failure messages
  • call: [XPASS(strict)] strict unexpected pass ×3

Failures, newest first

RunSame commit, other runsOther tests failedNamed in output, or message
report
2026-10-21 12:36		commit not recorded4call: [XPASS(strict)] strict unexpected pass

Recorded evidence

Sources identify a file and its 1-based testcase position. Failure records follow XML order. Attempt timing and total retry cost are not inferred.

report — exhausted

report.xml :: testcase 13 — reported fail

3 recorded attempts; 3 failed.

  • <rerunFailure> call: [XPASS(strict)] strict unexpected pass
  • <rerunFailure> call: [XPASS(strict)] strict unexpected pass
  • <failure> call: [XPASS(strict)] strict unexpected pass
test_xfail n/a

pytest retry capture.test_retries.py::test_xfail

Final failure rate (95% CI)unknown: no usable outcomes
Usable outcomesruns: 0; skipped: 1; excluded: 0
Passed on retry0 (explicit failed attempts before a pass)
Retry exhaustion0 (every recorded attempt failed)
Flip rateunavailable: no usable consecutive pairs
Same-context rerun disagreementno commits with multiple runs observed
Flakiness score0.000
Timing observations (without retries)0
test_xpass 0%

pytest retry capture.test_retries.py::test_xpass

Final failure rate (95% CI)0% [0%–79%] · 0 of 1 runs
Usable outcomesruns: 1; skipped: 0; excluded: 0
Passed on retry0 (explicit failed attempts before a pass)
Retry exhaustion0 (every recorded attempt failed)
Flip rateunavailable: no usable consecutive pairs
Same-context rerun disagreementno commits with multiple runs observed
Flakiness score0.000
Timing observations (without retries)1
pass passed on retry fail error skip not run unknown / excluded Rates use the final outcome of each run; retry attempts do not increase the sample size. Rows sorted broken → new → flaky → quiet → insufficient data → healthy, then by score. Columns are runs in chronological order. Click a row for detail. Hover a cell for the run.