errorbars — SWE-bench Verified, with error bars

Is the gap between two SWE-bench scores real?

Every SWE-bench Verified submission publishes which of the 500 tasks it resolved, so a difference between two scores can be tested instead of read off a leaderboard. Pick two of the public submissions with complete task outcomes, or paste a run of your own. The 441-task report is excluded from this fixed 500-task view; missing outcomes do not become observed failures. Nothing you paste leaves this page.

Compare two submissions

Use a run of my own as A

Approximate 95% intervals for A minus B. The paired-t interval is separate from the exact McNemar task test; they can disagree near the significance threshold. An interval containing zero does not establish equivalence. These are single comparisons without adjustment for browsing multiple pairs.
The comparison in numbers

The 500 tasks, by repository

  • both resolved
  • only A
  • only B
  • neither

Across the whole board

These retained study figures use paired-t approximations. The interactive task comparison above uses exact McNemar.

How to read this

Why a paired test
Both systems ran the same 500 tasks, and what one finds hard the other mostly does too. Only the tasks where they differ carry information about which is better, so the test works on those, and is far sharper than comparing two intervals.
Tasks as the sample
The headline uses exact McNemar on discordant binary outcomes, treating tasks as independent. The paired-t interval and p-value remain labeled approximations; they are not the confidence interval of the exact test.
Repositories as the sample
Answers: would A beat B on other codebases? The 500 tasks come from 12 repositories and 231 are Django, which leaves about 3.3 effective degrees of freedom. The interval is wide because there is that little to go on.
Not different is not equal
A gap inside the noise may still be real. What the test says is that this benchmark, at this size, cannot show it.

Data: per-task outcomes from SWE-bench/experiments at commit . Two submissions whose per-task file contradicts their stated score are left out. This page's statistics are checked against the Python package on real pairs — see web/src/paired.test.ts and docs/formulas.md.