errorbars — statistical power calculator

How many eval questions do I need?

Plan a paired evaluation before spending on model calls. Find the questions needed for an accuracy improvement, or the gap your existing benchmark can detect. Enter exact assumptions, compare budgets, and save a plan you can reproduce with errorbars power. Calculations stay in your browser.

Plan one comparison

What do you want to find?

Questions per model

—

Normal-approximation estimate. Check assumptions below.

What more questions buy you

Calculated budgets with the same assumptions
Questions per modelDetectable gapImprovement range
Reproduce this plan

The JSON download records all inputs, units, results, warnings and assumptions. No evaluation data is included.

View plan

Assumptions to check before running the eval

    Reading the numbers

    Baseline accuracy
    Roughly where your models score. Sets the per-question variance, p(1-p).
    Effect size (δ)
    The accuracy gap between two models you want to be able to detect, not just observe.
    Correlation (ρ)
    How often both models get the same questions right or wrong. Harder questions are harder for everyone, so ρ is usually positive — and pairing on it shrinks the standard error (see compare in the Python package).
    Cluster size & ICC
    If questions come in groups (e.g. several per reading passage) that share difficulty, effective sample size shrinks by the design effect, 1 + (size − 1) × ICC.

    Full derivations: docs/formulas.md §11. The test suite checks this page's TypeScript implementation against Python-generated vectors — see web/src/power.test.ts.