Statistical Significance: Why One Comparison Deceives
Model A scored 84%, model B scored 86%. Is B better, or did it get lucky on your test set? Significance testing separates a real difference from noise.
On this page
You run two models on your test set. A scores 84%, B scores 86%. You ship B. But you evaluated on a finite sample, and a different 200 examples might have flipped the result. The two-point gap could be a real improvement or it could be luck. Statistical significance is the discipline of telling those apart, and skipping it is how teams convince themselves that noise is progress.
The core problem: your test set is a sample
You never measure a model on all possible inputs. You measure on a sample and treat the result as an estimate of true performance. Like any estimate from a sample, it carries uncertainty — poll a different 200 people and you get a slightly different number, even with nothing else changed.
So when B beats A by 2 points, there are two explanations:
- B is genuinely better.
- B and A are equally good, and B happened to win on this particular sample.
Significance testing quantifies how likely explanation 2 is. If it is likely, you have learned nothing. If it is very unlikely, you have evidence for a real difference.
Sample size decides everything
The same 2-point gap means wildly different things at different scales.
- 50 examples: a 2-point difference is 1 example. Pure noise. Reshuffling would erase it.
- 200 examples: still well inside the range of random variation. Not trustworthy.
- 10,000 examples: a 2-point gap is now hundreds of examples moving the same way. Hard to explain by luck.
The intuition: small samples are jumpy, large samples are stable. A useful mental yardstick is that the uncertainty in a proportion shrinks roughly with the square root of the sample size — so cutting your uncertainty in half takes four times the data. This is why “we tested on 30 cases and B won” is not an argument, and why serious comparisons need either large test sets or the honesty to say the result is inconclusive.
p-values and what they actually mean
A significance test produces a p-value: the probability of seeing a difference at least this large if the two models were actually equal. Low p-value means the observed gap would be surprising under “no real difference”, so you have evidence there is one. The common threshold is 0.05 — a 5% chance of a false alarm.
Two misreadings to avoid, because they are everywhere:
- A p-value is not the probability that B is better. It is the probability of the data assuming the models are equal — a subtle but important difference.
- “Not significant” does not mean the models are equal. It means you lack the evidence to distinguish them, often because your test set is too small. Absence of proof is not proof of absence.
Confidence intervals say more than p-values
A p-value gives a yes/no verdict. A confidence interval gives the range of plausible values, which is almost always more useful. Instead of “B scored 86%”, report “B scored 86%, 95% CI [82%, 90%]”.
Now overlap tells the story at a glance. If A’s interval is [80%, 88%] and B’s is [82%, 90%], they overlap heavily — you cannot claim B is better. If the intervals are cleanly separated, you can. Intervals also convey magnitude: a difference can be statistically significant yet so small it does not matter in practice, and only the interval makes that visible. Prefer reporting intervals over bare point scores wherever you can.
Paired comparison: test on the same examples
There is a large, free improvement available when comparing two models: run both on the same test examples and compare them example by example, rather than comparing two independent averages.
Why it helps: much of the score variance comes from some examples being intrinsically hard and others easy. If both models see the same items, that shared difficulty cancels out, and you are left measuring the thing you care about — where they differ. A paired test (e.g. a paired test on per-example wins, or McNemar’s test for classification) detects real differences with far fewer examples than comparing separate runs. Always pair your comparisons when the setup allows it.
The multiple-comparisons trap
Test 20 prompt variants against a baseline at the 0.05 threshold and, even if none is truly better, you expect about one to look “significant” by chance — 5% of 20 is 1. Run enough comparisons and something will always cross the line. This is how eval dashboards produce phantom winners.
Guard against it: correct for the number of comparisons (Bonferroni is the simple, conservative option — divide your threshold by the number of tests), or hold out a fresh test set and confirm the winner there before believing it. This trap connects directly to benchmark contamination and Goodhart’s law — the more you test against a fixed set, the more you fit its noise. For live comparisons, the same logic governs A/B testing: decide the sample size up front, do not peek and stop the moment it looks significant.
What to remember
- Your test set is a sample, so any score carries uncertainty — a two-point gap can be a real gain or just luck on this sample.
- Sample size is decisive: uncertainty shrinks with roughly the square root of the count, so small tests cannot support strong claims.
- A p-value is the chance of the observed gap if the models were equal — not the chance B is better, and “not significant” means inconclusive, not equal.
- Report confidence intervals: overlap shows whether a difference is real, and width shows whether it is big enough to matter.
- Pair comparisons on the same examples to cancel shared difficulty, and correct for multiple comparisons so you do not crown a phantom winner.