Golden set size calculator
Most teams run 40 or 50 eval examples, watch the number move from 82% to 79%, and conclude the change made things worse. At that sample size, that difference is indistinguishable from noise — you’d see swings that large from rerunning the identical prompt.
This works out what your golden set can actually detect, and how big it would need to be to detect what you care about.
With that golden set, the smallest regression you can reliably detect is
—
For reference
| Golden set size | Smallest detectable drop | Meaning at your baseline |
|---|
Why the numbers are worse than you expected
Accuracy on a golden set is a proportion, and proportions are noisy. With 50 examples, a measured 85% has a 95% confidence interval of roughly ±10 points. Two runs of an unchanged system can easily differ by 6 points. Any "improvement" smaller than that is indistinguishable from resampling.
This is why eval suites get abandoned. They're built at a size that produces movement on every run, the movement doesn't correlate with anything real, people stop trusting it, and within a month nobody looks at the dashboard.
Paired evaluation is the cheap way out
The numbers above assume you're comparing two independent samples. But you're usually not — you run the same golden set before and after, which is a paired design. Paired comparison only counts the examples whose result actually changed, so it needs far fewer examples for the same sensitivity. The correct test is McNemar's test, not a two-proportion z-test.
That's what the paired line reports. It's the number worth designing around — but it depends on how many examples flip, which is why that's an input rather than an assumption.
What this doesn't model
A single binary metric on independent, representative examples. It doesn't account for correlated examples (ten questions about the same document aren't ten independent trials), multiple comparisons (testing eight metrics means one will look significant by chance), or the far more common problem that your golden set isn't representative of production traffic in the first place. Treat the output as a floor, not a target.