NeuroGenesysHow many samples before more samples hurt?
Best-of-N coverage keeps rising with N, but realized accuracy is gated by the verifier. Any exploitability ε > 0 — wrong answers the verifier scores high — makes accuracy peak at a computable budget N* and then decline. Enter your verifier's numbers to get N* and whether to select by argmax or by consensus.
Treat N* as a model, not a measured prediction. In the lab's pre-registered real-judge study, no reward model, LLM judge or self log-probability made accuracy fall by N = 32, and measured ε did not predict the best budget (findings 8.13).
Background: Thurstone bounds Best-of-N and When the exploit is constructed.
1 · Your verifier
Optional · estimate ε and p from a labelled validation set
Paste one candidate per line — its verifier score and whether it was correct (1/0) — or load a two-column CSV such as score,correct. ε is the fraction of wrong candidates scoring at or above the median correct score — the blind spots a Best-of-N search will surface. Nothing leaves your browser.
2 · Modelled budget and selection rule
| N | coverage | Best-of-N | majority | |
|---|---|---|---|---|
| 1 | 0.100 | 0.100 | 0.100 | |
| 2 | 0.190 | 0.175 | 0.100 | |
| 4 | 0.344 | 0.284 | 0.077 | |
| 8 | 0.570 | 0.423 | 0.041 | |
| 16 | 0.815 | 0.564 | 0.012 | |
| 32 | 0.966 | 0.667 | 0.001 | |
| 64 | 0.999 | 0.707 | 0.000 | |
| 128 | 1.000 | 0.692 | 0.000 | |
| 256 | 1.000 | 0.647 | 0.000 | |
| 512 | 1.000 | 0.588 | 0.000 | |
| 1,024 | 1.000 | 0.526 | 0.000 | |
| 2,048 | 1.000 | 0.465 | 0.000 | |
| 4,096 | 1.000 | 0.408 | 0.000 | |
| 8,192 | 1.000 | 0.355 | 0.000 |
Computed exactly by numerical integration of the verifier-bottleneck model (exploit margin 1.0, distractor mass 0.5); the Python CLI estimates the same curves by Monte-Carlo, so on a flat peak the two can pick neighbouring N*.
Caveats that travel with the number
- ε is pool-relative.The same verifier's ε moved 6.2× across upstream prefilters. Calibrate on the post-prefilter pool it will actually see, and never prefilter with a scorer correlated with this verifier.
- Constructed exploits. If an optimizer, leaderboard or adversary can probe the verifier, treat N* as an upper bound — the post-peak floor is 0, not a residual. Construction attacks ranking, not counting, so consensus matters more there, not less.
- Check the tails.Don't extrapolate a Φ-fit of ε to small values without the tail check in verifier-panel.
Run it locally (Python)
pip install numpy python verifier_budget.py --p 0.10 --dprime 2.0 --eps 0.01 python verifier_budget.py --table # the reference ε sweep # or install the CLI: pip install . → verifier-budget --p 0.10 --eps 0.01
- verifier_budget.pythe CLI
- verifier_bottleneck.pyMonte-Carlo model (required)
- vbcore.pyshared sampler
- pyproject.tomlMIT · pip install .
- README.md
