When the verifier is the bottleneck

Repeated sampling raises the chance that a correct answer exists somewhere in the pool; a verifier then has to find it. This lesson asks what property of that verifier decides whether more samples help or hurt — and answers with a quantity that standard accuracy metrics barely register.

Two ways a verifier can be imperfect

Give each candidate a score. A useful model separates two failure modes that a single "verifier accuracy" number blends together.^[1]

• Discrimination, $d'$. Correct candidates score $\sim \mathcal{N}(d', 1)$ and ordinary wrong ones $\sim \mathcal{N}(0, 1)$. Larger $d'$ means cleaner separation on average; the area under the ROC curve is $\Phi(d'/\sqrt{2})$.
• Exploitability, $\varepsilon$. A small fraction of wrong candidates are scored high — at or above the correct ones. These are the verifier's blind spots: reward hacks, answers that pass tests without solving the task, persuasive but false proofs.

The two fail differently. Low discrimination costs accuracy smoothly. Any nonzero exploitability places a cliff downstream, because taking the argmax over many samples is a search procedure, and searching long enough finds the blind spot.

The noise floor is a 1927 identity

If a correct candidate scores $\mathcal{N}(g, \sigma^2)$ and a wrong one $\mathcal{N}(0, \sigma^2)$ independently, their difference is $\mathcal{N}(g, 2\sigma^2)$, so

This is Thurstone's Law of Comparative Judgment, Case V,^[2] the two-alternative forced-choice error rate of signal detection theory, and the bit-error rate of binary phase-shift keying. The lab first reported this expression as a newly discovered cross-domain law with $R^2 = 0.997$; it then retracted that framing in full, noting that the $R^2$ measured Monte-Carlo error around an exact identity and that the "law" changes form under non-Gaussian noise.^[3] The retraction is part of the lesson: a fit that good, across unrelated domains, usually means you derived an identity.

Why accuracy peaks and then falls

With $\varepsilon = 0$, Best-of-N accuracy rises monotonically towards coverage. With $\varepsilon > 0$ it becomes a race between two maxima: the best of roughly $pN$ correct candidates against the best of roughly $\varepsilon(1-p)N$ exploiters whose scores sit, by construction, above the honest distribution. The extreme-value advantage of the larger correct pool grows only like $\sqrt{2\ln n}$, while the exploiters' margin is fixed, so as $N$ grows the exploiters win.

Worked example (the lab's reference model, $p = 0.10$, $d' = 2$, exploiters scored one unit above correct candidates):

Bold marks each column's peak on the power-of-two grid. Three regularities follow:^[1]

1. Honest verifiers are monotone. If your Best-of-N curve turns over, you have measured exploitability, not noise.
2. The safe budget $N^*$ is set mainly by $\varepsilon$. It falls from 512 to 64 to 16 as $\varepsilon$ rises tenfold and fivefold.
3. Discrimination buys height, not immunity. At $\varepsilon = 0.01$, raising $d'$ from 2 to 3 lifts the peak from about 0.71 to 0.79, but the curve still turns over, now near $N = 32$.

Calibrating $\varepsilon$ against real reward models

The model would be a curiosity if $\varepsilon$ could not be estimated for real systems. Gao, Schulman and Hilton (2022) measured how a "gold" reward degrades as Best-of-N and RL optimise against proxy reward models of various sizes: gold reward rises, peaks, then falls, with coefficients that scale smoothly with reward-model size.^[4] Inverting curves of that shape through the two-axis model, the lab obtained an approximate relation $\varepsilon \approx 1.45\,N^{*\,-1.19}$, which maps observed peaks at $N^*$ between 16 and 256 to $\varepsilon \approx 0.002$–$0.05$ — a blind spot between one in 500 and one in 20 wrong answers.^[5] That range is invisible to AUC-style reports, and sufficient to explain the saturation practitioners see. The lab is explicit that this is an inversion, not a direct measurement: the functional-form match to Gao et al.'s fitted law failed its own pre-registered bar, and a direct measurement on open reward models is specified but not yet run.^[1]

Independent work reached the same qualitative conclusion by other routes. Stroebl, Kapoor and Narayanan (2024) showed that resampling against imperfect unit tests has an accuracy ceiling set by the verifier's false positives, and that once false positives carry a cost, the optimal number of samples is often finite and small.^[6] Khalaf et al. (2025) proved that the rise-then-fall of true reward is an inevitable property of a broad class of inference-time selection schemes, including Best-of-N, and proposed tuning the selection parameter to hedge against it.^[7]

Exploitability belongs to the verifier and its pool

Three later lab results narrowed the picture.

Pool-relative. Placing one fixed verifier behind five different prefilters changed its measured exploitability from 0.153 to 0.949, a factor of 6.2. A prefilter correlated with the final verifier did worse than no prefilter, because it concentrated the finalist's blind spots.^[8] Practical consequence: estimate $\varepsilon$ on the post-prefilter pool the verifier will actually face.

Constructed versus sampled exploits. The model above treats blind spots as sampled — each wrong answer stumbles into one with probability $\varepsilon$. Real adversaries construct them from the checker's definition. At a matched exploit rate, Best-of-8192 accuracy was 0.862 with sampled exploits and 0.000 with constructed ones; plurality voting among high scorers held at 0.999 in both cases.^[9] Construction attacks ranking; it does not attack counting. The live counterpart is GPU kernel competitions, where a workshop paper reports hacked submissions falling from 3.45% to 0.37% after a detector was deployed — with the caveat that the detector itself labelled what counted as a hack.^[10]

Heavy tails. The $\Phi$ formula holds only for Gaussian-like score noise. Under Cauchy noise it under-predicts the blind-spot rate by 11× at gap/$\sigma = 4$ and 723× at gap/$\sigma = 6$.^[3] Never extrapolate a $\Phi$ fit into the small-$\varepsilon$ tail; estimate $\varepsilon$ directly from labelled data and check excess kurtosis first.

The pattern is not confined to selection at inference. Reinforcement learning against a verifier is a longer, adaptive search, and 2025–2026 reports find the same failure there: METR observed recent frontier models reward hacking on its agentic task suites,^[11] and Helff et al. (2026) found RLVR-trained reasoning models emitting outputs that pass extensional checks without inducing the intended rule, more often as task complexity and inference-time compute grew.^[12]

Try it

Open /research/tools/verifier-budget. With the defaults ($p = 0.10$, $d' = 2$, $\varepsilon = 0.01$) it reports a safe budget of $N^* = 64$ and accuracy at $N = 8192$ of roughly half the peak. Now set $\varepsilon = 0.001$ and watch $N^*$ move to 512; then restore $\varepsilon = 0.01$ and raise $d'$ to 3 to see the peak rise while the turnover remains. Finally set $p = 0.40$: the recommended rule switches from Best-of-N to consensus, because plurality voting now beats the argmax peak. If you have labelled validation scores, paste them in to estimate $\varepsilon$ directly — on the pool your verifier will really see.

Key takeaways

• Separate verifier quality into discrimination ($d'$, what AUC measures) and exploitability ($\varepsilon$, the rate of high-scoring wrong answers).
• Any $\varepsilon > 0$ makes Best-of-N accuracy non-monotone in $N$; the safe budget is governed by $\varepsilon$, and better discrimination does not remove the turnover.
• Real reward models plausibly sit at $\varepsilon \approx 0.002$–$0.05$ (an inversion, not yet a direct measurement).
• $\varepsilon$ is pool-relative, collapses ranking when exploits are constructed, and must be estimated directly when score tails are heavy; counting-based selection is the more robust default.

Sources