Test-time compute and verification: when more samples hurt

Coverage and accuracy are different quantities. Coverage is the chance that at least one of N candidates is correct; accuracy is the chance that the one you select is. In the reference simulation of Thurstone bounds Best-of-N (base rate p = 0.10, discrimination d′ = 2, exploitability ε = 0.01), coverage at N = 64 is 0.9988 while accuracy peaks at 0.70, and by N = 8192 accuracy has fallen to 0.36. Selection, not generation, is binding.

With an honest verifier (ε = 0), accuracy never decreased with N in the simulation. So if your Best-of-N curve turns over as you add samples, the paper's reading is that you have measured exploitability, not noise.

Verifier exploitability is the rate at which wrong candidates score high: the verifier's blind-spot rate. It is separate from discrimination (d′), which is roughly what AUC measures, and AUC is nearly blind to ε at the rates that matter. Inverting published reward-model overoptimization curves gives ε ≈ 0.002–0.05 for the proxy reward models studied, a blind spot between 1 in 500 and 1 in 20 candidates. See the glossary entry.

Exploitability is not a property of the verifier alone. In a follow-up note, the same verifier's ε ranged from 0.153 to 0.949, a factor of 6.2, depending only on the prefilter that handed it candidates. Calibrate ε on the pool the verifier actually sees.

Compute the safe budget N*(ε) before scaling N. Past it, spend extra compute on a less exploitable verifier, not on more samples: a better judge moves the ceiling, while more candidates only move you further along a curve whose far end is the exploit. The verifier-budget tool computes N*(ε) and the selection rule in the browser, and estimates ε from a labelled validation set. It is a Best-of-N budget calculator built from the papers on this page.

Often, because counting is harder to fool than ranking. Plurality voting among high-scoring candidates defeats one-off exploits; a shared exploit only wins the plurality when its rate among wrong answers exceeds p/(1−p), where p is the share of correct candidates (about 0.111 for p = 0.10). In When the exploit is constructed, at a matched exploit rate, picking the top score gave 0.862 accuracy against sampled exploits and 0.000 against exploits constructed against the verifier, while counting among high scorers held at 0.999.

Two limits. At a 5% solve rate, the vote can be won by a common, ordinary wrong answer, which the threshold formula does not model; that note corrected our own shipped threshold. And the sampled-versus-constructed split itself is prior art (Dwork et al., 2015).

Because their errors overlap. For n judges with average pairwise error correlation ρ, the classical effective number of independent judges is n / (1 + (n − 1)ρ), which levels off at 1/ρ; at ρ = 0.5, sixteen samples are worth fewer than two independent ones. Our correlated-critique notesays first that this is mostly classical ensemble theory going back to Clemen and Winkler (1985), and that the simple formula holds for homogeneous, simple-majority panels rather than in general. What it measured: a critic sharing the generator's poisoned context missed 12 of 12 planted errors that fresh-context critics caught. The verifier-panel tool computes effective panel size and the majority-vote ceiling from two numbers. The Research Desk post More thinking isn't more evidence applies the same arithmetic to reasoning models.

No. CorrectionThe pairwise form ε = Φ(−gap/(σ√2)) is Thurstone's Law of Comparative Judgment, Case V, from 1927: an identity, not a discovery. Our earlier R² = 0.997 claim measured Monte-Carlo error, and under heavy-tailed noise the formula under-predicts ε by 11× to 723×, so ε should be measured directly rather than extrapolated from a fit. The correction keeps the consensus threshold, the measured ε range and the tools. The original verifier exploitability paper is marked as a superseded draft; its successor is Thurstone bounds Best-of-N.

No. Iterating a weight-tied network for more steps is another. In Recurrent depth as test-time compute, a recurrent-depth network matched a feedforward network at 1/8 the parameters and extrapolated to about 8× the trained problem size on synthetic tasks, provided its update rule was contractive; a non-contractive update diverged or overthought.

• Paper • Preprint draft • Jul 27, 2026Thurstone bounds Best-of-N: a 1927 identity, calibrated for modern verifiersThe noise floor of Best-of-N selection is Thurstone's 1927 Case V identity, not a new law. Contributed: blind-spot rates ε ≈ 0.002–0.05 from inverting published overoptimization curves, an exact consensus threshold p/(1−p), and a budget rule N*(ε) with a working tool. Heavy-tailed score noise breaks the identity, so estimate ε directly.
• Paper • Superseded draft • Jun 27, 2026When more sampling hurts: verifier exploitability, not accuracy, bounds test-time computeBest-of-N sampling lifts coverage log-linearly, but realized accuracy is ceilinged by the verifier — and the property that matters is not discrimination but exploitability ε: the rate at which wrong candidates score high. Any ε > 0 makes accuracy non-monotone in N, peaking at a computable budget before declining.
• Paper • Correction • Jul 26, 2026Correction: the ε selection law is Thurstone's 1927 comparative judgment, not a discoveryOur ε law, ε = Φ(−gap/(σ√2)), is Thurstone's Case V identity; its R² = 0.997 measured Monte-Carlo error. It is not a universality class (723× off under heavy tails), and the forgetting row is Kim et al. 2022. The consensus threshold p/(1−p), the measured constants, and the tools stand.
• Paper • Research note • Jul 30, 2026When the exploit is constructed: ranking collapses, counting survivesAt a matched exploit rate, Best-of-8192 accuracy is 0.862 with sampled exploits and 0.000 with exploits constructed against the verifier — while plurality voting among high scorers holds at 0.999. The consensus threshold missed a dominant-wrong-answer regime and was corrected; the two-regime framing is prior art (Dwork et al. 2015).
• Paper • Research note • Jul 29, 2026Exploitability is a property of a verifier and its candidate pool, not the verifier aloneThe same verifier's exploitability ranged 6.2× (0.153–0.949) depending only on its prefilter, so ε belongs to the verifier–pool pair. Per-pool regime detection fails, but averaging over a window of ~50 pools succeeds. Against adaptive generators, decorrelated panels help only partly; an out-of-span channel did about 8× better.
• Paper • Research note • Sep 9, 2026A critic that shares the generator's prompt shares its errors: measured self-critique correlationSelf-critique failure is mostly classical correlated-jury theory, and the note says so first. Measured: a same-context critic missed 12/12 planted errors that fresh critics all caught, and a panel's (e, P_both) predicted its majority accuracy (0.841 vs 0.842). The lever was prompt scaffolding, and only for context-induced errors.
• Paper • Synthesis • Jul 27, 2026Recurrent depth as test-time compute: parameter-efficient, extrapolating, and bounded by contractivityA weight-tied recurrent-depth network matches a feedforward net at 1/8 the parameters and wins out of distribution (+0.36 ± 0.13, mean ± sd over 10 seeds; the original single-seed run gave +0.28), extrapolating about 8× in problem size. A non-contractive residual update causes divergence and overthinking; a gated contractive update fixes both.
• Paper • Preprint draft • Jun 27, 2026Catastrophic forgetting in class-incremental learning is a verifier-exploitability problemFreeze one near-perfect expert per task and all of class-incremental difficulty becomes selection — which expert to trust. Accuracy factorizes exactly as selection accuracy × within-task accuracy, and the selection term is pinned by the selector's exploitability ε. A November 2026 follow-up shows that selecting by a statistical model of shared features instead of expert confidence lifts Split-MNIST to 0.926.

• AI Academy • 4 lessonsTest-Time Compute & VerificationSpending compute at inference — sampling, Best-of-N, majority vote — and why the verifier, not the generator, so often sets the ceiling. Built on NeuroGenesys's own research and tools.
• Lesson • Test-Time Compute & VerificationSpending compute at inference: sampling, Best-of-N and coverage
• Lesson • Test-Time Compute & VerificationWhen the verifier is the bottleneck
• Lesson • Test-Time Compute & VerificationMajority vote, self-consistency and correlated judges
• Lesson • Test-Time Compute & VerificationReading AI results critically
• Research tool • In-browser · Python CLIverifier-budgetThe modelled Best-of-N sampling budget N*(ε) and the selection rule (argmax vs. consensus) from a verifier's measured exploitability, with an ε estimator for a labelled validation set. A model: real judges did not collapse by N = 32 in the lab's pre-registered test (findings 8.13).
• Research tool • In-browser · Python CLIverifier-panelHow much an LLM-judge panel is really worth: error correlation, effective panel size, the majority-vote ceiling — from two numbers or straight from a critics × items miss matrix — and a tail check before trusting a Φ-based ε.
• Free tool • In-browserTest-Time Compute Cost EstimatorWhat Best-of-N and self-consistency cost per query, and the accuracy they can buy: oracle coverage and majority vote against the number of samples.
• Free tool • In-browserAI Eval Sample Size CalculatorHow many eval items you need: the margin of error on one model's accuracy, and the sample size to detect a difference between two models, paired or not.
• Research Desk • Nov 05, 2026More thinking isn't more evidence: test-time compute meets correlated error.Sampling a reasoning model sixteen times does not give you sixteen opinions. New preprints put numbers on an old statistical warning.