Test-Time Compute & Verification

Spending compute at inference — sampling, Best-of-N, majority vote — and why the verifier, not the generator, so often sets the ceiling. Built on NeuroGenesys's own research and tools.

4 lessons • 35 min read

1. 01

   Spending compute at inference: sampling, Best-of-N and coverage

   A trained model's weights are fixed, but how much computation you spend per question is not: you can sample many answers, revise, search, or iterate a recurrent block longer. This lesson sets out the arithmetic of that choice — what repeated sampling buys (coverage), what it does not buy on its own (accuracy), and how the field learned to tell them apart.

   8 min read
2. 02

   When the verifier is the bottleneck

   Repeated sampling raises the chance that a correct answer exists somewhere in the pool; a verifier then has to find it. This lesson asks what property of that verifier decides whether more samples help or hurt — and answers with a quantity that standard accuracy metrics barely register.

   9 min read
3. 03

   Majority vote, self-consistency and correlated judges

   Asking several models — or one model several times — and taking a vote is the cheapest defence against a bad verifier. This lesson works out when voting helps, why it often helps far less than the head-count suggests, and how two measurable numbers predict the ceiling.

   9 min read
4. 04

   Reading AI results critically

   A headline number in an AI paper — "solves 56% of issues", "beats a 14× larger model", "a new law with R^2 = 0.99" — is the output of a measurement chain: a benchmark, a selection rule, a judge, a compute budget, a statistical test. This lesson gives a working checklist for auditing that chain, drawn from the failure modes covered in this course and from the NeuroGenesys lab's own published corrections.

   9 min read