NeuroGenesysIs sampling more worth what it costs?
Sampling a model several times and keeping the best or most common answer is the simplest way to spend more compute at inference time. Each extra sample adds its full output cost. This estimator puts the bill next to what you get for it: majority-vote accuracy, which needs no verifier, and coverage, the most a perfect verifier could get.
Background: test-time compute topic hub.
Free · runs in your browser · updated
1 · Model and task
2 · At N = 8
3 · Accuracy and cost against N
| N | cost / query | cost / month | majority | coverage |
|---|---|---|---|---|
| 1 | $0.02 | $230 | 60.0% | 60.0% |
| 2 | $0.04 | $433 | 60.0% | 84.0% |
| 4 | $0.08 | $839 | 75.4% | 97.4% |
| 8 | $0.17 | $1,651 | 87.8% | 99.9% |
| 16 | $0.33 | $3,275 | 96.2% | 100.0% |
| 32 | $0.65 | $6,523 | 99.5% | 100.0% |
| 64 | $1.30 | $13,019 | 100.0% | 100.0% |
| 128 | $2.60 | $26,011 | 100.0% | 100.0% |
A tie with the leading wrong answer counts as a miss. Samples are treated as independent; real samples from one model are correlated, so both curves are optimistic. A learned verifier sits between the two curves and can fall below majority vote at large N when it can be gamed — the verifier-budget research tool finds the safe N for a measured verifier. Prices: Claude Sonnet 5.5 list price as of Nov 09, 2026.
Questions
What is test-time compute?
Compute spent when the model answers rather than when it is trained: longer reasoning, several samples per question, or search over candidate answers. It trades cost and latency per query for accuracy.
What is self-consistency?
Sampling several answers to the same question and returning the most common one (majority vote). It helps when wrong answers are scattered and the right one recurs, and stops helping when one wrong answer is as popular as the right one.
What is pass@k or coverage?
The chance that at least one of k samples is correct. It is the ceiling for Best-of-N: you reach it only if you can tell which sample is right, for example by running unit tests or checking a proof.
Why is Best-of-N with a reward model not shown?
Its accuracy depends on how good and how gameable the reward model is, which a single number cannot capture. Our verifier-budget research tool models it from a verifier's measured exploitability and finds the N past which accuracy starts to fall.
Related
- Topic hubTest-time compute topic hub
- Researchverifier-budget (research tool)
- Free toolLLM API Cost CalculatorCompare what a workload costs per call, per day and per month across Claude, GPT and Gemini models, with prompt caching and long-context pricing.
- Free toolToken Counter & Context Window CheckerPaste text or load a file to count its tokens, see what sending it costs as input, and check which models it fits in. Nothing leaves your browser.
- Free toolAI Eval Sample Size CalculatorHow many eval items you need: the margin of error on one model's accuracy, and the sample size to detect a difference between two models, paired or not.
- Free toolLLM Training Compute CalculatorTraining FLOPs from parameters and tokens (6ND), GPU-hours at a realistic utilization, wall-clock time and cost — or the compute-optimal model for a budget.
- Free toolVRAM Calculator: Can I Run This LLM?GPU memory for an LLM at any quantization and context length — weights, KV cache and overhead — and which GPUs and Apple Silicon Macs can hold it.
- Free toolLLM Inference Speed Estimator (Tokens per Second)Tokens per second and time to first token for a model on a GPU or Mac, from memory bandwidth and compute, and how batching changes throughput.
- Free toolAgent Tool Count CalculatorHow often an agent picks the wrong tool as its library grows, from published benchmarks: every tool in the prompt, embedding or BM25 shortlists, with the measured range marked.
- Free toolLLM Judge Agreement CalculatorCheck an LLM-as-judge against human labels: Cohen's kappa with a 95% interval, the confusion matrix, sensitivity and specificity, and how many human labels the judge is worth.
- Free toolEU AI Act Risk CheckerAnswer a few questions to see your EU AI Act risk tier, the obligations with their articles, and the deadlines after the 2026 Digital Omnibus.
