NeuroGenesysHow many eval items do you need?
A two-point gain on a 300-question benchmark is usually noise. Before you trust a leaderboard delta or pay for labels, check how big a difference your eval can resolve. Pick a question: how precisely one score is measured, or how many items it takes to tell two models apart.
Free · runs in your browser · updated
1 · Your question
2 · Answer
Assumes items are independent. Items that come in clusters (several questions per document, several turns per conversation) behave like fewer items; use the number of clusters, or inflate the answer.
Questions
How many questions does an LLM benchmark need?
It depends on the difference you want to see. Pinning an 80% accuracy to within ±2 points at 95% confidence takes about 1,540 items. Detecting a 3-point gain between two models on separate item sets takes thousands per model; on the same items it can take far fewer.
Why is comparing models on the same items more efficient?
Items both models get right, or both get wrong, cancel out. Only the items where they disagree carry information about which is better, so a paired test (McNemar's) needs fewer items, especially when the models agree on most of them.
Where do I get the disagreement rate?
Run both models on a pilot set of a few hundred items and count the share where exactly one is correct. It is at least the accuracy gap, and usually several times it.
What about running the same item several times?
Repeated samples on one item reduce sampling noise for that item but don't add new items. For deciding between models, more distinct items almost always beat more samples per item.
Related
- Researchthreshold-audit (research tool)
- Free toolLLM API Cost CalculatorCompare what a workload costs per call, per day and per month across Claude, GPT and Gemini models, with prompt caching and long-context pricing.
- Free toolToken Counter & Context Window CheckerPaste text or load a file to count its tokens, see what sending it costs as input, and check which models it fits in. Nothing leaves your browser.
- Free toolTest-Time Compute Cost EstimatorWhat Best-of-N and self-consistency cost per query, and the accuracy they can buy: oracle coverage and majority vote against the number of samples.
- Free toolLLM Training Compute CalculatorTraining FLOPs from parameters and tokens (6ND), GPU-hours at a realistic utilization, wall-clock time and cost — or the compute-optimal model for a budget.
- Free toolVRAM Calculator: Can I Run This LLM?GPU memory for an LLM at any quantization and context length — weights, KV cache and overhead — and which GPUs and Apple Silicon Macs can hold it.
- Free toolLLM Inference Speed Estimator (Tokens per Second)Tokens per second and time to first token for a model on a GPU or Mac, from memory bandwidth and compute, and how batching changes throughput.
- Free toolAgent Tool Count CalculatorHow often an agent picks the wrong tool as its library grows, from published benchmarks: every tool in the prompt, embedding or BM25 shortlists, with the measured range marked.
- Free toolLLM Judge Agreement CalculatorCheck an LLM-as-judge against human labels: Cohen's kappa with a 95% interval, the confusion matrix, sensitivity and specificity, and how many human labels the judge is worth.
- Free toolEU AI Act Risk CheckerAnswer a few questions to see your EU AI Act risk tier, the obligations with their articles, and the deadlines after the 2026 Digital Omnibus.
