NeuroGenesysCan you trust your LLM judge?
An LLM judge that agrees with humans 90% of the time can still be close to useless if 90% of the answers pass anyway. Paste judge and human labels, or enter a 2×2 table, to see agreement corrected for chance, where the judge errs, and how many human labels it can stand in for.
Free · runs in your browser · updated
1 · Your labels
2 · Agreement
Kappa 0.739 is “substantial” on the Landis & Koch scale. Chance alone would give 70.7% raw agreement here, which is why 92.3% agreement can still be weak when one label dominates.
| human ↓ · judge → | positive | negative |
|---|---|---|
| positive | 42 | 8 |
| negative | 15 | 235 |
3 · What the judge is worth
Questions
What is a good Cohen's kappa for an LLM judge?
By the common Landis & Koch convention, 0.61–0.80 is substantial and above 0.80 almost perfect. These bands are conventions, not tests. What matters more is the interval (a kappa of 0.7 from 50 items could easily be 0.5) and whether the judge's errors are the ones you care about, which sensitivity and specificity show.
Why use kappa instead of percent agreement?
Percent agreement counts matches that would happen by chance. If 90% of answers pass and the judge always says pass, it agrees 90% of the time with kappa 0. Kappa subtracts the agreement expected from each rater's label frequencies.
How many human labels do I need to validate an LLM judge?
Enough to make the kappa interval narrow and to see the rare class. For a failure rate near 10%, a few hundred labels give only a few dozen failures, so sensitivity stays uncertain. Label more of the cases the judge is unsure about, or enrich for failures, and keep a random sample for the overall rate.
How can a judge replace human labels without biasing the result?
Prediction-powered inference (PPI++, Angelopoulos et al.) combines a small human-labelled set with many judge verdicts. It uses the human set to correct the judge's bias and weights the judge by how well it tracks humans, so the estimate stays valid even when the judge is poor; a good judge makes the interval as tight as several times more human labels.
Which label should be 'positive'?
The outcome you want the judge to catch, usually fail, wrong or unsafe. Sensitivity is then the share of real failures the judge flags, and specificity the share of good answers it lets through.
Related
- Free toolAI eval sample size calculator
- ResearchSelection-reliability tool
- Free toolLLM API Cost CalculatorCompare what a workload costs per call, per day and per month across Claude, GPT and Gemini models, with prompt caching and long-context pricing.
- Free toolToken Counter & Context Window CheckerPaste text or load a file to count its tokens, see what sending it costs as input, and check which models it fits in. Nothing leaves your browser.
- Free toolTest-Time Compute Cost EstimatorWhat Best-of-N and self-consistency cost per query, and the accuracy they can buy: oracle coverage and majority vote against the number of samples.
- Free toolLLM Training Compute CalculatorTraining FLOPs from parameters and tokens (6ND), GPU-hours at a realistic utilization, wall-clock time and cost — or the compute-optimal model for a budget.
- Free toolVRAM Calculator: Can I Run This LLM?GPU memory for an LLM at any quantization and context length — weights, KV cache and overhead — and which GPUs and Apple Silicon Macs can hold it.
- Free toolLLM Inference Speed Estimator (Tokens per Second)Tokens per second and time to first token for a model on a GPU or Mac, from memory bandwidth and compute, and how batching changes throughput.
- Free toolAgent Tool Count CalculatorHow often an agent picks the wrong tool as its library grows, from published benchmarks: every tool in the prompt, embedding or BM25 shortlists, with the measured range marked.
- Free toolEU AI Act Risk CheckerAnswer a few questions to see your EU AI Act risk tier, the obligations with their articles, and the deadlines after the 2026 Digital Omnibus.
