NeuroGenesysNine judges, how many votes?
Critics that share a model, a prompt, or a training distribution share their mistakes. From two measured numbers — per-critic error and pairwise co-error — this tool recovers the shared error component, the accuracy ceiling no amount of same-substrate sampling crosses, and the panel's effective size. It then says whether your next unit of budget should buy more samples or a decorrelated channel.
Background: Measured self-critique correlation and the ε correction.
Panel · how much is your judge panel actually worth?
s = (P_both − e²)/(1 + P_both − 2e), ρ = (P_both − e²)/(e(1−e)), n_eff = N/(1 + (N−1)ρ). Classical mathematics — Ladha 1992, Clemen–Winkler 1985, Breiman 2001 — made actionable. At small N a better-but-correlated critic can still beat worse-but-independent ones.
From data · measure e and P_both from a miss matrix
Run your critics over items where the answer under review is wrong, then record one row per item and one column per critic: 1 if the critic missed the error (endorsed the wrong answer), 0 if it caught it, blank or NA if it abstained. A header row is optional. Error bars are a 1,000-draw bootstrap over items, as in the research code. Nothing leaves your browser.
Check-tail · can you trust a Φ-based ε?
The Gaussian ε law ε = Φ(−gap/(σ√2)) is Thurstone's 1927 Case V identity — exact for Gaussian scores and uniformly optimistic under heavy tails (measured 11× at gap/σ = 4 and 723× at gap/σ = 6). Paste verifier scores to test the precondition; excess kurtosis separates the regimes. Nothing leaves your browser.
Run it locally (Python, standard library only)
python verifier_panel.py panel --e 0.30 --pboth 0.15 --n 5 --target-acc 0.9 python verifier_panel.py check-tail scores.txt python verifier_panel.py check-tail --gap 1.0 --sigma 0.25
- verifier_panel.pyno dependencies
- README.md
