NeuroGenesysIs your verifier exploitable?
A verifier with good accuracy can still get worse the more candidates you give it, if some wrong answers score high. That rate is its exploitability ε. This tool measures ε from your own Best-of-N log instead of assuming a score model. It reports how often a wrong answer outscores every right one, the measured accuracy curve and its peak N*, and whether ε sits below the threshold ε* = p/(1−p) where consensus voting still protects you.
Exploitability belongs to a verifier and the pool it faces, so ε is broken out by pool. A verifier must be audited on problems it was not trained on, so problems marked as seen are compared with unseen ones. Background: When more sampling hurts and Exploitability belongs to the verifier and its pool. To model a budget from (p, d′, ε) instead of measuring it, use verifier-budget.
Audit · a Best-of-N log
One JSON object per scored candidate: problem, score and correct, plus optional pool (the prefilter or pool the verifier faced) and seen (the verifier was trained or tuned on this problem). Nothing leaves your browser.
Run it locally (Python, standard library only)
pip install selrel # or the tarball on the selection-reliability page selrel simulate --mode bon --out bon.jsonl selrel audit bon.jsonl # verdict, ε, ε*, measured N* selrel audit bon.jsonl --json
- README.mdlog format and limits
- selrel/audit.py
