When a reward pays a verdict it cannot check, RL honesty becomes a race against bluffing
• 5 min read • By Research Desk
Key findings
- Across seven conditions on propositional Datalog, two code models and math abstention, no GRPO run with p·r/b above 1, where p is the model's success rate before RL, ended collapsed into bluffing.
- Just below the threshold, at p·r/b = 0.95, all four seeds trained under a checker recovered to 68–85% honest after 160 steps, while every matched run under a lottery with the same expected payoff collapsed.
- In nine matched pairs below the threshold, five checked runs survived and all nine lottery runs collapsed (one-sided Fisher exact test, 0.015).
- Paying the honest verdict too much backfired: at p·r/b = 3, two small code models called only 45% and 52% of correct programs correct after 100 steps, against 80% and 74% before RL.
- Training generously first and then cutting the payoff kept a depth-3 Datalog model at least 67% honest in three seeds, where the same model trained at the lower payoff from the start fell to 4–15%.

Reinforcement learning from verifiable rewards works because a program can grade the answer. But many tasks have one verdict a program can confirm and one it cannot. A test can show that a function is buggy by finding an input on which it fails; no test can show that it is correct. A reward that pays "CORRECT" without proof pays it on every buggy program the tests miss, and the model may learn to say it. In Honesty Is a Race, posted to arXiv on November 11, Nathaniel Bottman (Incubilate) and Alexei Oblomkov (University of Massachusetts Amherst) ask when GRPO training learns that bluff, and whether it can be predicted before training starts. Their reward pays b for the unchecked verdict and r for a certificate the checker accepts: a counterexample that exposes a bug, or, on propositional Datalog, a closure showing a goal cannot be derived.
The baseline prediction fits in one line. In a GRPO group that contains both a bluff and an attempt, the bluff pays b for certain and the attempt pays r with probability p, so the update favors honesty exactly when p·r/b exceeds 1. For abstention rewards this is Chow's 1970 rule for answering versus abstaining. The authors tested it in seven conditions: Qwen2.5-3B-Instruct on Datalog with proofs two and three levels deep, each before and after a "sharpening" phase that raises p; Qwen2.5-Coder-3B-Instruct and Phi-3.5-mini-instruct judging MBPP programs and their single-edit mutants; and Qwen2.5-1.5B-Instruct on grade-school math with an "I don't know" option. With no fitted constant, no run above 1 ended collapsed in any condition. Four of the six measured brackets contain 1, and two lie below it, at 0.80–0.95 for the depth-2 base and 0.78–0.94 for abstention.
Those two brackets below 1 are the race of the title. Under a checker, the model's attempts become valid more often as training goes on, so a run that starts just below the threshold can win if its success rate climbs past it before bluffs crowd the attempts out. The control is a lottery that pays the honest verdict at random, with the same expected payoff at the start, but nothing for getting better. At p·r/b = 0.95 on the depth-2 base, all four checked seeds fell and came back to 68–85% honest after 160 steps, and every lottery seed collapsed. At 0.95, of the checked runs' 12-point gain in valid attempts, the authors attribute about 11 points to improvement and about 1 to the model dropping instances it fails. A fitted model of the race, recorded before the depth-3 runs at 0.92, gave each checked seed a 0.04–0.13 chance of winning; none won, every lottery seed collapsed, and the checked seeds stayed above them from step 60 on.
A first-order analysis of a GRPO step explains two departures from the baseline. The bluff's verdict is the right answer on the other half of the data (correct programs, derivable goals), and updates there push the shared verdict the same way as the bluff. That pull, which the authors call κ and measure from gradient inner products, makes the condition p·r/b > 1 + κ. Since κ starts large and fades as the other class settles, runs just above 1 first dip toward bluffing: on code, runs at 1.25 fell nearly to zero and were still climbing back when training ended. The same coupling predicts a ceiling, placed by couplings measured before RL at about p·r/b = 2.15–2.2 on code, above which the push to say BUGGY spills onto correct programs. At 2, both code models still called correct programs correct about as often as before RL; at 3, only 45% and 52% after 100 steps, against 80% and 74% at the start. Past a point, paying more for the honest verdict made the models worse. A linear probe fit before RL and held fixed still separated derivable from underivable instances in all ten collapsed depth-2 models (AUROC 0.92–0.99), so the collapse is bluffing rather than lost knowledge.
The practical result is a schedule: train generously under the checker, then cut the payoff once the trained model's own success rate clears the threshold. On depth-3 Datalog, a model sharpened first stayed at least 67% honest for 160 steps at a ratio where the base model trained from the start fell to 4–15% and ended at 20–47%. The authors' advice for reward design is to keep p·r/b above 1 with a margin but below the ceiling, measure p on the model being trained, pay generously with the check before cutting, and keep the check.
The scope is small, and the paper says so. This is a preprint, not peer-reviewed. The models have 1.5B to 3.8B parameters and were trained for 30 to 200 GRPO steps, about 810 GPU-hours in all, on one synthetic task, one code task and grade-school math. The Datalog results come from one 3B model. Most bracket ends rest on a single seed, every code run has one seed, and the code threshold is bracketed only loosely, between 0.95 and 2.0. The step analysis is first order and replaces gradient inner products by class averages; with Adam's momentum included it still places the turn of honesty 4 to 8 steps early. The schedule was tested from one generous model at each depth. The analysis assumes mean-centered advantages; standard GRPO's division by the group's standard deviation moves the threshold. Whether any of this holds at larger scale or on open-ended tasks is untested, and the code and data are promised for a future version.
For NeuroGenesys the paper is a careful version of a lesson from our own selector abstention campaign. There, a penalty for wrong answers with no other-task data did nothing at mild thresholds and hurt at strict ones: an abstain output trained that way fired almost equally on its own task and on others, so it carried no information about competence. Bottman and Oblomkov give a mechanism for that kind of spillover and a number for where it starts, and their lottery control shows why the check matters: a reward with the same mean that does not track validity cannot let the honest answer improve. Their frozen probe also echoes our synthesis on confident errors, which found hidden-state probes among the best self-derived error signals. The open question for us is whether the threshold, measured on a model before training, can set abstention payoffs for the selector in an expert library before we train it, instead of sweeping them afterward. That is a direct test we can run on our existing harness.
Sources
- Honesty Is a Race: Predicting When Verifiable-Reward RL Teaches a Model to Bluff — Bottman and Oblomkov, arXiv 2026
- Honesty Is a Race (v1) — Bottman and Oblomkov, arXiv 2026 (full text, HTML)
Related research
- Research Desk • Machine LearningPenalising wrong answers only works if someone else is gradingExperts that never saw a task still vote on it with near-certainty. We ran ten pre-registered fixes. Penalties inside the expert failed; a selector that ignores expert confidence reached 0.926 on Split-MNIST.
- Research Desk • Field WatchInterpretability tools carry across models, once the map has seen the right promptsA University of Houston preprint carries a refusal direction, a correctness direction and a sentiment probe between 110 pairs of open models, from 1B to 72B parameters, through linear maps fit on unlabeled prompts. Fit on ordinary questions, a map recovers 54% of the target's own refusal steering. Add 16 unlabeled prompts per class from the tool's region and recovery reaches 102%.
- Publication • Continual LearningGiving experts a way to say "not mine": ten pre-registered selector and abstention experiments
- Publication • Test-Time ComputeWhen more sampling hurts: verifier exploitability, not accuracy, bounds test-time compute
- Publication • EvaluationStable wrong beliefs evade every self-check: a literature synthesis on model self-knowledge and confident errors
