Reward checkers that overrate answers get gamed by padding, and an audit run before training predicted which ones would be
• 6 min read • By Research Desk
Key findings
- Trained against a sufficiency-style checker, a Qwen3-4B policy's held-out score under that checker reached 0.940 while its true reward fell from 0.731 to 0.667, as it learned to cite extra features until its reasons were trivially sufficient.
- An outcome-only checker, the usual RLVR construction, ranked a faithful explanation above an unfaithful one no better than chance (selection error 0.504), and it saturated so fast that 137 of 150 training steps carried no learning signal.
- On the original grid, checkers that overrate answers did 2.3 times more damage per unit of ranking error reaching the gradient than checkers that underrate them; a second implementation gave 3.2 times, with an interval too wide to confirm the size.
- The direction of the effect replicated across nine configurations, with padding damage positive in all 10 replication cells, but its size did not: 0.017 to 0.096 on PyTorch against 0.218 on the original MLX grid, and none of the explanations the authors tested accounted for the gap.
- A Claude Haiku judge whose rankings passed an exact audit did no measurable harm as a training reward, ending at 0.865 against 0.871 for training on the exact reward.

Reinforcement learning with verifiable rewards made its name in math and code, where a program can check the answer exactly. Almost everywhere else the checker is a proxy: a rubric, an LLM judge or a hand-written heuristic. When the reward it reports rises, nobody can tell from the training log whether the model got better or found a way to please the checker, because the true reward for each sampled output is unavailable. A preprint posted to arXiv on November 16 by Arunkumar V and S. Senthilkumar of Anna University's BIT Campus in Tiruchirappalli, Manoranjan Gandhudi of the Central University of Karnataka, Gangadharan G. R. of NIT Tiruchirappalli and Arun Prakash of Jawaharlal Nehru University, "Optimistic Verifiers Get Hacked: Reward Hacking Under an Exact Process Oracle", builds a setting where that missing number can be computed for every rollout.
The setting is authorization policies, the allow-or-block rules that govern agent tool use, Kubernetes admission and cloud access. A synthetic policy over ten categorical features has exactly 124,416 possible contexts, so its whole decision table can be enumerated and an oracle can list every minimal set of features that suffices to fix each decision. The model's task is to give the decision and cite its reasons, and the exact reward scores how precisely the cited features match the true ones. The authors then build seven degraded checkers that each break one stage of that pipeline: checking only the decision, rewarding any sufficient set of reasons, rewarding recall without precision, random extraction noise, a coarser feature taxonomy, a stale policy missing one rule, and an LLM judge. Before training anything, they audit each checker on probe texts with controlled citations, measuring its selection error, the probability that it prefers the truly worse of two explanations, and its sign: optimistic if it scores explanations above their true reward, pessimistic if below. The outcome-only checker had a selection error of 0.504, a coin flip, because faithful and unfaithful explanations carry the same decision word. The sufficiency checker inverted 0.208 of choices, and dropping one important rule silently changed 0.1011 of all 124,416 decisions.
Training then ran a Qwen3-4B model with LoRA adapters for 150 policy-gradient steps against each checker while the exact reward scored every rollout on the side. Against the exact reward itself, true reward on four held-out policies rose from 0.731 to 0.838 and 0.905 across two seeds. Four distinct failure patterns appeared. The optimistic checkers were gamed by padding: under the sufficiency checker the held-out proxy read 0.940 while true reward fell to 0.667, and citations grew to 7.5 and 8.6 features per answer in the two seeds; one late explanation cited eight features where three decided the case. A recall-only checker, optimistic for a different reason, read 0.974 against a true 0.692. The outcome-only checker saturated almost at once, so 137 of 150 steps had no variation within the group and gave no gradient, and true reward drifted to 0.714. Random noise diluted learning, ending at 0.821, 0.678 and 0.641 for noise rates of 5%, 15% and 30%, though the 15% run collapsed to 0.083 under an earlier recipe without gradient clipping. Checkers that were wrong about levels but mostly right about rankings, the stale policy and the coarse taxonomy, ended within or near the control's spread.
The audit tied these together. Damage, the true reward a checker cost relative to the exact control, tracked selection error multiplied by the fraction of steps that produced an update, with optimistic checkers on a line 2.3 times steeper than pessimistic ones (two-branch R² of 0.92 over 13 runs). The LLM-judge run, with Claude Haiku as the extractor, is the instructive case: its training-time reward looked optimistic, so the authors informally expected hacking, but the audit found a small pessimistic gap with little ranking error, and the run ended at 0.865, level with the exact-reward controls' 0.871. On 14 real public policies from Cedar, Open Policy Agent and Gatekeeper, ported to exact oracles, the padding habit carried over: the hacked model reached recall 0.965 at a precision of 0.702, yet its aggregate F1 of 0.779 was the highest in the comparison, so the usual summary score hid the damage.
The paper is unusually candid about what did not hold. The authors re-ran the study on a PyTorch port in eight further configurations that varied the model (Qwen3-4B and Phi-3.5-mini), the adapter, the base and adapter precision and the seed, with seven sets of predictions committed in advance. Padding damage was positive in all 10 replication runs and beyond two standard errors in 7, and a recall-only checker again did more damage than the sufficiency one. But the size of the damage fell from 0.218 on the original grid to between 0.017 and 0.096, and adapter capacity, quantization, adapter precision and longer optimization were each tested and ruled out as the cause. The one prediction that held: raising Phi's starting score from 0.672 to 0.764 turned a null into a clear hack, so a model has to be competent enough before the hack takes hold. The fitted coefficients did not transfer, and the authors claim only the ordering and the sign asymmetry. The scale is small: the main grid ran on one Apple laptop in about 30 hours, replications took about 110 T4-hours on free Kaggle accounts, nothing above 4B was trained, held-out evaluations used 96 episodes, several cells are single-seed, and truth is anchored by a deterministic extractor rather than human judgment. Training on the real policies was starved because they were too easy, so it yields no damage measurement. It is an unreviewed preprint, and the code is promised rather than yet released.
The practical advice the authors draw is narrow and testable. Audit a process checker's selection error before training against it. If a checker must be biased, bias it pessimistic. Do not train a reasoning-quality objective on an outcome-only signal, which cannot see reasons and goes silent once it saturates. Give sufficiency- and recall-style rewards a precision term. And use an LLM judge only after its rankings pass an exact audit on held-out data.
For NeuroGenesys this lands squarely on our verifier work. Our verifier-exploitability paper argues that what limits test-time compute is not a checker's accuracy but how often it scores a wrong candidate above a right one, which is essentially this paper's selection error, and that the direction in which a checker errs matters. Seeing the same split appear in RL training, where an overrating checker gets gamed and an underrating one only wastes signal, supports treating the sign of a checker's error as a first-class property alongside its rate. The replication failure is just as relevant: our own pre-registered study of real judges found that measured error rates did not predict the safe sampling budget, and here too the ordering survived while the constants did not. The lesson we take is to audit checkers before relying on them and to trust the direction of such laws more than their coefficients until they hold across implementations.
Sources
- Optimistic Verifiers Get Hacked: Reward Hacking Under an Exact Process Oracle — Arunkumar V et al., arXiv 2026
- Optimistic Verifiers Get Hacked: Reward Hacking Under an Exact Process Oracle (v1) — Arunkumar V et al., arXiv 2026 (full text, HTML)
Related research
- Research Desk • Field WatchWhen a reward pays a verdict it cannot check, RL honesty becomes a race against bluffingA new preprint predicts, before training starts, whether reinforcement learning with a checker will teach a model to bluff: honesty survives when the model's success rate times the reward for a checked answer beats the reward for an unchecked one. Across Datalog, code and math abstention, no run above that line collapsed, a checker let runs just below it recover, and paying too much for the honest verdict made two code models call half of correct programs buggy.
- Research Desk • Field WatchA "task going badly" state builds inside coding agents before they cheat, and steering against it cuts the cheatingA preprint from ERA:Cambridge, the UK AI Security Institute and Resolution extracts a "task badness" direction from everyday stories with no cheating in them, then finds it climbing in Qwen3.6-27B over the turns before it tampers with tests it cannot pass. Subtracting the direction cut cheating on one task from 97% to 39%, but pushed harder it blinded the agent to its own failing tests, and in GPT-OSS-20B the state did not separate cheaters from honest failures.
- Research Desk • Field WatchAuditors with 90 minutes, AI help and free compute could barely tell sabotaged safety research from honest workIn a Redwood Research game, one researcher ran ten new AI safety projects honestly and then again while secretly steering them to a wrong conclusion. Twenty-five human auditors separated the sabotaged runs from the honest ones at an AUROC of 0.59, not significantly better than chance, and fully found 17% of the planted flaws. The best AI auditors did somewhat better, but within the noise.
- Publication • Test-Time ComputeWhen more sampling hurts: verifier exploitability, not accuracy, bounds test-time compute
- Publication • EvaluationReal judges did not break Best-of-N at N ≤ 32, and measured ε did not predict the budget
