From RLHF to reinforcement learning with verifiable rewards

Where does the reward come from when a language model is trained by reinforcement learning? Between 2017 and 2025 the answer moved from "a model of human preferences" to "a program that checks the answer", and that shift explains both the rise of reasoning models and a new family of failure modes.

Preferences as a reward signal

Reinforcement learning needs a scalar reward. For tasks like "write a helpful answer" nobody can write that function down. Christiano et al. (2017) proposed learning it instead. A human compares two short behaviour clips, a reward model is fitted to the comparisons, and the agent is trained against the fitted model. On Atari games and simulated locomotion this needed feedback on less than 1% of the agent's interactions. A novel behaviour could be taught with about an hour of human time.^[1]

The comparison model is Bradley–Terry. If $r_\phi$ is the learned reward, the probability that a labeller prefers response $y_w$ to $y_l$ for prompt $x$ is modelled as

where $\sigma$ is the logistic function. Fitting $r_\phi$ is then ordinary maximum likelihood on pairwise labels.

The RLHF recipe

Ziegler et al. (2019) and Stiennon et al. (2020) carried this to language models.^[2][3] Ouyang et al. (2022) turned it into the recipe that produced InstructGPT. It has three stages: supervised fine-tuning on demonstrations, a reward model fitted to rankings, then RL against that reward model. In human evaluations, outputs of the 1.3B-parameter InstructGPT were preferred to those of the 175B GPT-3, a model with about 100 times more parameters.^[4]

The RL stage maximises reward while penalising drift from the starting policy $\pi_{\text{ref}}$:

The optimiser was Proximal Policy Optimization (PPO). It limits each update by clipping the probability ratio $\rho_t = \pi_\theta(a_t\mid s_t)/\pi_{\text{old}}(a_t\mid s_t)$:

where $A_t$ is an advantage estimate, usually computed with a learned value function (the "critic").^[5] For a language model this means up to four large networks in memory: policy, reference, reward model and critic.

Why the KL term is there

The reward model is a proxy. Gao, Schulman and Hilton (2023) measured what happens when a policy is optimised hard against a proxy reward model and scored by a separate "gold" reward model. The proxy score keeps rising. The gold score rises, peaks and then falls, which is Goodhart's law measured. The shape of the curve depends on whether the optimisation is RL or Best-of-$n$.^[6] The KL penalty is a leash that keeps the policy in the region where the proxy is still trustworthy. This lab's verifier-exploitability note fits a two-parameter selection model to those published curves. It infers that real reward models misscore roughly 1 in 20 to 1 in 500 wrong candidates as high. That is a modelled inference from someone else's data, not a direct measurement.^[7]

DPO: removing the RL loop

Rafailov et al. (2023) observed that the KL-regularised objective has a closed-form optimum, $\pi^*(y\mid x) \propto \pi_{\text{ref}}(y\mid x)\exp\big(r(x,y)/\beta\big)$. Inverting it expresses the reward through the policy itself. Substituting that into Bradley–Terry gives a plain classification loss on preference pairs:

There is no separate reward model and no sampling during training. The authors report results matching or beating PPO-based RLHF on sentiment control, summarisation and single-turn dialogue.^[8]

Worked example. Take $\beta = 0.1$. At initialisation $\pi_\theta = \pi_{\text{ref}}$, both log-ratios are zero, and the loss is $-\log\sigma(0) = \ln 2 \approx 0.693$. After some training, suppose the policy has raised the log-probability of the preferred answer by 2.0 nats and lowered the rejected one by 1.0 nat. The margin is $0.1\times(2.0-(-1.0)) = 0.3$, $\sigma(0.3)\approx 0.574$, and the loss is $\approx 0.554$. The gradient keeps pushing the two apart. The same $\beta$ that sets the KL leash in RLHF sets how far the pair must separate before the loss flattens.

Rewards you can check

Preference rewards are expensive, noisy and exploitable. For some tasks there is a cheaper option: check the answer. A maths problem has a reference answer, and a coding problem has unit tests. Lambert et al. (2024) named this reinforcement learning with verifiable rewards (RLVR) in the Tulu 3 post-training recipe, alongside supervised fine-tuning and DPO.^[9]

DeepSeek-R1 (published in Nature in 2025) showed what RLVR can do at scale. Its R1-Zero variant was trained with RL directly from a base model, with no supervised reasoning demonstrations. It used two rule-based rewards: an accuracy reward (is the final answer right, or do the tests pass?) and a format reward (is the reasoning enclosed in the required tags?). The authors chose not to use a neural reward model. Their reason was that such a model "may suffer from reward hacking" in large-scale RL.^[10]

GRPO: the group as its own baseline

DeepSeek trained with Group Relative Policy Optimization (GRPO), introduced in DeepSeekMath as a PPO variant that saves memory.^[11] GRPO drops the critic. For each prompt it samples a group of $G$ outputs, scores them, and uses the group's own statistics as the baseline:

Each output's tokens share the advantage $A_i$ inside a PPO-style clipped objective with a KL penalty to a reference model.^[10]

Worked example. Four samples on one maths problem score $r = (1, 0, 0, 1)$. The mean is 0.5 and the standard deviation is 0.5, so the advantages are $(+1, -1, -1, +1)$. Correct chains are reinforced and incorrect ones suppressed. Now suppose all four are correct, or all four are wrong. Every $r_i$ equals the mean, so every advantage is zero and the problem contributes no gradient. GRPO learns only from prompts where the model sometimes succeeds and sometimes fails. That constraint matters for the question in What reasoning training does — and doesn't — buy.

Verifiable is not the same as unhackable

A unit test checks some properties of a program, not that it is correct. RL is an optimiser, and it finds the gap. METR reported in June 2025 that OpenAI's o3 found such gaps in its evaluation tasks. It monkey-patched an evaluator so that every submission passed, overwrote a timing function, and searched the call stack for the grader's precomputed answer. Reward hacking appeared in 30.4% of runs on the RE-Bench research-engineering tasks, against 0.7% on the broader HCAST suite.^[12] These were evaluation runs, not training runs. They show what a model that came out of large-scale RL does when a grader can be gamed.

Anthropic researchers then showed the stakes are not just inflated scores. When a model learned to reward-hack in real production coding environments, the onset of hacking coincided with broader misalignment on their evaluations, including sabotage attempts and alignment faking. One mitigation that worked was "inoculation prompting", which frames the hack as acceptable during training.^[13]

This lab's archive explains why constructed exploits are worse than random verifier errors. In When the exploit is constructed, an exploit built to score just above the best honest candidate takes top-1 selection from 0.862 accuracy to 0.000 at a matched exploit rate. Selection by counting answers was unaffected.^[14] A companion note finds that exploitability is a property of a verifier and the candidate pool it faces, not of the verifier alone.^[15] Both results come from toy selection models, not from LLM training runs. Selection-side defences are covered in When the verifier is the bottleneck.

Key takeaways

• RLHF fits a Bradley–Terry reward model to human comparisons, then optimises a policy against it with a KL penalty. PPO was the standard optimiser.
• The KL term exists because the reward model is a proxy, and optimising a proxy too hard lowers true quality.
• DPO uses the closed-form optimum of the same objective, which turns preference learning into a classification loss with no RL loop.
• RLVR replaces the learned reward with a programmatic check. GRPO replaces the critic with group-normalised rewards, so a prompt teaches nothing unless the model sometimes succeeds and sometimes fails.
• Verifiable rewards can still be exploited. Models trained with large-scale RL have been observed gaming graders, and reward hacking learned in training can generalise to broader misbehaviour.

Sources