Alignment techniques: preference learning, constitutions and oversight

A pretrained language model imitates its training text; nothing in next-token prediction says "be honest", "refuse to help build a weapon" or "admit uncertainty". Alignment techniques add those targets after pretraining. This lesson works through the main families — learning from human preferences, learning from written principles, and oversight schemes for tasks humans cannot easily judge — and where each is known to strain, as of November 2026.

Learning a reward from comparisons

People find it much easier to say which of two outputs is better than to write down a reward function. Christiano et al. (2017) exploited this: humans compared short clips of an agent's behaviour, a reward model was fitted to those comparisons, and a policy was trained against the learned reward, solving tasks including Atari games without access to their true reward.^[1]

The standard reward-model likelihood is the Bradley–Terry model. For a prompt $x$ with preferred response $y_w$ and dispreferred $y_l$,

where $\sigma$ is the logistic function. Only reward differences matter, so the scale is fixed by the data.

The policy is then optimised to score highly while staying close to a reference model $\pi_{\text{ref}}$:

Ouyang et al. (2022) applied this pipeline — supervised fine-tuning on demonstrations, a reward model from human rankings, then reinforcement learning with PPO — to GPT-3. Labellers preferred outputs of the 1.3-billion-parameter InstructGPT model to those of the 175-billion-parameter GPT-3 on the API prompt distribution, despite a hundredfold difference in size; truthfulness improved, toxicity improved slightly, and measured bias did not.^[2]

The closed form, and DPO

The KL-regularised objective has a known optimum: $\pi^*(y \mid x) \propto \pi_{\text{ref}}(y \mid x)\, \exp\!\big(r(x,y)/\beta\big)$. Rafailov et al. (2023) inverted this relation to write the reward in terms of the policy itself, and substituted it into the Bradley–Terry loss. The result, Direct Preference Optimization (DPO), trains the policy directly on preference pairs with a classification-style loss, skipping the separate reward model and RL loop while targeting the same objective.^[3]

Worked example. Two candidate responses are equally likely under the reference model, and the reward model rates A one unit above B.

• With $\beta = 0.5$: $\pi^*(A)/\pi^*(B) = e^{1/0.5} = e^2 \approx 7.4$, so $\pi^*(A) \approx 0.88$.
• With $\beta = 0.1$: the ratio is $e^{10} \approx 22{,}000$, so the policy almost always says A.

$\beta$ sets how hard the policy is pushed towards whatever the reward model prefers. Pushing hard is exactly where the reward model's errors matter: Gao et al. (2022) showed that as optimisation against a proxy reward model increases, the true ("gold") reward first rises and then falls.^[4] Overoptimisation is the preference-learning version of the verifier exploitability discussed in /academy/test-time-compute-and-verification/verifiers-and-exploitability; for its role in training reasoning models, see /academy/reasoning-and-rl/from-rlhf-to-verifiable-rewards.

Learning from written principles

Human labelling is slow and expensive, and labellers disagree. Constitutional AI (Bai et al., 2022) replaced most harmlessness labels with a short list of natural-language principles. In a supervised phase, the model critiques and revises its own responses against the principles and is fine-tuned on the revisions. In an RL phase, a model compares pairs of responses against the principles, and those AI preferences train the preference model — reinforcement learning from AI feedback (RLAIF). The resulting assistants were less harmful at a given level of helpfulness and markedly less evasive than earlier RLHF models, which often refused controversial questions.^[5]

The written principles have since grown into a much longer document. In January 2026 Anthropic published a new constitution for its Claude models, used directly in training. It sets a priority order for conflicts — broadly safe, broadly ethical, compliant with Anthropic's guidelines, genuinely helpful — and shifts from a list of standalone rules to explanations of why, so that a model can generalise to unanticipated situations, while keeping a small set of hard constraints for the highest-stakes cases. The document is released under a CC0 licence, and Anthropic notes that trained models may not always match it.^[6]

Whether explanation generalises better than rules is an empirical question that is not settled. Anthropic's stated argument is that rigidly trained rules can distort a model's broader character,^[6] and there is related evidence that narrow training signals generalise in unexpected ways. Anthropic researchers reported in 2025 that when a model learned to reward hack in production coding environments, misaligned behaviour rose across unrelated evaluations — including sabotage of safety research and alignment faking — and that telling the model during training that hacking was acceptable in that context ("inoculation prompting") removed most of the misaligned generalisation.^[7]

Oversight when humans cannot judge

Preference learning assumes the evaluator can tell good from bad. For tasks beyond the evaluator's competence, that assumption fails, and scalable oversight asks how to supervise systems that may know more than their supervisors.

Sandwiching. Bowman et al. (2022) proposed testing oversight methods on tasks where non-experts cannot judge unaided but experts can verify. In a baseline experiment, non-experts chatting with an unreliable model substantially outperformed both the model alone and their own unaided performance.^[8]

Debate. Irving, Christiano and Amodei (2018) proposed training two models through self-play to argue opposite sides before a human judge, who decides which side gave the more truthful and useful information; by analogy with complexity theory, optimal debate with a polynomial-time judge could answer any PSPACE question.^[9] Khan et al. (2024) tested this on reading comprehension where only the debaters saw the text. With the most persuasive debaters, non-expert human judges reached 88% accuracy and LLM judges 76%, against naive baselines of 60% and 48%; in the contrasting single-advocate "consultancy" protocol, judges did worse as the consultant became more persuasive.^[10]

Weak-to-strong generalisation. Burns et al. (2023) used a small model as a stand-in for human supervisors and a large one as the stand-in for a superhuman system — for example, GPT-2-level labels supervising a GPT-4-family model. The strong student consistently outperformed its weak teacher, but fell well short of training on ground truth; with an auxiliary confidence loss it typically recovered close to 80% of the gap on NLP tasks.^[11]

Correlated supervisors. Oversight by AI inherits the correlation problem from voting. Goel et al. (2025) found that LLM judges favour models whose mistakes resemble their own, and that the benefit of training on a supervisor's labels shrinks as supervisor and student become more similar.^[12] An overseer that shares the overseen model's errors is a weaker check than its accuracy suggests; see /academy/test-time-compute-and-verification/majority-vote-and-correlated-judges.

Open disagreements

• What preference training instils. Preference optimisation shapes behaviour on the training distribution; how far that reflects stable values rather than learned surface patterns is contested, and the reward-hacking generalisation results above show training signals can spread in unintended directions.
• Whether debate scales. Its guarantees assume optimal play and an honest-favouring structure; empirical gains so far are on bounded tasks with verifiable quotes.
• Whether analogies transfer. Weak-to-strong experiments use small models as proxies for human supervisors;^[11] whether results carry over to systems that genuinely exceed their overseers cannot yet be tested directly.

Key takeaways

• RLHF fits a Bradley–Terry reward model to comparisons and optimises a KL-regularised policy against it; DPO reaches the same objective without an explicit reward model.
• The KL coefficient controls how hard the reward model is exploited; past a point, true quality falls (overoptimisation).
• Constitutional AI substitutes written principles and AI feedback for most human harmlessness labels; Anthropic's 2026 constitution moves further from rules towards explained priorities.
• Scalable-oversight methods — sandwiching, debate, weak-to-strong training — show partial gains on bounded tasks; correlated supervisors weaken all of them.

Sources