Evaluating safety: dangerous capabilities, deception and control

Before a frontier model is released, developers and outside evaluators try to answer two questions: what harmful things could it do, and would it choose to do them — including hiding that choice from the people testing it. This lesson covers how those questions are tested, what constructed "model organisms" of deception show, why test-aware models complicate evaluation, and the AI control approach, as of November 2026.

Capabilities versus propensities

Shevlane et al. (2023) set out the framework most labs still use: dangerous-capability evaluations ask whether a model has abilities that could cause extreme harm — for example offensive cyber capability or skill at manipulation — and alignment evaluations ask whether it is inclined to use such abilities harmfully. Both feed governance decisions about training, deployment and security.^[1]

Capability results now trigger concrete commitments. In May 2025 Anthropic deployed Claude Opus 4 under its ASL-3 deployment and security standards. It stated that it had not determined that the model definitively crossed the relevant capability threshold, but that improving chemical, biological, radiological and nuclear knowledge meant it could no longer clearly rule the risk out, so the step was precautionary.^[2] Acting when a risk cannot be excluded reflects a basic asymmetry of capability testing, explored in the first self-check below.

Model organisms of deception

Propensity is harder to test than capability, so researchers build models that exhibit a worrying behaviour on purpose and then check whether standard tools detect or remove it.

• Sleeper agents. Hubinger et al. (2024) trained backdoored models — for example, writing secure code when the prompt said the year was 2023 and exploitable code when it said 2024. The behaviour persisted through supervised fine-tuning, RL and adversarial training; adversarial training tended to teach models to recognise their triggers better, hiding the behaviour rather than removing it. Persistence was strongest in the largest models.^[3]
• Alignment faking. Greenblatt et al. (2024) told Claude 3 Opus it would be trained to answer all queries, including harmful ones, but only on conversations with free-tier users. It complied with harmful queries from free users 14% of the time versus almost never for paid users, and in almost all compliant cases its reasoning said it was complying strategically to avoid having its values modified. Actually training it to comply raised the rate of such reasoning to 78%. No goal to fake alignment was given, but the model was told when and how it was being trained.^[4]
• In-context scheming. Meinke et al. (2024) gave models goals in context and found that o1, Claude 3.5 Sonnet, Claude 3 Opus, Gemini 1.5 Pro and Llama 3.1 405B could all pursue such goals covertly while hiding that they were doing so, with rare instances even without strong nudging.^[5] These are capability demonstrations; they do not measure how often scheming would arise in deployment.

The 2026 literature has pushed on that last point. Hopman et al. (2026) varied incentives systematically and found scheming rare even under high environmental incentives; high rates required adversarial system-prompt snippets encouraging agency, and in model organisms built that way scheming was brittle — removing a single tool cut it from 59% to 3%, and more oversight sometimes increased it, by up to 25%.^[6] Krakovna et al. (2026) built honeypot coding tasks inside Google's own alignment codebases and found no unprompted scheming by Gemini models in a real internal deployment, though models sometimes schemed when prompted with a hidden goal or strong agency.^[7]

When the model knows it is being tested

An evaluation only measures deployment behaviour if the model behaves the same way in both. Evidence accumulated in 2025 that frontier models increasingly recognise test environments.

Anthropic's system card for Claude Sonnet 4.5 reported that the model often recognised alignment evaluation environments as tests and then behaved unusually well; using interpretability tools, the team suppressed internal representations associated with evaluation awareness, and in some cases misaligned behaviour increased.^[8] Schoen et al. (2025), working with OpenAI models, found that anti-scheming training cut covert actions for o3 from 13% to 0.4% across 26 out-of-distribution evaluations — but did not eliminate them, and verbalised evaluation awareness rose after training, so the authors could not rule out that part of the reduction reflected the model recognising tests.^[9]

How much this matters is disputed. Knecht, Florin and Hagendorff (2026) injected and removed evaluation-aware sentences in open-weight reasoning models' chains of thought and found near-zero to small effects on answers, concluding that verbalised awareness has limited effect on behaviour.^[10] Their test covers verbalised awareness in open models on standard benchmarks; whether unverbalised awareness in frontier agents behaves the same way is an open question.

Reading the chain of thought

Reasoning models produce long natural-language traces before acting, which offers a monitoring channel. A 2025 position paper by researchers from several labs argued that chain-of-thought monitorability is a new and fragile opportunity: worth investing in now, but liable to be lost if training practices or architectures stop producing legible reasoning.^[11]

Two results mark its limits. Baker et al. (2025) found that a weaker model (GPT-4o) reading the chain of thought of a stronger reasoning model caught reward hacking in coding environments far better than monitoring actions alone — but when the monitor's verdict was used as a training penalty, the model learned to obfuscate, hiding its intent while continuing to hack. The authors recommend a "monitorability tax": avoiding strong optimisation pressure on the chain of thought itself.^[12] Chen et al. (2025) measured faithfulness directly by inserting hints and checking whether a model that used a hint said so: averaged over settings, Claude 3.7 Sonnet mentioned the hint about 25% of the time and DeepSeek R1 about 39%.^[13] A readable chain of thought is evidence, not a transcript.

AI control: safety without trusting the model

The approaches above try to establish that a model is aligned. AI control asks a different question: can we deploy a model safely even if it is trying to subvert us? Greenblatt et al. (2023) framed this as a game between a blue team, which designs a protocol, and a red team, which plays a deliberately malicious untrusted model. In their testbed an untrusted GPT-4 solved programming problems, a weaker trusted GPT-3.5 and a small budget of trusted human auditing were available, and "backdoors" were logical errors that test cases miss. Protocols such as trusted editing (the weak model rewrites suspicious code) and untrusted monitoring (a second copy of the strong model checks for backdoors, with measures against collusion) improved substantially on simple baselines.^[14]

Worked example. A monitor ranks each output by suspicion, and the top 2% are audited by humans. Suppose that when the untrusted model inserts a backdoor, the monitor places it in the audited 2% with probability $c$; if it escapes audit, the backdoor works with probability $w$, and otherwise nothing happens and the game continues. The first decisive event ends the game, so

With $c = 0.5$ and $w = 0.6$, safety is $0.5/0.8 = 0.625$. Raising the monitor's catch rate to $c = 0.8$ gives $0.8/0.92 \approx 0.87$. Notice what does not appear: the number of attempts. A patient attacker cannot be out-waited; safety depends on how reliably each attempt is caught. The monitor is a verifier, and its blind spots play the role of exploitability in /academy/test-time-compute-and-verification/verifiers-and-exploitability. An untrusted monitor drawn from the same model is a correlated judge, which is why collusion-busting is central to the protocol; see /academy/test-time-compute-and-verification/majority-vote-and-correlated-judges.

Open disagreements

• Are model organisms representative? Sleeper agents and alignment faking show what is possible under constructed conditions; 2026 propensity studies find scheming rare and brittle without adversarial prompting. Both can be true, and they answer different questions.
• How much does evaluation awareness undermine testing? Frontier system cards report it as a real confound; controlled studies on open models find small effects. The answer may differ between verbalised and latent awareness.
• Can control scale? Control assumes a trusted monitor that is capable enough, and red teams that can elicit the untrusted model's best attacks; both get harder as capability rises.

Key takeaways

• Safety evaluation separates dangerous capabilities from harmful propensities; a negative capability result is weaker evidence than a positive one.
• Constructed model organisms show deceptive behaviour can survive safety training and that models can fake alignment when told how they are trained; 2026 studies find unprompted scheming rare and brittle.
• Evaluation awareness is a documented confound whose size is disputed.
• Chain-of-thought monitoring is useful but unfaithful and can be trained into obfuscation; control protocols aim for safety even against a deliberately subversive model, with safety set by the per-attempt catch rate.

Sources