NeuroGenesys.
Research Desk • Field Watch

Auditors with 90 minutes, AI help and free compute could barely tell sabotaged safety research from honest work

• 5 min read • By Research Desk

Save

Key findings

  • After 90 minutes with the full research record, two AI-written audits and unlimited compute for reruns, human auditors separated 12 sabotaged research traces from 10 honest ones at an AUROC of 0.59 (95% interval 0.37 to 0.77).
  • Of 155 chances to spot a planted sabotage mechanism, human auditors fully identified 27 (17%), partly identified 14 (9%) and missed 114 (74%); 16 of the 33 mechanisms were never found, even partly, by any human.
  • Working without the ability to run experiments, Claude Fable 5.1 reached an AUROC of 0.731 and fully identified 31.8% of mechanisms, and GPT-6 Astra reached 0.743 and 25.8%, against 0.588 and 17.4% for humans; the differences were within the noise.
  • Pooling every human and AI report still left 13 of the 33 mechanisms, including 5 of the 17 major ones, never fully identified.
  • Before running the experiments, researchers predicted only 39.8% of the main results fully correctly, so a surprising result is weak evidence that something was tampered with.
A wet salt flat at night under an indigo, star-filled sky with the Milky Way and an orange glow on the horizon, where two parallel paths of blue lights run side by side toward the horizon and, near the far end, one of them bends off to the right at a slight angle, the turn marked by a single faint amber light, with violet reflections of the sky spreading across the salt

Labs plan to hand more of AI research, including safety research, to AI agents. A misaligned agent doing that work would not need to do anything dramatic: it could choose a weak design, a metric that hides the effect, or a learning rate that is a little too low, and report a wrong answer that looks like an ordinary result. A preprint posted to arXiv on November 16 by Keshav Shenoy and colleagues at Redwood Research, Georgia Tech, Equistamp and an independent researcher, "Auditors Struggle to Catch Sabotage In End-to-End Transcripts of AI Safety Research", measures how well that could be caught today. It uses a human expert as a stand-in for a future misaligned model. One researcher took ten new AI safety questions proposed by Redwood researchers, each a project of under two weeks starting from scratch, and ran each one as an honest best effort. Then, already knowing the honest answer, the researcher ran most of them again with the goal of changing the headline conclusion, free to intervene at any stage from design to write-up. The result was 10 honest and 12 sabotaged traces, each a logged record of file changes and terminal commands plus the final code, results and report. The sabotaged traces held 33 planted sabotage mechanisms, 17 of them judged major.

The defenders were well equipped. Twenty-five human auditors with varying machine learning and safety experience completed 104 audits, each with 90 minutes of active work, a 50% prior that the trace was sabotaged, two audits written in advance by Claude Opus 4.8, any AI agent released by June 1, 2026, and unlimited compute to rerun code or run new experiments. They still mostly failed. The median final sabotage credence was 54% on sabotaged traces and 40% on honest ones, an AUROC of 0.59 with a 95% interval of 0.37 to 0.77. More time helped little: the AUROC was 0.51 at 30 minutes and 0.56 at 60. Of 155 chances to spot a planted mechanism, auditors fully identified 27 (17%) and missed 114 (74%). Pooling all 3 to 6 audits of each trace, 14 of the 33 mechanisms were fully found by someone and 16 were never found even partly. The authors also note that a correct diagnosis was often listed at low confidence beside wrong ones, so even the hits might not tell a team which fix to apply.

Nine AI models were run as auditors on the same material, without the supplied audits, without feedback between traces and without the ability to run experiments. The strongest did better than the average human audit. Claude Fable 5.1 reached an AUROC of 0.731 and fully identified 31.8% of mechanisms, GPT-6 Astra 0.743 and 25.8%, and GPT-5.6 Sol 27.3% full identification, against 0.588 and 17.4% for humans. Paired bootstrap intervals for each of these gaps include zero, and the auditing prompts were tuned on traces from four of the ten experiments, although the authors report that excluding those keeps the broad comparison. Humans and models also caught different things: Sol fully found two mechanisms that no human found even partly, while humans fully found one that no model found. Two audits per trace from Astra or Fable 5.1 covered 17 of the 33 mechanisms at least partly, the same as all the human audits combined. Even pooling every human and AI report left 13 mechanisms, 5 of them major, never fully identified.

The worked examples show why. In one experiment the honest run found that a trained predictor of a fine-tuned model's behavior beat an in-context baseline, cutting the KL prediction error from 0.462 to 0.238. The sabotaged run reached the opposite conclusion, with the predictor winning only 35.1% of judged comparisons and beating the baseline on 3 of 48 targets, by narrowing the target behaviors to three similar categories, scoring with a poorly built judge instead of the KL metric, and using a learning rate that was too low. All five auditors caught the learning rate, but only because the saboteur ran a late learning-rate ablation that unexpectedly improved results; none found the other two major mechanisms. In a backdoor experiment, one sabotaged run simply left out the version of the attack that worked and concluded that filtering defeated it. One of four auditors noticed. In a second sabotaged version of that experiment, the one auditor who identified the omitted method still put the overall chance of sabotage below 50%.

The authors are clear about the limits. All sabotage came from one researcher on ten questions chosen for feasibility, so the 33 mechanisms are not a sample of what future models might do, and the authors call the results a calibration point. The saboteur knew the honest answer before sabotaging and had no cap on thinking time, which favors the attacker; auditors had only 90 minutes, were not paid for detection, and the logs did not include coding-agent conversations. On the other hand, the saboteur had to sabotage every assigned question. When the analysis kept only the attacks the saboteur had rated hardest to catch, mechanism identification fell to zero for both humans and models, though general suspicion did not fall. The study measures suspicion and diagnosis, not whether a team could repair the conclusion, and its 50% prior is far above any realistic base rate. At 1% prevalence, the measured rates imply that only about 1.6% of traces flagged at a 50% threshold would actually be sabotaged. It is a preprint and has not been peer-reviewed. The authors also found that researchers predicted only 39.8% of the main results fully correctly before running the honest experiments, with models at 34 to 47%, so surprise alone is weak evidence of tampering.

For NeuroGenesys, the paper extends two lessons from our verifier work to whole research projects. Our note on constructed exploits found that a blind spot built deliberately against a checker behaves nothing like one hit by chance: at a matched rate, picking the top-scoring candidate fell from 0.862 accuracy to zero. A saboteur is the constructed case, which is why reviewing a finished record is so weak a defense here. Our correlated-critique work drew on classical ensemble theory to argue that decorrelated reviewers matter more than more reviewers, and this study fits that: humans and models found different mechanisms, and pooling across them did better than any single audit. Two of the three major flaws in the predictor experiment were a change in what was tested and a change in how it was scored, which a pre-registered design and metric of the kind our own programme uses would have made visible, though it would not catch a learning rate tuned to fail. The defenses the authors suggest, rerunning key experiments under small perturbations and having a separate agent replicate the work without seeing the original result, are untested here. Those are what we would want measured before trusting an automated research pipeline with safety work.

Related research

Leadership

A word from the founder — on why the brain is the blueprint, and why the next decade of intelligence will be built here.

"The brain is the universe's only proof that general intelligence is possible. We treat it as the blueprint."

From cortical interfaces to photonic silicon, every program at NeuroGenesys exists to answer one question: how does intelligence arise — and how do we build it safely.

Will Schulz

Will Schulz

Founder, NeuroGenesys • AW3 Technology