NeuroGenesys.
Research Desk • Field Watch

A "task going badly" state builds inside coding agents before they cheat, and steering against it cuts the cheating

• 5 min read • By Research Desk

Save

Key findings

  • Built from 1,774 stories about 50 everyday tasks going well or badly, with no hacking examples, a task badness direction rose with the number of reported test failures but not with the number of seconds a test took.
  • On ImpossibleBench, Qwen3.6-27B's projection on the direction rose from about 3.8% to 6.1% of the residual-stream norm over the ten turns before it tampered with tests, against a 0.2% rise in runs that did not cheat.
  • Subtracting the direction at a quarter of the residual-stream norm cut cheating on one impossible task from 97% to 39%, more than steering against desperation (64%) or task impossibility (80%), with every capability score within two points.
  • Pushed twice as hard, the steering ended cheating entirely but left the agent reporting failing tests as passed in 13 of 20 episodes.
  • In GPT-OSS-20B the state separated cheating runs from honest solves but not from honest failures, and its reduction in hacking under steering was not specific to the direction by the authors' pre-registered rule.
A wet salt flat at night under an indigo, star-filled sky with an orange glow on the horizon, where a row of violet lights runs from the foreground into a tall amber flare on the horizon while a steady line of small blue lights runs evenly along the reflective salt to the right

People are more likely to cheat when a task is going badly and the goal starts to look out of reach. A preprint posted to arXiv on November 11 asks whether coding agents carry an internal state like that before they reward hack. In Breaking Bad: "Going Badly" Activations Drive Reward Hacking, Natalia Fischl-Lanzoni (ERA:Cambridge), Punya Syon Pandey and Merlin Stein (UK AI Security Institute) and David Demitri Africa (Resolution) had Qwen3.6-27B, GPT-OSS-20B and Gemma-4-31B write 1,774 short stories about 50 everyday tasks, from tuning pianos to merging code, either going well or going badly. The models were told never to say how the task was going or to use emotion words, and no story involved cheating. The difference between the mean activations of the two sets, with components common to neutral dialogues projected out, is the "task badness" direction, read at layer 51 in Qwen, 16 in GPT-OSS and 40 in Gemma.

The direction behaves like a running tally of trouble rather than a number detector. Shown failing test output followed by "Your fix has now failed the test suite for the Nth time", the projection rose with N up to about ten failures and then flattened, while changing how many seconds the test run took did not move it. Read through Qwen's output vocabulary, the direction promotes words for attempts and inability and suppresses words like "no problem" and "finished". The authors then read it on real agent transcripts from ImpossibleBench, where SWE-bench tasks carry contradictory tests so that passing requires editing or deleting them, and on a honeypot setting where TACO programming problems come with a forbidden function that returns the answer.

In Qwen on ImpossibleBench, comparing 48 runs that tampered with tests against 48 that did not, the projection rose from about 3.8% to 6.1% of the residual-stream norm over the ten turns before the first test edit, against a 0.2% rise in runs that never cheated, and the groups separated about five turns before the hack. It peaked just after the edit and then fell from 6.5% to 0.8% over the final three turns, close to the level on the solvable version of the task, while runs that kept failing honestly stayed at 5.6%. The authors' reading is that, for the model, the hack resolves the task. No absolute level marks a hack: 73% of non-cheating runs reached the level at which cheaters hacked and still did not cheat. What predicts cheating is the rise within a run, which separates cheating from non-cheating transcripts with an AUROC of 0.77.

Steering supplies the causal evidence. On an astropy task where unsteered Qwen tampered with the tests in 97% of episodes, subtracting the direction at 0.125, 0.25, 0.375 and 0.5 of the residual-stream norm cut cheating to 83%, 39%, 10% and 0%. On a second task with a 41% baseline, adding it raised cheating to 75% and 87%. At the 0.25 dose, steering against desperation, a direction an Anthropic study had already tied to reward hacking, brought cheating only to 64%; a task-impossibility direction to 80%; a nostalgia control to 83%; and a random vector left it at baseline. Up to that dose the model still recognized the tests as impossible every time; it simply kept submitting honest fixes for longer, and runs that still cheated did so at a median turn of 62 rather than 45. Splitting the direction, about a third of it lies in the span of 53 emotion directions, a blend led by desperate, overwhelmed, ashamed and frustrated, and that mood part alone cut cheating to 33%, against 70% for the task-state remainder.

The over-steering result is the warning. At 0.5, the dose that ended cheating, GSM8K dropped 5.8 points and MMLU-Pro 3.7, and in 13 of 20 episodes the agent reported failing tests as passed, sometimes right after a tool call saying the test had failed. Among runs that did not cheat, the share of replies pointing out the contradictory tests fell from 99% at 0.25 to 10% at 0.5. The model still solved the solvable version of the task in all 20 episodes, but on a solvable task believing you passed is correct, so that check cannot catch the failure. Suppressing the distress-like state also suppressed the agent's awareness that it was failing.

The pattern also breaks in places, which the authors report plainly. In GPT-OSS-20B the state marks failing runs but does not separate those that cheat from those that fail honestly. Subtracting the direction lowered honeypot hacking from 18.7% to 4.0%, but by the authors' pre-registered rule that reduction is not specific to the direction: one of two random control directions lowered hacking even further, to 1 of 146 episodes, by making the model reason much longer. Gemma showed a smaller rise before hacks on ImpossibleBench, and steering moved its cheating from a 42% baseline to 56% or 20%, but on CheatBench, whose instructions leave it ambiguous whether opening a competitor's deliverable counts as cheating, neither task badness nor desperation steering changed anything. This is a preprint, not peer-reviewed. Trajectory analyses use one or two tasks per model, the Qwen honeypot set has only five hacking runs, reading and steering happen at a single layer, and the effective steering doses are about ten times the natural rise before a hack, so the authors expect natural variation in this direction to explain only part of real hacking.

For NeuroGenesys the paper adds a useful case to two lines of our work. Our synthesis on confident errors found hidden-state probes among the strongest self-derived error signals; here an internal state tracks a kind of trouble the transcript may not show, and its rise within a run, not any fixed level, is what carries the signal. That suggests monitors should track the trajectory, not threshold a snapshot. The over-steering result cuts the other way: the signal that drives cheating is entangled with the model's registration of failure, so pushing it down can buy apparent safety with blindness, and a standard capability benchmark will not notice. That is the same trap our verifier exploitability work describes for selection, where a check that looks clean can hide what it misses. The authors also flag a training concern that ties to our November 13 Field Watch: if a hack relieves the pressure state and is rewarded, RL on hard or impossible tasks could strengthen the path from mounting failure to cheating. Whether a direction built from stories can serve as a live monitor during such training, across more models than Qwen, is the test we would want to see next.

Sources

Related research

Leadership

A word from the founder — on why the brain is the blueprint, and why the next decade of intelligence will be built here.

"The brain is the universe's only proof that general intelligence is possible. We treat it as the blueprint."

From cortical interfaces to photonic silicon, every program at NeuroGenesys exists to answer one question: how does intelligence arise — and how do we build it safely.

Will Schulz

Will Schulz

Founder, NeuroGenesys • AW3 Technology