NeuroGenesys.
Research Desk • Field Watch

Most of what RL teaches an image generator fits in a few constant vectors, the same for every prompt

• 6 min read • By Research Desk

Save

Key findings

  • On six public RL-tuned image models built on SD3.5-M and FLUX.1-dev, constant steering vectors added to the frozen base model recovered 90.0% to 96.3% of the RL reward gain, and on two RL-tuned video models 89.8% and 111.9%.
  • On SD3.5-M the vectors hold 288,768 parameters, 1.5% of the 18.8M in the rank-32 LoRA that Flow-GRPO trains, and they are fitted to the RL-tuned model's outputs with no reward in the objective.
  • Trained from zero with Flow-GRPO on a frozen SD3.5-M, the vectors alone reached 104.6% of the OCR-tuned model's gain and 103.6% of the PickScore-tuned model's, and 25 raters judged their images on par with the RL-tuned models' (50% to 52% win rates).
  • Simply averaging the activation changes RL induces recovered far less, as little as 12.2% of the gain on GenEval and 21.5% on text rendering.
  • The constant shift reaches the image through what image tokens read from text tokens in joint attention: resetting that route to the base model left only 20.5% of the vectors' text-rendering gain and 26.4% of the RL-tuned model's.
A wet salt flat at night under an indigo, star-filled sky with an orange glow on the horizon, where a row of slender violet and blue light columns of different heights and shapes rises from the reflective salt, all crossed at the same height by a single straight amber beam

Reinforcement learning has become a standard last step for image and video generators. Methods such as Flow-GRPO and DanceGRPO update a diffusion transformer's weights against a reward, an OCR check that the requested words were rendered, a GenEval check that the requested objects were composed, or a preference model such as PickScore, and the tuned models get visibly better. Because those skills change with every prompt, the natural assumption is that RL must be teaching the network new prompt-dependent transformations. A preprint posted to arXiv on November 19 by Tuna Han Salih Meral and Pinar Yanardag of Virginia Tech, Kaan Oktay and Atakan Kara of fal, Alec Helbling of Georgia Tech and IBM Research, and Duen Horng Chau of Georgia Tech, "RL Post-Training in Diffusion Transformers Is Activation Steering", tests how much of the gain survives if that assumption is dropped entirely.

The probe is deliberately crude. The authors freeze the base model and add one fixed vector to every token at the output of each transformer block. The vector may differ by block, by image or text stream, between the conditional and unconditional guidance passes, and between early and late sampling steps, but it never depends on the prompt, the token or the random seed. On SD3.5-M that comes to 288,768 parameters, 65 times fewer than the 18.8M in the rank-32 LoRA Flow-GRPO trains. The vectors are fitted by regression: cache samples from a public RL-tuned checkpoint, re-noise them, and train the vectors so the steered base model predicts the same velocity as the RL-tuned one. No reward model appears anywhere in the fit, which took at most 17 H100 GPU-hours per image model and 70 per video model.

Across eight public RL-tuned checkpoints the constant shift recovered most of the gain. On six image models spanning SD3.5-M and FLUX.1-dev, two RL algorithms, LoRA and full fine-tuning, and PickScore, HPSv2.1, OCR and GenEval rewards, it recovered 90.0% to 96.3% of each model's reward gain on held-out test prompts; on the Wan2.1 and Wan2.2 video models it recovered 89.8% and 111.9%. It also held on hard cases: on 146 GenEval prompts that the base model failed at three of four seeds, the vectors recovered 97.7% of the GenEval-tuned model's gain at the held-out seed. The vectors reproduced side effects too, gaining about as much as the PickScore-tuned model on HPSv2.1 and ImageReward, which it was never trained on, and copying its drop in color energy and its diversity. The obvious shortcut fails, however: averaging the RL-tuned model's activation changes into a vector recovered only 12.2% to 79.0% of the image gains, and on text rendering the part of the fitted vectors orthogonal to that average kept 96% of their gain.

The second experiment removes the RL-tuned model from the picture. Running Flow-GRPO on a frozen SD3.5-M with only the steering vectors trainable, starting from zero, reached 104.6% of the OCR-tuned model's gain and 103.6% of the PickScore-tuned model's. In a pairwise study with 25 raters over 50 test prompts, images from the steered base model beat the base model's 60% to 65% of the time and were statistically indistinguishable from the RL-tuned models' at 50% to 52%. A LoRA with as many parameters as the vectors, fitted the same way, showed no detectable advantage on the SD3.5-M, FLUX.1-dev and Wan2.2 models and did worse on Wan2.1. The contrast case is supervised fine-tuning that teaches a subject the base model cannot draw: there the vectors recovered only 68% to 83% of the identity gain of DreamBooth LoRAs for six fictional people, and a same-size LoRA did better.

How can one shared vector produce different text for different prompts? The authors' answer is that the frozen network does the prompt-specific work, mixing the constant into prompt-dependent attention and MLP computations. On SD3.5-M they trace the route: when image tokens read the text tokens' keys and values from the unsteered base model, with everything else unchanged, only 20.5% of the vectors' OCR gain and 10.5% of their PickScore gain survive, and the RL-tuned models keep 26.4% and 24.5% under the same reset, so both reach the image the same way. Vectors limited to the text stream, the conditional pass and the early steps, 35,328 parameters or 0.19% of the LoRA, still recovered 89.5% of the OCR gain and 73.7% of the PickScore gain. The paper's hypothesis is that policy-gradient RL post-training mainly steers the base model toward what it can already compute, which fits earlier findings the authors cite on language models, where RL mostly raises the probability of answers the base model already samples and steering vectors trained with RL on a math reward match or nearly match full RL fine-tuning.

The claims are carefully bounded. The authors show sufficiency, not necessity: the RL-tuned models do not themselves have to work this way, and the authors explicitly do not claim RL updates are low-rank. Every model studied was trained with policy gradients, so preference optimization and reward backpropagation are untested. The video evidence rests on two models, and only two of the six public RL-tuned video models the authors evaluated showed a reward gain whose 95% interval cleared zero. The rewards are automated scorers, so a shift that recovers a reward gain recovers whatever that scorer measures, including its blind spots, and the user study is small. Whether these rewards could benefit from genuinely new input-dependent transformations, or whether current RL methods simply cannot find them, is left open. The paper is an unreviewed preprint.

For NeuroGenesys the result sharpens a question that runs through our continual-learning work: when post-training improves a model, how much new computation did it actually add? Here the answer for RL on image generators is very little, and a recent paper we covered found something similar for language models, where zeroing fewer than 0.05% of weights can match full fine-tuning. If most of a reward gain lives in a few hundred thousand constant numbers sitting outside the frozen weights, skills of this kind could be stored, swapped and removed as small modules without disturbing what the base model knows, which is the cheapest possible answer to forgetting. The other side matters as much: a method that mostly re-steers existing capability is not evidence that a model has learned something it could not do before, and the DreamBooth contrast shows where constants stop being enough. We would want the same test run on RL-trained language reasoning models before reading capability gains from RL as new capacity.

Related research

Leadership

A word from the founder — on why the brain is the blueprint, and why the next decade of intelligence will be built here.

"The brain is the universe's only proof that general intelligence is possible. We treat it as the blueprint."

From cortical interfaces to photonic silicon, every program at NeuroGenesys exists to answer one question: how does intelligence arise — and how do we build it safely.

Will Schulz

Will Schulz

Founder, NeuroGenesys • AW3 Technology