Most of what RL teaches an image generator fits in a few constant vectors, the same for every prompt
• 6 min read • By Research Desk
Key findings
- On six public RL-tuned image models built on SD3.5-M and FLUX.1-dev, constant steering vectors added to the frozen base model recovered 90.0% to 96.3% of the RL reward gain, and on two RL-tuned video models 89.8% and 111.9%.
- On SD3.5-M the vectors hold 288,768 parameters, 1.5% of the 18.8M in the rank-32 LoRA that Flow-GRPO trains, and they are fitted to the RL-tuned model's outputs with no reward in the objective.
- Trained from zero with Flow-GRPO on a frozen SD3.5-M, the vectors alone reached 104.6% of the OCR-tuned model's gain and 103.6% of the PickScore-tuned model's, and 25 raters judged their images on par with the RL-tuned models' (50% to 52% win rates).
- Simply averaging the activation changes RL induces recovered far less, as little as 12.2% of the gain on GenEval and 21.5% on text rendering.
- The constant shift reaches the image through what image tokens read from text tokens in joint attention: resetting that route to the base model left only 20.5% of the vectors' text-rendering gain and 26.4% of the RL-tuned model's.

Reinforcement learning has become a standard last step for image and video generators. Methods such as Flow-GRPO and DanceGRPO update a diffusion transformer's weights against a reward, an OCR check that the requested words were rendered, a GenEval check that the requested objects were composed, or a preference model such as PickScore, and the tuned models get visibly better. Because those skills change with every prompt, the natural assumption is that RL must be teaching the network new prompt-dependent transformations. A preprint posted to arXiv on November 19 by Tuna Han Salih Meral and Pinar Yanardag of Virginia Tech, Kaan Oktay and Atakan Kara of fal, Alec Helbling of Georgia Tech and IBM Research, and Duen Horng Chau of Georgia Tech, "RL Post-Training in Diffusion Transformers Is Activation Steering", tests how much of the gain survives if that assumption is dropped entirely.
The probe is deliberately crude. The authors freeze the base model and add one fixed vector to every token at the output of each transformer block. The vector may differ by block, by image or text stream, between the conditional and unconditional guidance passes, and between early and late sampling steps, but it never depends on the prompt, the token or the random seed. On SD3.5-M that comes to 288,768 parameters, 65 times fewer than the 18.8M in the rank-32 LoRA Flow-GRPO trains. The vectors are fitted by regression: cache samples from a public RL-tuned checkpoint, re-noise them, and train the vectors so the steered base model predicts the same velocity as the RL-tuned one. No reward model appears anywhere in the fit, which took at most 17 H100 GPU-hours per image model and 70 per video model.
Across eight public RL-tuned checkpoints the constant shift recovered most of the gain. On six image models spanning SD3.5-M and FLUX.1-dev, two RL algorithms, LoRA and full fine-tuning, and PickScore, HPSv2.1, OCR and GenEval rewards, it recovered 90.0% to 96.3% of each model's reward gain on held-out test prompts; on the Wan2.1 and Wan2.2 video models it recovered 89.8% and 111.9%. It also held on hard cases: on 146 GenEval prompts that the base model failed at three of four seeds, the vectors recovered 97.7% of the GenEval-tuned model's gain at the held-out seed. The vectors reproduced side effects too, gaining about as much as the PickScore-tuned model on HPSv2.1 and ImageReward, which it was never trained on, and copying its drop in color energy and its diversity. The obvious shortcut fails, however: averaging the RL-tuned model's activation changes into a vector recovered only 12.2% to 79.0% of the image gains, and on text rendering the part of the fitted vectors orthogonal to that average kept 96% of their gain.
The second experiment removes the RL-tuned model from the picture. Running Flow-GRPO on a frozen SD3.5-M with only the steering vectors trainable, starting from zero, reached 104.6% of the OCR-tuned model's gain and 103.6% of the PickScore-tuned model's. In a pairwise study with 25 raters over 50 test prompts, images from the steered base model beat the base model's 60% to 65% of the time and were statistically indistinguishable from the RL-tuned models' at 50% to 52%. A LoRA with as many parameters as the vectors, fitted the same way, showed no detectable advantage on the SD3.5-M, FLUX.1-dev and Wan2.2 models and did worse on Wan2.1. The contrast case is supervised fine-tuning that teaches a subject the base model cannot draw: there the vectors recovered only 68% to 83% of the identity gain of DreamBooth LoRAs for six fictional people, and a same-size LoRA did better.
How can one shared vector produce different text for different prompts? The authors' answer is that the frozen network does the prompt-specific work, mixing the constant into prompt-dependent attention and MLP computations. On SD3.5-M they trace the route: when image tokens read the text tokens' keys and values from the unsteered base model, with everything else unchanged, only 20.5% of the vectors' OCR gain and 10.5% of their PickScore gain survive, and the RL-tuned models keep 26.4% and 24.5% under the same reset, so both reach the image the same way. Vectors limited to the text stream, the conditional pass and the early steps, 35,328 parameters or 0.19% of the LoRA, still recovered 89.5% of the OCR gain and 73.7% of the PickScore gain. The paper's hypothesis is that policy-gradient RL post-training mainly steers the base model toward what it can already compute, which fits earlier findings the authors cite on language models, where RL mostly raises the probability of answers the base model already samples and steering vectors trained with RL on a math reward match or nearly match full RL fine-tuning.
The claims are carefully bounded. The authors show sufficiency, not necessity: the RL-tuned models do not themselves have to work this way, and the authors explicitly do not claim RL updates are low-rank. Every model studied was trained with policy gradients, so preference optimization and reward backpropagation are untested. The video evidence rests on two models, and only two of the six public RL-tuned video models the authors evaluated showed a reward gain whose 95% interval cleared zero. The rewards are automated scorers, so a shift that recovers a reward gain recovers whatever that scorer measures, including its blind spots, and the user study is small. Whether these rewards could benefit from genuinely new input-dependent transformations, or whether current RL methods simply cannot find them, is left open. The paper is an unreviewed preprint.
For NeuroGenesys the result sharpens a question that runs through our continual-learning work: when post-training improves a model, how much new computation did it actually add? Here the answer for RL on image generators is very little, and a recent paper we covered found something similar for language models, where zeroing fewer than 0.05% of weights can match full fine-tuning. If most of a reward gain lives in a few hundred thousand constant numbers sitting outside the frozen weights, skills of this kind could be stored, swapped and removed as small modules without disturbing what the base model knows, which is the cheapest possible answer to forgetting. The other side matters as much: a method that mostly re-steers existing capability is not evidence that a model has learned something it could not do before, and the DreamBooth contrast shows where constants stop being enough. We would want the same test run on RL-trained language reasoning models before reading capability gains from RL as new capacity.
Sources
- RL Post-Training in Diffusion Transformers Is Activation Steering — Meral et al., arXiv 2026
- RL Post-Training in Diffusion Transformers Is Activation Steering (v1) — Meral et al., arXiv 2026 (full text, HTML)
Related research
- Research Desk • Field WatchZeroing fewer than 0.05% of a model's weights can match full fine-tuning, and the model forgets lessAn NYU and University of Calgary paper post-trains LLMs by learning only which pretrained weights to set to zero. Across RL, supervised fine-tuning and self-distillation, the resulting one-bit mask matches full fine-tuning at the largest models tested, from 3B to a 30B mixture-of-experts model, while keeping prior-task accuracy within a point of the base model.
- Research Desk • Field WatchInterpretability tools carry across models, once the map has seen the right promptsA University of Houston preprint carries a refusal direction, a correctness direction and a sentiment probe between 110 pairs of open models, from 1B to 72B parameters, through linear maps fit on unlabeled prompts. Fit on ordinary questions, a map recovers 54% of the target's own refusal steering. Add 16 unlabeled prompts per class from the tool's region and recovery reaches 102%.
- Research Desk • Field WatchTraining reasoning models under a fixed token cap teaches them to stop using more, and given room they fall behind their base modelsReinforcement learning for reasoning usually caps how long each training rollout may run, to save compute. A JetBrains Research preprint finds the cap becomes part of what the model learns: trained models win at the cap but barely use extra tokens, and on competition math they end up 13 to 30 points behind their own base models when both get more room.
