NeuroGenesys.
Research Desk • Field Watch

Replaying RL experts' own training logs merges them into one model at a third of the cost, and the order of the logs matters most

• 5 min read • By Research Desk

Save

Key findings

  • Replaying three RL experts' training logs in their original order produced a 4B student scoring 58.6 averaged over reasoning, search and SQL, against 58.1 for multi-teacher on-policy distillation from the same experts and 59.3 for the experts themselves.
  • On eight H100s the replay took 9 hours against 29 for on-policy distillation with task reward, and about 14% of the compute used to train the experts, because it samples nothing and calls no tool or environment.
  • Plain supervised fine-tuning on the replayed logs in order (57.5) already beat online GRPO, PPO and RAFT trained on all three domains at once (52.5, 52.6 and 51.5).
  • Shuffling the 300 logged steps cost 0.6 points with the full method and 2.6 to 3.8 points without the distillation term, while cutting the logs to 576 of 1,536 samples per step cost only 0.2.
  • Every configuration was run once, on a single 4B dense model, three domains and 300 steps, and the authors have not tested a student that starts from a different checkpoint than the experts.
A wet salt flat at night under an indigo, star-filled sky with the Milky Way rising on the right and an orange glow on the horizon, where three dotted lines of violet, blue and amber lights run in from the horizon and meet in the middle distance, continuing toward the camera as a single line of larger golden lights reflected in the wet, cracked salt

A common recipe in frontier post-training is to train one reinforcement-learning expert per domain from a shared checkpoint, so no domain is traded off against another, and then fold the experts into one model by multi-teacher on-policy distillation (MOPD): the student generates its own trajectories in every environment and each domain's expert scores them token by token. That second stage is expensive for long-horizon agents, since every step waits for its slowest rollout and every tool and sandbox has to keep running. A preprint posted to arXiv on November 18 by Zhenrui Yue, Gongxiaohui Chen and colleagues at the University of Illinois Urbana-Champaign, Argonne National Laboratory, Amazon and an independent researcher, "Replay, Don't Resample: Off-Policy Consolidation of Long-Horizon RL Experts", points out that the experts' RL runs already produced the data consolidation needs and then threw it away. Every RL step logs a batch of rollouts with their rewards and sampling probabilities; for three experts trained for 300 steps that is about 460,000 trajectories.

Their method, multi-expert replay policy distillation (MRPD), treats those logs as a schedule rather than a dataset. At step t the student, which starts from the same checkpoint as the experts, trains on every expert's step-t batch: it recomputes the experts' own clipped policy-gradient loss on the logged trajectories and rewards, and it minimizes the KL divergence to each final expert's full next-token distribution at every logged prefix. Because the student follows the same path the experts took, it stays close to the policies that wrote the logs. The gap between the student's and the logged log-probabilities peaked at 0.126 around step 20, with 4% of tokens clipped, then fell to 0.019 by the end, so the replay is off-policy in origin but, in the authors' term, quasi-on-policy throughout.

The test used Qwen3-4B after a supervised cold start, with GRPO experts for competition math with a Python sandbox, Wikipedia search question answering and multi-turn SQL, evaluated on 16 held-out sets. Averaged over the three domains, MRPD scored 58.6, MOPD 58.1 and MOPD with an added task reward 58.2. Online GRPO, PPO and RAFT trained on all three environments at once reached only 52.5, 52.6 and 51.5. TIES merging of the expert weights reached 56.8 and a uniform weight average 46.1. The GRPO experts, each scored on its own domain, averaged 59.3, so MRPD sits 0.7 below its teachers. The cost gap is larger than the accuracy gap. On eight H100s a replay step took 110 seconds against 350 for MOPD with task reward, and the full run 9 hours against 29. The experts had taken 30, 32 and 4.5 hours, so consolidation cost about 14% of their training compute.

The ablations are the most interesting part. Most of the gain comes from the logged trajectories themselves: plain supervised fine-tuning on all of them, in order, already reached 57.5, and filtering to the successful ones reached 57.6, against 51.5 for the same RAFT recipe on the student's own rollouts. The dense distillation term added the most on top of that, and the policy gradient added a little more late in training. Order mattered more than quantity or reward. Keeping only 576 of each step's 1,536 samples cost 0.2 points and zeroing the rewards cost 0.4, but shuffling the 300 steps cost 0.6 with the full method and 2.6 to 3.8 points without the distillation term. The shuffled runs were not slower early on; they kept pace for 200 steps and then stalled. Replay also avoided one inherited flaw: the final search expert hit its turn limit in about 60% of rollouts, and MOPD's student copied the habit, hitting the limit in 80% from step 50 on and finishing with the weakest search score of the trained students (40.6 against 42.0 for MRPD).

The limits are substantial. Every configuration was run once; a second MRPD run scored 58.5, and the authors put the binomial standard error of the overall score at about 0.3, so the 0.4 to 0.5 point lead over MOPD is within one or two standard errors. The study covers a single 4B dense model with same-sized experts, three domains and 300 steps. The authors say it remains to be shown whether the results carry to larger models or to the mixture-of-experts models where consolidation is actually used. The method relies on the student starting from exactly the experts' checkpoint, and nobody has yet tested what happens when it does not. The replay also needs logs that most RL pipelines discard. Reasoning kept the largest gap to its expert, 50.4 against 53.8 on AIME 2024. The paper is an unreviewed preprint, and the code link it gives did not resolve when we checked.

For NeuroGenesys, the result bears directly on a question our note on merging and splitting experts left open. That note argues weight-space merging is only safe for experts that share an origin, and plans an experiment (MS1) comparing weight averaging with distillation for consolidating confusable experts. Here, with a shared origin, distillation along the experts' own path beat both TIES and a plain weight average, and the uniform average collapsed on search and SQL. The finding that order matters more than amount also fits our rehearsal results, where what is replayed, and not just how much, decided what a model kept. What we would want before relying on it is the test the authors flag themselves: a student that does not share the experts' starting point, since a growing expert library is exactly the case where origins drift apart. Storing the ordered training logs of each expert, rather than only its final weights, looks like a cheap habit to adopt in the meantime.

Related research

Leadership

A word from the founder — on why the brain is the blueprint, and why the next decade of intelligence will be built here.

"The brain is the universe's only proof that general intelligence is possible. We treat it as the blueprint."

From cortical interfaces to photonic silicon, every program at NeuroGenesys exists to answer one question: how does intelligence arise — and how do we build it safely.

Will Schulz

Will Schulz

Founder, NeuroGenesys • AW3 Technology