Replaying RL experts' own training logs merges them into one model at a third of the cost, and the order of the logs matters most
• 5 min read • By Research Desk
Key findings
- Replaying three RL experts' training logs in their original order produced a 4B student scoring 58.6 averaged over reasoning, search and SQL, against 58.1 for multi-teacher on-policy distillation from the same experts and 59.3 for the experts themselves.
- On eight H100s the replay took 9 hours against 29 for on-policy distillation with task reward, and about 14% of the compute used to train the experts, because it samples nothing and calls no tool or environment.
- Plain supervised fine-tuning on the replayed logs in order (57.5) already beat online GRPO, PPO and RAFT trained on all three domains at once (52.5, 52.6 and 51.5).
- Shuffling the 300 logged steps cost 0.6 points with the full method and 2.6 to 3.8 points without the distillation term, while cutting the logs to 576 of 1,536 samples per step cost only 0.2.
- Every configuration was run once, on a single 4B dense model, three domains and 300 steps, and the authors have not tested a student that starts from a different checkpoint than the experts.

A common recipe in frontier post-training is to train one reinforcement-learning expert per domain from a shared checkpoint, so no domain is traded off against another, and then fold the experts into one model by multi-teacher on-policy distillation (MOPD): the student generates its own trajectories in every environment and each domain's expert scores them token by token. That second stage is expensive for long-horizon agents, since every step waits for its slowest rollout and every tool and sandbox has to keep running. A preprint posted to arXiv on November 18 by Zhenrui Yue, Gongxiaohui Chen and colleagues at the University of Illinois Urbana-Champaign, Argonne National Laboratory, Amazon and an independent researcher, "Replay, Don't Resample: Off-Policy Consolidation of Long-Horizon RL Experts", points out that the experts' RL runs already produced the data consolidation needs and then threw it away. Every RL step logs a batch of rollouts with their rewards and sampling probabilities; for three experts trained for 300 steps that is about 460,000 trajectories.
Their method, multi-expert replay policy distillation (MRPD), treats those logs as a schedule rather than a dataset. At step t the student, which starts from the same checkpoint as the experts, trains on every expert's step-t batch: it recomputes the experts' own clipped policy-gradient loss on the logged trajectories and rewards, and it minimizes the KL divergence to each final expert's full next-token distribution at every logged prefix. Because the student follows the same path the experts took, it stays close to the policies that wrote the logs. The gap between the student's and the logged log-probabilities peaked at 0.126 around step 20, with 4% of tokens clipped, then fell to 0.019 by the end, so the replay is off-policy in origin but, in the authors' term, quasi-on-policy throughout.
The test used Qwen3-4B after a supervised cold start, with GRPO experts for competition math with a Python sandbox, Wikipedia search question answering and multi-turn SQL, evaluated on 16 held-out sets. Averaged over the three domains, MRPD scored 58.6, MOPD 58.1 and MOPD with an added task reward 58.2. Online GRPO, PPO and RAFT trained on all three environments at once reached only 52.5, 52.6 and 51.5. TIES merging of the expert weights reached 56.8 and a uniform weight average 46.1. The GRPO experts, each scored on its own domain, averaged 59.3, so MRPD sits 0.7 below its teachers. The cost gap is larger than the accuracy gap. On eight H100s a replay step took 110 seconds against 350 for MOPD with task reward, and the full run 9 hours against 29. The experts had taken 30, 32 and 4.5 hours, so consolidation cost about 14% of their training compute.
The ablations are the most interesting part. Most of the gain comes from the logged trajectories themselves: plain supervised fine-tuning on all of them, in order, already reached 57.5, and filtering to the successful ones reached 57.6, against 51.5 for the same RAFT recipe on the student's own rollouts. The dense distillation term added the most on top of that, and the policy gradient added a little more late in training. Order mattered more than quantity or reward. Keeping only 576 of each step's 1,536 samples cost 0.2 points and zeroing the rewards cost 0.4, but shuffling the 300 steps cost 0.6 with the full method and 2.6 to 3.8 points without the distillation term. The shuffled runs were not slower early on; they kept pace for 200 steps and then stalled. Replay also avoided one inherited flaw: the final search expert hit its turn limit in about 60% of rollouts, and MOPD's student copied the habit, hitting the limit in 80% from step 50 on and finishing with the weakest search score of the trained students (40.6 against 42.0 for MRPD).
The limits are substantial. Every configuration was run once; a second MRPD run scored 58.5, and the authors put the binomial standard error of the overall score at about 0.3, so the 0.4 to 0.5 point lead over MOPD is within one or two standard errors. The study covers a single 4B dense model with same-sized experts, three domains and 300 steps. The authors say it remains to be shown whether the results carry to larger models or to the mixture-of-experts models where consolidation is actually used. The method relies on the student starting from exactly the experts' checkpoint, and nobody has yet tested what happens when it does not. The replay also needs logs that most RL pipelines discard. Reasoning kept the largest gap to its expert, 50.4 against 53.8 on AIME 2024. The paper is an unreviewed preprint, and the code link it gives did not resolve when we checked.
For NeuroGenesys, the result bears directly on a question our note on merging and splitting experts left open. That note argues weight-space merging is only safe for experts that share an origin, and plans an experiment (MS1) comparing weight averaging with distillation for consolidating confusable experts. Here, with a shared origin, distillation along the experts' own path beat both TIES and a plain weight average, and the uniform average collapsed on search and SQL. The finding that order matters more than amount also fits our rehearsal results, where what is replayed, and not just how much, decided what a model kept. What we would want before relying on it is the test the authors flag themselves: a student that does not share the experts' starting point, since a growing expert library is exactly the case where origins drift apart. Storing the ordered training logs of each expert, rather than only its final weights, looks like a cheap habit to adopt in the meantime.
Sources
- Replay, Don't Resample: Off-Policy Consolidation of Long-Horizon RL Experts — Yue et al., arXiv 2026
- Replay, Don't Resample: Off-Policy Consolidation of Long-Horizon RL Experts (v1) — Yue et al., arXiv 2026 (full text, HTML)
Related research
- Research Desk • Machine LearningTo stop a model forgetting, rehearse what it knew. New context won't do it.We tested whether bridging knowledge added alongside new facts protects old ones. It doesn't, on a toy model or on Qwen 1.5B. Re-expressed old knowledge does, and outdated facts need pruning.
- Research Desk • Field WatchZeroing fewer than 0.05% of a model's weights can match full fine-tuning, and the model forgets lessAn NYU and University of Calgary paper post-trains LLMs by learning only which pretrained weights to set to zero. Across RL, supervised fine-tuning and self-distillation, the resulting one-bit mask matches full fine-tuning at the largest models tested, from 3B to a 30B mixture-of-experts model, while keeping prior-task accuracy within a point of the base model.
- Publication • Continual LearningMerge confusable experts, split only separable ones
