To stop a model forgetting, rehearse what it knew. New context won't do it.

When a model learns new facts, it loses some old ones. A natural idea is to send supporting material along with each update: paraphrases, context that links the new fact to what the model already knows, the facts that follow from it. If the new knowledge arrives well connected, perhaps it disturbs less. We tested that idea directly, first on a small transformer trained on a synthetic knowledge base and then on Qwen2.5-1.5B-Instruct with LoRA, and it did not hold.

In the toy model, every supplement got the same training budget as a control of unrelated filler facts. Paraphrases of the new facts and new context about the people at risk changed old-fact retention by about −2 points, within noise. Adding the facts implied by an update was worse, costing 11.4 points, because those facts overwrote old ones of the same kind. What worked was the old knowledge itself, even reworded: old facts phrased in a wording the model had never trained on kept 94.2% of old knowledge, against 47.6% with nothing added and 95.8% for verbatim replay.

The 1.5B model told the same story. We probed it for real facts it answered correctly, then trained 60 counterfactual edits plus 15 genuine changes from 2024–2026, among them a new Pope, new prime ministers and new chief executives. It learned 95–100% of the updates in every condition. Asked in a wording never used in training, it kept 33.2% of its old facts with the updates alone and 32.3% with bridging facts added. Re-expressed old facts kept 88.9% and verbatim replay 91.8%. Rehearsing half of the old facts protected the other half too, which survived at 82–84%, and a 99-question general-knowledge check stayed at 95–96% with rehearsal against 83.8% without it.

Which old facts to rehearse matters less than rehearsing some. In the toy model, choosing the facts the model was least confident about beat random selection by 2.9 points, and by 4.5 points on a quarter of the budget, which fits Jin and Ren's forecast-then-replay results, where predicted-at-risk replay cut accuracy loss to 1.63% from 5.77% for random replay on a 400M model. It did not carry over to 1.5B. There the least-confident facts were a worse pick than random, 6.7 points behind at full budget, and the model's confidence barely predicted which facts an update would knock out (AUROC 0.57, against 0.74 in the toy). Running the update on a copy and rehearsing whatever broke protected the facts it checked and nothing else: on a quarter of the budget it gained 14.8 points on checked facts and lost 3.3 on the rest, and at full budget it tied random replay for 2.1 times the compute.

The source of the rehearsal mattered more than the selection. Distilling toward the frozen original model's outputs, rather than replaying the stored answers, kept 5.4 points more of the old facts that were never rehearsed when asked in the trained wording, and 4.0 in a new wording, within noise, at no cost overall. Rehearsing on the original model's own answers as fixed targets did no better than the stored answers.

The finding we least expected concerns knowledge that has gone out of date. Rehearsal protects old answers indiscriminately, including the ones an update was meant to replace. Leaving superseded facts in the rehearsal pool at their natural share cost 11 points of update accuracy in the toy model. Pruning them fixed the direct answer, a person's new employer, but not what followed from it: 93% of updated people still gave their old city, because nothing taught the new one. Any rehearsal at all roughly doubled those stale answers compared with none. At 1.5B the problem was larger, because superseded facts made up about 30% of a natural rehearsal pool, and pruning them raised update accuracy by 22.8 points. Pruning the facts derived from them cut stale second-hop answers by 37 points, far more than in the toy, but 60% stayed stale. Training the new consequences fixed exactly the people it covered, 96% correct, and none of the others. This is the in-weights version of the gap RippleEdits measured for knowledge editors, where putting the edit in the prompt scored about 82 on consistency of consequences and weight-editing methods scored well below.

Nor did an LLM need to write the supplements. Using the model's own answers on old questions as rehearsal targets, a toy stand-in for generated context, came within 2.6 points of answers taken from the knowledge base, an approach close to self-synthesized rehearsal. But when 30% of those generated targets were wrong, retention of the very facts they were meant to protect fell 31.5 points. Generated material helps only once it has been checked. Training on facts a model does not already know carries its own cost, as Gekhman and colleagues showed: once learned, such examples increase the model's tendency to hallucinate.

These are small experiments: a two-layer transformer, a 1.5B model, 3 to 5 seeds, and no correction for multiple comparisons. At 1.5B we dropped the fictional-fact track and the filler control when the machine was overloaded, and the learning rate came from a single pilot. The practical recipe they point to is still clear. When a fact changes, prune the outdated fact and everything derived from it, add the new consequences for every affected entity, rehearse a broad random sample of old knowledge, re-expressed or distilled from the original model, rather than a confidence-targeted one, and test on wordings it has never seen. We have pre-registered a larger study of the hardest part, keeping consequences consistent across thousands of sequential updates.