To keep what a model knows, control what it is allowed to move

In October 2025, a team at Meta FAIR posted Continual Learning via Sparse Memory Finetuning. They replaced one feed-forward layer of a 1.3B model with a memory layer of about a million slots. When the model learned new TriviaQA facts, only the 500 slots that the new data used far more than 1,000 batches of pretraining data were allowed to change. At equal learning of the new facts, the model's NaturalQuestions score fell 89% under full fine-tuning, 71% under LoRA and 11% under sparse memory finetuning. It is one model, one kind of knowledge and a preprint. It is also the clearest demonstration yet that existing knowledge survives best when new learning has somewhere else to go.

We reviewed the continual-learning literature from the classic regularisers to work posted this autumn, and checked the headline figures against the primary papers. The common thread is that freezing important weights is the weakest form of protection. On class-incremental Split MNIST, van de Ven and colleagues measured EWC at 20.6% and Synaptic Intelligence at 21.2%, barely above the 19.9% of a network with no protection at all, while replay methods reached 88.8–94.4%. Per-weight importance scores are noisy, the penalties pile up task after task, and the network gradually stiffens.

What holds up is keeping the model close to what it already does. Ibrahim et al. showed that mixing 5% of old data into continued pretraining of a 10B model brought final loss to 1.89, against 1.87 for retraining on everything. RL's Razor (ICLR 2026) found that forgetting tracks how far the model's outputs drift from the original on the new task, which is why on-policy reinforcement learning forgets less than supervised fine-tuning. A May 2026 preprint put numbers on this at 7–8B over eight TRACE tasks. On Qwen2.5-7B, plain sequential fine-tuning lost 13.9 points on earlier tasks, replaying 1% of old data lost 4.3, and replaying the model's own best answers lost 2.3. The same group's self-distillation fine-tuning (ICML 2026) lost only 1.0, though it learned some new tasks less well.

The harder problem is deciding what to overwrite. Knowledge editors that look surgical break down in sequence: Gupta et al. saw MEMIT collapse after about 1,400 edits on GPT-J, and The Mirage of Model Editing found single-edit success falls from 96.8% to 38.5% when the model answers freely. Arbitration between old and new knowledge is crude. In ClashEval, models adopted wrong retrieved text over their own correct answer more than 60% of the time. Retiring knowledge is no easier. Łucki et al. took a model whose WMDP-Bio score had been unlearned from 64.4% to 29.9% and brought it back to 61.6% by fine-tuning on ten unrelated examples.

That last result cuts both ways. If unlearning mostly hides knowledge, so does much of what gets called forgetting. The spurious forgetting paper (ICLR 2025) found that much of the drop in sequentially fine-tuned language models is lost task alignment that can be recovered, not erased knowledge. Real erosion comes from somewhere else: Dohare et al. showed in Nature that networks trained on thousands of tasks gradually lose the ability to learn at all, ending below a linear baseline. Their fix, continual backprop, resets a tiny fraction of little-used units at each step. Every method that protects old knowledge spends some of that plasticity.

The newest work combines protection with renewal, as the brain is thought to. On October 27, Google Research described a wake–sleep framework in which a model consolidates short-term memory into newly unlocked low-rank experts, resets its fast-changing parameters and rehearses on data it generates itself. The paper reports near-zero forgetting across stages of continual translation into Manchu and Kalamang, but does not state model size and has not been replicated. A September preprint from Johns Hopkins, Continual Learning Mechanisms Compose, tested every combination of four protection mechanisms over 100 tasks. Combining them raised retention from 1.2% to 34.9%, which is a large relative gain and still poor in absolute terms.

Almost all of this evidence comes from models of 14B parameters or fewer, over a few dozen tasks at most. The practical recipe for now is to keep a slice of old data in every update, train on the model's own outputs where possible, give new facts their own capacity, and test whether apparent losses can be recovered before counting them as forgotten. Deciding which facts should be retired remains essentially unsolved. The result to watch for is any of these methods holding up on a frontier-scale model through a long stream of real updates.