NeuroGenesys.
Research Desk • Field Watch

Zeroing fewer than 0.05% of a model's weights can match full fine-tuning, and the model forgets less

• 5 min read • By Research Desk

Save

Key findings

  • A learned binary mask over the gate projections, deleting fewer than 0.05% of all weights in every GRPO run, scored 83.55% on GSM8K for Llama3-8B-Instruct against 83.24% for full fine-tuning.
  • Under GRPO the mask matched full fine-tuning on Qwen2.5-7B (52.32% vs. 51.89%), Qwen2.5-14B (55.43% vs. 54.87%) and the Qwen3-30B-A3B mixture-of-experts model (58.54% vs. 58.43%), stored in 6.9, 11.4 and 29.6 MB.
  • During GRPO on GSM8K, the mask's prior-task accuracy stayed within one point of the base model at all three Llama scales, while full fine-tuning ended 1.3 to 3.1 points below and LoRA and random sparse fine-tuning fell up to 5.4 points.
  • One-shot Wanda pruning, the mask's starting point, stayed within 2.4 points of the base model on all six backbones, while training the mask gained up to 53.2 points.
  • At smaller backbones the mask recovered 90% to 99% of full fine-tuning's accuracy, trailing most on Llama3.2-1B (54.44% vs. 60.50% on GSM8K).
A wet salt flat at night under an indigo, star-filled sky with an orange glow on the horizon, where a wide grid of small blue lights stretches across the reflective salt toward the horizon, divided down the middle by a single line of amber lights that runs to a violet glow on the horizon

How little of a pretrained language model has to change for it to learn something new? A paper posted to arXiv on November 7 by Jack Lu and Zhenbang Yang (New York University), Mike Lasby, Tejas Pote and Yani Ioannou (University of Calgary) and Mengye Ren (NYU), "The Surprising Effectiveness of Deleting Weights in LLM Post-Training", pushes the question to an extreme: what if post-training may only set existing weights to zero? Their method, Deletion Tuning, freezes the model and gives each weight in the gate projection of every transformer MLP block a trainable logit. During training each weight is multiplied by a sigmoid of its logit, which is sharpened over training into a hard keep-or-delete threshold. At deployment the adapter is one bit per masked weight: a weight keeps its pretrained value or becomes zero, and no new values are ever written. The mask starts from Wanda, a standard one-shot pruning score, and the training decides how many weights to delete. The authors test it under three post-training regimes: GRPO reinforcement learning with verifiable rewards, supervised fine-tuning (SFT) and on-policy self-distillation (SDFT).

Under GRPO, every run deleted fewer than 0.05% of all weights. On GSM8K, Llama3-8B-Instruct reached 83.55% with the mask against 83.24% with full fine-tuning, up from 60.27% for the base model. On the SimpleRL-Zoo math setup, scored as the mean of six held-out benchmarks including MATH500, AIME24 and AMC23, the mask scored 52.32% on Qwen2.5-7B against 51.89% for full fine-tuning, 55.43% against 54.87% on Qwen2.5-14B, and 58.54% against 58.43% on Qwen3-30B-A3B, a mixture-of-experts model in which the authors masked the gate projection inside each of its 128 experts and left the router alone. Smaller models fell short: the mask recovered 90% to 99% of full fine-tuning's accuracy, trailing most on Llama3.2-1B (54.44% against 60.50%). After compression the mask is 0.5 MB at Qwen2.5-0.5B and 8.4 MB at Llama 8B, 3,600 to 4,700 times smaller than the full checkpoints and smaller than a rank-1 LoRA at every scale.

Two checks show that the gains come from training the mask, not from the pruning it starts from. Wanda pruning alone, at the best of eight sparsity levels, stayed within 2.4 points of the base model on all six backbones, while the trained mask starting from the same scores gained up to 53.2 points. And the gain is not only a sharpening of answers the base model could already reach: at 128 samples per problem, the base model's pass@128 stayed 4.3 points below the mask's on Llama 8B and 6.1 points below on Qwen2.5-7B. The result also held outside RL. Under SFT, averaged over 14 MMLU-Pro subjects and 23 BIG-Bench Extra Hard subtasks, the mask matched full fine-tuning on Llama3.2-3B (22.28% against 21.19% on MMLU-Pro, and 11.39% for both on BBEH), though not at 1B on BBEH (5.65% against 8.35%). Under SDFT on Qwen2.5-7B-Instruct it scored 69.07% against 68.04% on ToolAlpaca tool use and 70.61% against 71.20% on a chemistry science set.

The result that matters most for continual learning is what the models kept. The authors tracked forward KL to the base model on the training task, which earlier work by Shenfeld and colleagues found predicts forgetting, along with mean accuracy on six earlier skills (HellaSwag, MMLU, TruthfulQA, WinoGrande, IFEval and HumanEval) during GRPO on GSM8K at the three Llama scales. Full fine-tuning's forward KL reached 1.0 on Llama 1B and 1.5 on Llama 8B, and its prior-task accuracy ended 1.3 to 3.1 points below base. The mask's forward KL stayed at or below 0.05, and its prior-task accuracy stayed within one point of base throughout training. The comparison that isolates deletion is with random sparse fine-tuning at 99.95% sparsity, which trains 0.05% of all parameters, close to the 0.035% to 0.046% the mask deletes. Yet random sparse fine-tuning and LoRA both reached a forward KL of 0.4 to 1.3, and their prior-task accuracy fell up to 5.4 points below base. Editing few weights is therefore not what protects old skills. Restricting each edit to a deletion appears to be.

Why can deletion alone keep up with full fine-tuning? The authors compare the weight changes on three geometric measures. Earlier work found that RL with verifiable rewards barely moves a matrix's singular values and steers its updates away from the largest weights of its low-rank approximation. The mask reproduces that pattern: its spectrum drift stays within 1.4 times full fine-tuning's, and the share of its changes that land on those principal weights is 0.43 to 0.46, matching full fine-tuning's below-random 0.41 to 0.46, where LoRA sits near the random baseline of 0.5. LoRA is capped at a low rank and random sparse fine-tuning edits weights fixed before training; the mask has neither restriction, and the stable rank of its update, 66 to 93, falls between LoRA's 1 to 2 and random sparse fine-tuning's 410 to 650.

The limits are real. The arXiv entry and the project page list the paper for NeurIPS 2026; we read the arXiv version. Parity with full fine-tuning holds at the largest model in each setting (Llama3.2-3B for SFT, Qwen2.5-7B for SDFT and up to 30B for GRPO), not consistently at 1B. The paper reports best-checkpoint accuracy, with no repeated seeds or error bars, and many of the gaps it calls matches are under a point in either direction. Forgetting was measured only for GRPO on GSM8K with Llama models, over a single task, so the method has not yet been tested on what continual learning actually requires: a sequence of tasks, each with its own mask, trained without access to the last. The authors name that as the natural next test, along with merging masks for different tasks. Training also keeps one floating-point logit per masked weight, so it uses less memory than full fine-tuning but more than LoRA.

For NeuroGenesys, the result is a sharp version of the lesson in our November 9 survey of continual learning: the methods that hold up limit what moves. Deletion Tuning limits not only where an update lands but what it can do there. That raises a question our own note on EWC and stale facts leaves open. There, stiffness that protected unrelated facts also froze the facts that new data contradicted. A delete-only update cannot write a new value either, so we would want to know whether it can overwrite a contradicted fact, or only add new behavior around it. The format also suits a library of experts: a task costs a few megabytes of bits on one shared base, and switching tasks only scatters zeros into it. Whether masks trained in sequence stay compatible, and whether a router can select among them without the interference a shared set of weights invites, are the experiments this paper makes cheap enough to run.

Sources

Related research

Leadership

A word from the founder — on why the brain is the blueprint, and why the next decade of intelligence will be built here.

"The brain is the universe's only proof that general intelligence is possible. We treat it as the blueprint."

From cortical interfaces to photonic silicon, every program at NeuroGenesys exists to answer one question: how does intelligence arise — and how do we build it safely.

Will Schulz

Will Schulz

Founder, NeuroGenesys • AW3 Technology