Mixture-of-experts, quantization and distillation

There are three main ways to make a language model cheaper per token without making it proportionally worse. Activate only part of it for each token (mixture-of-experts). Store and move fewer bits per weight (quantization). Train a small model to imitate a large one (distillation). By late 2026 the leading open-weight models use all three at once. This lesson explains how each works, what it actually saves, and where it stops paying.

Mixture-of-experts: more parameters, same FLOPs

In a mixture-of-experts (MoE) layer, the single feed-forward network is replaced by $E$ expert FFNs and a small router. For a token vector $x$, the router scores the experts, keeps the top $k$ and mixes their outputs:

Compute per token scales with $k$, while capacity scales with $E$. Shazeer et al. introduced the sparsely gated MoE layer in 2017, with up to thousands of experts, and reported better language-modelling and translation results at lower computational cost than dense baselines.^[1] GShard used MoE to train a 600B-parameter translation model across 2,048 TPU v3 accelerators.^[2] The Switch Transformer simplified routing to a single expert per token and reported 4× faster pretraining than T5-XXL.^[3] Mixtral 8×7B brought the idea to widely used open weights. It has 8 experts per layer with 2 selected per token, so each token can draw on 47B parameters but uses only 13B.^[4]

Two design shifts followed. DeepSeekMoE split experts into many smaller ones and added always-on shared experts that hold common knowledge, so the routed experts can specialize.^[5] Routers also need to spread load evenly, or a few experts saturate while others sit idle. That was traditionally enforced with an auxiliary loss. DeepSeek-V3 introduced a load-balancing strategy that needs no auxiliary loss.^[6]

Sparsity ratios have kept falling

Qwen3's flagship routes each token to 8 of 128 experts.^[7] Kimi K2 routes to 8 of 384.^[8]

What MoE does not save

MoE cuts FLOPs per token. It does not cut the memory needed to hold the model, because every expert must be resident somewhere. At serving batch sizes it also does not cut bandwidth by as much as the active fraction suggests.

Worked example. Suppose routing were uniform over Qwen3's 128 experts, with 8 chosen per token. The chance that a given expert is not chosen by one token is $120/128=0.9375$. For a batch of $b$ tokens in one decode step:

At batch 64, about 98% of experts are touched, so a decode step reads nearly all expert weights, not 6% of them. Real routing is not uniform, but the lesson holds. MoE's bandwidth saving is largest at small batch, and at large batch MoE mostly buys capacity per FLOP. Experts spread over many chips also add all-to-all communication. This is one reason interconnect bandwidth binds in MoE decode, as discussed in inference economics.

Quantization: fewer bits per number

A symmetric $b$-bit integer quantizer maps a weight $w$ to $\hat w = s\cdot\mathrm{round}(w/s)$, with scale $s=\max|w|/(2^{b-1}-1)$ set over some group of weights. Rounding error is roughly uniform on $[-s/2,s/2]$, with variance $s^2/12$, so everything depends on keeping $s$ small.

Worked example. Quantize the block $(0.9,\,-0.3,\,0.05,\,-1.2)$ to 4 bits (integer levels $-7,\dots,7$). Then $s=1.2/7=0.171$ and the block becomes $(5,-2,0,-7)\cdot s = (0.857,\,-0.343,\,0,\,-1.2)$. Every error is at most 0.05. Now replace $-1.2$ with an outlier of $12.0$. The scale becomes $s=1.71$, and $0.9$ rounds to $1.71$ while the two small weights round to zero. One outlier has destroyed the block. Two families of fixes follow from this example.

• Smarter rounding. GPTQ quantizes weights one column at a time, using approximate second-order information to push each rounding error onto weights not yet quantized. It compressed 175B-parameter models to 3–4 bits in about four GPU-hours with little accuracy loss.^[11] AWQ observed that protecting roughly 1% of salient weights, identified by activation magnitude rather than weight magnitude, removes much of the error. It does this by scaling those channels before quantizing.^[12]
• Smaller groups with their own scales. FP8 standardized two 8-bit floating-point encodings, E4M3 for weights and activations and E5M2 for gradients, and matched 16-bit training accuracy in the authors' experiments.^[13] The OCP microscaling (MX) formats give each block of 32 values a shared power-of-two scale. MXFP4 stores 4-bit E2M1 elements.^[14] NVIDIA's NVFP4 shrinks the block to 16 values and uses a more precise FP8 (E4M3) scale per block, for 4.5 bits per value on average.^[15]

The deployed state of the art has moved to 4 bits. gpt-oss-120b ships its MoE weights in MXFP4 so that the model fits on a single 80 GB GPU.^[9] DeepSeek-V4-Pro stores expert weights in FP4 and most other weights in FP8.^[10] For scale: 70B parameters take 140 GB in BF16, 70 GB in FP8 and about 39 GB in NVFP4.

The floor is real, though. In NeuroGenesys experiments, about 4 bits was where precision began to bind and 2 bits was a cliff.^[16][17] In a prefill test on a real checkpoint, a KV cache produced with quantized weights worked as well as fully quantized inference, but accuracy under naive quantization fell from 1.000 at 5 bits to 0.111 at 4.^[18] Digital hardware's move to native 4-bit also removed much of the low-precision advantage that analog accelerators had relied on (see beyond the GPU).^[19]

Distillation: learning from a teacher's probabilities

Hinton, Vinyals and Dean proposed training a small student to match a large teacher's full output distribution, softened with a temperature $T$:^[20]

The softened distribution carries information that hard labels do not. With teacher logits $(4,2,0)$, the probabilities are $(0.87,0.12,0.02)$ at $T=1$ but $(0.51,0.31,0.19)$ at $T=4$. That exposes how the teacher ranks the wrong answers. The $T^2$ factor keeps gradient magnitudes comparable as $T$ changes.^[20]

For language models, distillation often means fine-tuning on text the teacher generates. DeepSeek released six dense models, from 1.5B to 70B parameters, fine-tuned on reasoning traces from DeepSeek-R1. Its distilled Qwen-7B scored 55.5% on AIME 2024, and on a 32B base, distillation beat running RL directly.^[21] NVIDIA combined pruning with distillation to shrink Nemotron-H 56B to 47B, keeping similar accuracy with 20% faster inference.^[22] IBM used distillation on self-generated data to make LLMs robust to analog-hardware noise.^[23]

How the three compose, and when they do not

The three methods target different resources. MoE reduces FLOPs per token, quantization reduces bytes per weight, and distillation reduces the parameter count needed for a given quality. Their savings can multiply, but only when they act on different bottlenecks. A useful rule from NeuroGenesys's hardware audits is not to multiply gains that consume the same resource.^[16] For example, quantizing an MoE model's experts and claiming the full bandwidth saving at batch 1 double-counts if the deployment actually runs at batch 64, where nearly every expert is read anyway.

Key takeaways

• MoE decouples capacity from compute. Active fractions fell from about 28% (Mixtral) to about 3% (Kimi K2, DeepSeek-V4-Pro) by 2026.
• MoE does not reduce memory footprint, and at large batch nearly every expert is read on each decode step.
• Quantization error is set by the scale over a group. Second-order rounding (GPTQ), salience-aware scaling (AWQ) and small-block formats (MXFP4, NVFP4) push deployment to about 4 bits, near where precision binds.
• Distillation transfers a teacher's full distribution or generated traces, and is how most small reasoning models are made.

Sources