Scaling laws and compute-optimal training

Given a fixed compute budget, how large should a model be and how many tokens should it see? Scaling laws answer this with fitted power laws. They have steered billion-dollar training runs since 2020, and the right answer has shifted twice since then: first when Chinchilla corrected the allocation, and again when inference cost and test-time compute entered the budget.

Loss falls as a power law

Kaplan et al. trained hundreds of transformers and found that test loss followed smooth power laws in non-embedding parameter count $N$, dataset size $D$ (tokens) and compute $C$. The trends held over many orders of magnitude, while architectural details such as depth versus width mattered little within a broad range.^[1] When the other factors are not limiting:

The exponents are small. Doubling $N$ multiplies this loss term by $2^{-0.076}\approx0.949$, about a 5% reduction, so each further gain costs exponentially more resources. On a log–log plot, though, the curves are straight lines, so they can be extrapolated from small, cheap runs to large, expensive ones. Kaplan et al. concluded that the most compute-efficient strategy was to train very large models on relatively modest data and stop well before convergence.^[1]

The compute identity

Training costs about 6 FLOPs per parameter per token: about 2 for the forward pass (see the transformer block) and about 4 for the backward pass. So

This turns "how should I spend $C$?" into a constrained optimization over $N$ and $D$.

Chinchilla: scale data with the model

Hoffmann et al. revisited the question with more than 400 models, from 70M to over 16B parameters, trained on 5B to 500B tokens. They fitted a loss surface of the form

where $E$ is an irreducible floor (the entropy of text itself). Minimizing it subject to $C=6ND$ gave a different answer from Kaplan's: parameters and training tokens should grow in roughly equal proportion with compute.^[2] The paper's three estimation methods broadly agreed, and its headline model applied the rule. Chinchilla had 70B parameters and was trained on 1.4T tokens, about 20 tokens per parameter. It used the same compute as the 280B-parameter Gopher with about 4× more data, and it outperformed Gopher across a wide range of tasks.^[2] By this analysis, earlier large models had been substantially undertrained.

The result has itself been audited. Besiroglu et al. reconstructed the data behind Chinchilla's third fitting method from the paper's figures. They found that the published parametric fit matched the data poorly and implied about 70 tokens per parameter, inconsistent with the 20 actually used. Their corrected fit agreed with the other two methods.^[3] The qualitative conclusion survived. The precise constants are less certain than they first looked.

Worked example

Suppose you have $C=10^{24}$ FLOPs and follow the "20 tokens per parameter" rule, so $D=20N$:

That is a model of about 91B parameters trained on 1.8T tokens. Now check a real model. Llama 3 405B was trained on 15.6T tokens, and Meta reports $3.8\times10^{25}$ FLOPs.^[4] The identity gives $6\times405\times10^9\times15.6\times10^{12}\approx3.8\times10^{25}$, which matches. That is about 38 tokens per parameter. The naive 20:1 rule would have put about 560B parameters on 11T tokens for the same budget, yet Meta's own scaling-law fits found 405B approximately compute-optimal.^[4] Fitted optima depend on the data, the tokenizer and the fit, so "20 tokens per parameter" is a rough guide, not a constant of nature.

Training-optimal is not deployment-optimal

Chinchilla minimizes training compute for a given loss. A deployed model, however, spends compute on every token it serves, at about $2N$ FLOPs each. Sardana et al. added inference demand to the objective. They found that a team expecting heavy usage, on the order of a billion requests, should train a smaller model on more tokens than Chinchilla prescribes. In their experiments, quality kept improving out to extreme ratios of up to 10,000 tokens per parameter.^[5] Industry practice moved the same way. Meta trained the smaller Llama 3 models "much longer than is compute-optimal",^[4] and Alibaba pretrained its Qwen3 family, from 0.6B to 235B parameters, on 36T tokens.^[6]

Two further constraints now shape the allocation.

• Data is finite. Muennighoff et al. found that repeating data for up to about 4 epochs gives nearly the same loss as fresh data, with steeply diminishing returns after that.^[7]
• "Parameters" became ambiguous. In a mixture-of-experts model, compute per token follows active parameters while memory follows total parameters. DeepSeek-V3 has 671B total and 37B active parameters and was trained on 14.8T tokens for 2.788M H800 GPU-hours in its final run.^[8] Applying $6ND$ to active parameters gives about $3.3\times10^{24}$ FLOPs (our estimate, not a reported figure). Sparsity and compression covers why.

New axes: inference-time and RL compute

Since 2024, a second scaling dimension has opened up. OpenAI reported that its o1 reasoning model improved both with more reinforcement-learning compute during training and with more "thinking" compute at inference.^[9] Snell et al. showed that adaptively allocating test-time compute can be more than 4× more efficient than best-of-N sampling. In a FLOPs-matched comparison, on problems where the small model already had some success, it beat a 14× larger model.^[10] RL compute appears to follow different curves from pretraining. A 400,000-GPU-hour study by Khatri et al. found that a sigmoidal curve in log-compute, rising to a ceiling, fit RL performance better than a power law.^[11] The test-time compute course covers why such curves saturate, often at limits set by the verifier.

What scaling laws do and do not predict

Scaling laws predict loss on the training distribution, and they predict it well. They do not directly predict downstream capabilities. Benchmark accuracy can look as if it jumps abruptly even while loss declines smoothly, and the apparent jump is often an artefact of the metric or the sampling. The emergence audit tool lets you test such claims. Scaling laws also say nothing about which data, architecture or post-training to use. They hold those fixed and measure the slope.

Key takeaways

• Language-model loss falls as a power law in parameters, data and compute, which makes extrapolation from small runs possible.
• Training compute is about $6ND$. Chinchilla showed that $N$ and $D$ should scale roughly equally, about 20 tokens per parameter, though the exact constant is fit-dependent.
• Deployment changes the optimum: heavily used models are deliberately "overtrained" relative to Chinchilla.
• Mixture-of-experts separates compute (active parameters) from memory (total parameters).
• As of November 2026, test-time and RL compute are separate scaling axes with their own, often saturating, curves.

Sources