Next-token prediction: what a language model is

A language model is a probability distribution over sequences of text. In practice it is built from one narrow skill: given everything so far, predict the next token. This lesson explains why that single objective is enough to train a chat assistant, a code generator and a reasoning model, and what it costs at inference time.

The chain rule is the whole idea

Any joint distribution over a sequence $x_1,\dots,x_T$ can be factored exactly by the chain rule of probability:

This is an identity, not an approximation. Modelling the joint distribution is therefore the same task as modelling every conditional $p(x_t\mid x_{<t})$. An autoregressive language model is a single function $f_\theta$ that maps a prefix to a probability vector over a vocabulary $V$:

The framing is old. In 1951 Claude Shannon had people guess English text letter by letter and estimated that, with long-range context, printed English carries about one bit per letter.^[1] A language model plays Shannon's guessing game at scale.

Tokens, not words

The vocabulary is a set of tokens: subword units learned from data. The standard method adapts byte-pair encoding. It starts from characters or bytes and repeatedly merges the most frequent adjacent pair until the vocabulary reaches a target size.^[2] Frequent words become single tokens, rare words split into pieces, and no input is ever out of vocabulary. Llama 3, for example, uses a 128,000-token vocabulary.^[3] Tokenization is a design choice, not a law. Meta's Byte Latent Transformer works directly on bytes. It groups them into variable-length patches according to how hard the next byte is to predict, and the authors report better scaling than token-based models at fixed inference cost in their tests up to 8B parameters.^[4]

From counting to learning

Before neural models, the conditionals were estimated by counting n-grams. That approach runs into a combinatorial wall. With a 128,000-token vocabulary, a trigram table has $128{,}000^3 \approx 2.1\times10^{15}$ cells, and almost all of them are never observed. Bengio and colleagues proposed the fix in 2003. They mapped each word to a learned dense vector and let a neural network compute the conditional from those vectors. Similar words then get similar vectors, so evidence from one sentence generalizes to many sentences the model has never seen.^[5] Every modern language model keeps that structure: an embedding table, a network that mixes the context (today a transformer), and a softmax over the vocabulary.

The training objective, worked through

Training is maximum likelihood. Over a corpus, minimize the average negative log-probability of each actual next token:

This is the cross-entropy between the data and the model, and it is measured in nats when the log is natural. Perplexity is $e^{\mathcal{L}}$: the effective number of equally likely choices the model is hedging between.

Worked example. Take the prefix "The capital of France is". Suppose the model assigns these probabilities:

If the corpus continues with " Paris", the loss on this position is $-\ln 0.5 = 0.693$ nats, which is exactly 1 bit. Now average over a four-token span where the model assigned 0.5, 0.9, 0.2 and 0.05 to the true tokens:

On this span the model is about as uncertain as a fair choice among four options. Notice that one confident miss (0.05) contributes more than half the total. The log loss punishes overconfidence heavily.

Training is efficient for a structural reason. A causal mask stops each position from seeing the future (see attention), so a single forward pass over a sequence of $T$ tokens yields $T$ prediction problems at once. This is called teacher forcing: during training the model always conditions on the true prefix, never on its own earlier guesses.

Generation: one token at a time

To generate text, sample $x_{t}\sim p_\theta(\cdot\mid x_{<t})$, append it to the context and repeat. A temperature $\tau$ reshapes the distribution before sampling:

As $\tau\to0$ this becomes greedy decoding (always take the argmax). At $\tau=1$ it samples the model's own distribution, and larger $\tau$ flattens it. Training is parallel across positions, but generation is sequential: every new token needs another full forward pass. That asymmetry drives most of the engineering in inference economics.

Why such a narrow objective goes so far

Predicting text well requires modelling whatever produced the text: grammar, facts, arithmetic, the intentions of characters in a story. That is the prediction-is-compression argument made concrete. The empirical case arrived with scale. GPT-2, a 1.5-billion-parameter model trained only on next-token prediction over web text, reached state-of-the-art results on 7 of 8 language-modelling benchmarks zero-shot, with no task-specific training.^[6] GPT-3, at 175 billion parameters, could perform many tasks from a few examples placed in the prompt, again with no gradient updates.^[7]

A pretrained base model continues documents. It is not yet an assistant. Post-training reshapes its distribution toward helpful answers. In the InstructGPT work, human raters preferred outputs from a 1.3B-parameter model fine-tuned on human feedback over outputs from the 175B GPT-3.^[8] The objective at the bottom of the stack did not change. Post-training only changes which continuations the model makes likely. The reasoning and RL course covers that step.

What is changing (as of November 2026)

Next-token prediction remains the dominant pretraining objective, but three variations are active.

• Predicting several tokens at once. Gloeckle et al. trained models with extra output heads that predict the next $n$ tokens from a shared trunk. They reported higher sample efficiency (their 13B models solved 12% more HumanEval coding problems than next-token baselines) and up to 3× faster inference when the extra heads drive speculative decoding.^[9] DeepSeek-V3 used a multi-token prediction objective and reports that it improved benchmark performance.^[10]
• Non-autoregressive generation. Diffusion language models such as LLaDA learn to fill in masked tokens across the whole sequence and generate by iterative unmasking. Its authors report an 8B model competitive with Llama 3 8B.^[11] These are the authors' own evaluations, and diffusion models have not displaced autoregressive models at the frontier.
• Reasoning models. These are still next-token predictors. They are trained to emit long intermediate chains of tokens before answering, which moves cost from training to inference (see reasoning models).

Key takeaways

• By the chain rule, modelling $p(\text{text})$ is exactly modelling every next-token conditional. Nothing is lost in the factorization.
• The training loss is cross-entropy, or average negative log-likelihood. Perplexity $e^{\mathcal{L}}$ is its effective branching factor.
• Learned dense representations (Bengio et al., 2003) are what let these models beat count-based n-grams.
• Training parallelizes across positions. Generation is serial, one forward pass per token, which is why inference cost scales with output length.
• Multi-token prediction, diffusion LMs and byte-level models are active variations. None has replaced next-token pretraining at the frontier.

Sources