Maximum likelihood: where loss functions come from

Squared error for regression, cross-entropy for classification, next-token log-loss for language models: these loss functions look like separate conventions, but they are one idea applied to different noise models. Maximum likelihood says to choose the parameters under which the data you actually observed would have been most probable.

The likelihood function

A model is a family of probability distributions $p(x \mid \theta)$ indexed by parameters $\theta$: a coin's bias, a Gaussian's mean, or the billions of weights of a neural network. Given independent observations $x_1, \dots, x_n$, the likelihood is the probability of those data viewed as a function of $\theta$:

The likelihood is not a probability distribution over $\theta$: it need not integrate to one, and treating it as a posterior requires a prior (see Bayes' rule). It is a score for how well each candidate $\theta$ explains the data.

The maximum likelihood estimate is $\hat\theta = \arg\max_\theta L(\theta)$. R. A. Fisher set out the method and its theory in his 1922 paper on the foundations of theoretical statistics, which also introduced the now-standard distinction between a population parameter and a sample statistic.^[1]

Because a product of many small numbers underflows, and because logarithm is increasing (so it does not move the maximum), one always works with the log-likelihood and, by convention in machine learning, minimises its negative:

That negative log-likelihood (NLL) is the loss function.

Worked example: a biased coin

Observe 10 flips with $k = 7$ heads. Model each flip as heads with probability $\theta$. The log-likelihood is

Set the derivative to zero: $\frac{7}{\theta} - \frac{3}{1-\theta} = 0$, so $7(1-\theta) = 3\theta$ and $\hat\theta = 0.7$. In general $\hat\theta = k/n$: the maximum likelihood estimate of a probability is the observed frequency. Unsurprising, but derived rather than assumed.

The same calculation shows a weakness. With 3 flips and 3 heads, $\hat\theta = 1$: the model declares tails impossible. Maximum likelihood trusts the sample completely. Laplace's rule of succession, $(k+1)/(n+2)$, which results from a uniform prior on $\theta$, gives $4/5$ instead.^[2] In language modelling the same failure appears as assigning zero probability to unseen word sequences, which is why classical $n$-gram models needed smoothing.

Different noise, different loss

Gaussian noise gives squared error

Suppose a model predicts $f_\theta(x)$ and the observed target is $y = f_\theta(x) + \varepsilon$ with $\varepsilon \sim \mathcal N(0, \sigma^2)$. Then

With $\sigma$ fixed, the second term is a constant, so maximising likelihood is exactly minimising the sum of squared errors.^[3] Least squares is maximum likelihood under Gaussian noise. If the noise were Laplace-distributed, the same derivation would give absolute error instead, which is less sensitive to outliers. Choosing a loss is choosing a noise model, whether or not you say so.

Categorical outcomes give cross-entropy

For a classifier over $K$ classes, the model outputs a distribution $q_\theta(\cdot \mid x)$, usually via a softmax. The probability of the observed label $y$ is $q_\theta(y \mid x)$, and the NLL over a dataset is

This is the cross-entropy between the data's labels and the model's predictions. Maximum likelihood is equivalent to minimising the KL divergence from the empirical data distribution to the model, because the two differ only by the data's own entropy, which does not depend on $\theta$.^[4] The information theory course develops that view.

Language models are maximum likelihood estimators

An autoregressive language model factorises a text with the chain rule, $p_\theta(x_{1:T}) = \prod_t p_\theta(x_t \mid x_{<t})$. Its log-likelihood is a sum over positions, so the training loss for a corpus is the average per-token NLL:

Every token in the training data is a classification example whose label is the token that actually came next. Bengio and colleagues' 2003 neural language model was trained by maximising this log-likelihood over its training corpus,^[5] and the same objective, scaled up by many orders of magnitude, underlies modern pretraining. The empirical scaling laws of Kaplan and colleagues are statements about this cross-entropy loss as a function of model size, data and compute.^[6]

Worked example: loss and perplexity

A model predicts four tokens of a sentence, assigning the true tokens probabilities 0.5, 0.25, 0.8 and 0.1. The per-token NLLs in nats are

The mean is $4.605/4 = 1.151$ nats per token. Perplexity is its exponential, $e^{1.151} \approx 3.16$: the model was, on average, as uncertain as a uniform choice among about three tokens. Equivalently, perplexity is one over the geometric mean of the assigned probabilities, $(0.5 \times 0.25 \times 0.8 \times 0.1)^{-1/4} = 0.01^{-1/4} \approx 3.16$. Note how the single confident miss (0.1) contributes half the loss: log-loss punishes assigning low probability to what actually happened far more than it rewards being right with high confidence.

What maximum likelihood guarantees, and what it does not

Under regularity conditions and with a correctly specified model, the maximum likelihood estimator is consistent (it converges to the true parameter as $n \to \infty$) and asymptotically efficient (no consistent estimator has smaller asymptotic variance). Fisher's 1922 paper introduced consistency and efficiency as criteria for estimators and argued that maximum likelihood meets them; fully rigorous proofs came later.^[1]

Three caveats matter more in practice:

• Overfitting. With many parameters and finite data, the likelihood can be pushed up by memorising the sample. The coin with three heads is the smallest example. Adding a penalty, such as $\lambda\|\theta\|^2$ (weight decay), is equivalent to maximising a posterior under a Gaussian prior on the weights, known as MAP estimation.^[3] See Generalization.
• Misspecification. If the true distribution is not in the model family, maximum likelihood finds the member closest in KL divergence. That can still be useful, but "best in the family" is not "correct".
• Likelihood is not the goal. A language model trained to maximise the likelihood of internet text learns to reproduce that text's distribution, errors included. Later stages of training (preference learning, reinforcement learning with verifiable rewards) exist precisely because the maximum likelihood objective does not specify helpfulness or correctness; see From RLHF to verifiable rewards.

Key takeaways

• Maximum likelihood picks the parameters that make the observed data most probable; in practice one minimises the negative log-likelihood.
• The loss encodes a noise model: Gaussian noise gives squared error, categorical outcomes give cross-entropy.
• Language-model pretraining is maximum likelihood estimation: minimising average per-token NLL, reported as loss in nats or as perplexity, its exponential.
• Maximum likelihood is consistent and efficient for well-specified models with enough data, but it overfits small samples; priors and regularisation are the correction.

Sources