Entropy: measuring surprise

How much information does an outcome carry, and how few bits can you get away with to describe a random source? Entropy answers both questions at once, and the answer turns out to be the quantity every language model is trained to estimate.

Surprise as a number

Start with a single event of probability $p$. We want a measure of how surprising it is, $s(p)$, with three properties: certain events carry no surprise ($s(1)=0$), rarer events are more surprising ($s$ decreases in $p$), and the surprise of two independent events adds ($s(pq) = s(p) + s(q)$). The only continuous function satisfying all three is a logarithm:

The base of the logarithm sets the unit. Base 2 gives bits; the natural log gives nats ($1\ \text{nat} \approx 1.443$ bits). A fair coin flip landing heads has surprisal $-\log_2 \tfrac12 = 1$ bit. Drawing the ace of spades from a shuffled deck has surprisal $\log_2 52 \approx 5.70$ bits.

Entropy is expected surprise

For a discrete random variable $X$ with distribution $p$, the entropy is the average surprisal:

with the convention $0 \log 0 = 0$. Claude Shannon introduced this quantity in 1948 and showed that, up to the choice of unit, it is the unique measure of uncertainty satisfying a short list of natural requirements: continuity in the probabilities, growth with the number of equally likely outcomes, and consistency when a choice is broken into successive sub-choices.^[1]

Three facts follow directly from the definition:^[2]

• $H(X) \ge 0$, with equality only when one outcome has probability 1.
• For $K$ outcomes, $H(X) \le \log K$, with equality only for the uniform distribution. Uniform is maximally uncertain.
• Entropy depends only on the probabilities, not on the labels. Relabelling the outcomes does not change it.

Worked example: a biased coin

A coin lands heads with probability $0.9$. Its entropy in bits is

The rare tails outcome is very surprising (3.32 bits), but it happens so seldom that it contributes only a third of a bit on average. The common heads outcome is barely surprising at all. Overall, the coin carries less than half the information of a fair coin.

Entropy is the limit of compression

The definition would be only a curiosity if it did not also measure something operational. Shannon's source coding theorem shows that it does. For symbols drawn independently from $p$, no lossless code can use fewer than $H(X)$ bits per symbol on average. Codes exist that come arbitrarily close to that limit when symbols are encoded in long blocks.^[1][2] Entropy is the irreducible cost of describing the source.

The connection is easiest to see when probabilities are powers of two. Take four symbols with probabilities $\tfrac12, \tfrac14, \tfrac18, \tfrac18$:

The entropy is $\tfrac12(1)+\tfrac14(2)+\tfrac18(3)+\tfrac18(3) = 1.75$ bits. The code gives each symbol a codeword whose length equals its surprisal, and no codeword is a prefix of another, so a stream of bits decodes unambiguously. Its average length is exactly 1.75 bits, compared with the 2 bits a fixed-length code needs. Huffman's 1952 algorithm builds an optimal prefix code of this kind for any finite distribution.^[3] When the probabilities are not powers of two, a symbol-by-symbol code can lose up to one bit per symbol. Coding longer blocks, or using arithmetic coding (covered in Prediction is compression), closes that gap.

The lesson to keep is that surprisal is code length. An outcome with probability $p$ "deserves" a codeword of about $-\log_2 p$ bits, and entropy is the average length when the code matches the source.

Joint, conditional and mutual information

Real data are sequences, and symbols depend on their context. For two variables $X$ and $Y$:

• Joint entropy $H(X,Y) = -\sum_{x,y} p(x,y)\log p(x,y)$.
• Conditional entropy $H(Y\mid X) = -\sum_{x,y} p(x,y)\log p(y\mid x)$. This is the average uncertainty left in $Y$ once $X$ is known.
• Chain rule $H(X,Y) = H(X) + H(Y\mid X)$.
• Mutual information $I(X;Y) = H(Y) - H(Y\mid X) \ge 0$. This is how much knowing $X$ reduces uncertainty about $Y$.

Because mutual information is never negative, conditioning cannot increase entropy on average: $H(Y\mid X)\le H(Y)$.^[2] More context can only help prediction, at least in expectation.

Applied repeatedly, the chain rule factors the entropy of a whole sequence:

This is exactly the factorisation an autoregressive language model uses. The model predicts each token from everything before it, and the total uncertainty of a text is the sum of its per-token conditional uncertainties.

How much information is in English?

Shannon measured this directly in 1951. Treating letters as independent with their observed frequencies gives about 4.14 bits per letter, below the 4.7 bits of a uniform 26-letter alphabet. Conditioning on the previous letter (digram statistics) brings this down to about 3.56 bits. When people were asked to guess the next letter of a passage with up to about 100 letters of context, the estimate fell to roughly one bit per letter, which implies that English is around 75% redundant.^[4]

That ladder of numbers describes what language modelling does. Each richer model of context lowers the conditional entropy it assigns to the next symbol. A modern language model's training loss, measured in bits per character or per token, places it on the same ladder. The true conditional entropy of the text sits underneath as a floor that no model can go below.

Entropy in modern models

Entropy appears in AI systems in several places beyond the loss:

• The predictive distribution. At each step a language model outputs a distribution over its vocabulary. The entropy of that distribution measures how uncertain the model is about the next token. A low value means one continuation dominates, and a high value means many are plausible.
• Sampling temperature. Logits $z_i$ are turned into probabilities with $p_i \propto \exp(z_i/T)$. As $T\to 0$ the distribution collapses onto the argmax and its entropy goes to 0. As $T\to\infty$ it approaches uniform, with entropy $\log K$. Temperature is therefore a direct control over output entropy.
• Continuous variables. For densities, the analogue $h(X) = -\int f\log f$ (differential entropy) can be negative and changes under a change of variables, so it does not have the same "bits to describe" meaning.^[2] Differences of differential entropies, such as mutual information, keep their meaning.

Key takeaways

• Surprisal $-\log p$ is the only additive, continuous measure of how unexpected an outcome is. Entropy is its expectation.
• Entropy is the minimum average number of bits per symbol needed to describe a source losslessly. An outcome with probability $p$ deserves a codeword of about $-\log_2 p$ bits.
• The chain rule breaks the entropy of a sequence into per-step conditional entropies, which is the structure autoregressive language models exploit.
• More context never increases entropy on average. Shannon's estimates for English fell from about 4.14 to about 1 bit per letter as context grew.
• Temperature and predictive entropy are the everyday faces of entropy in deployed models.

Sources