Information Theory

Shannon's measure of surprise, the cross-entropy loss every language model minimizes, and the deep equivalence between predicting data and compressing it.

3 lessons • 21 min read

1. 01

   Entropy: measuring surprise

   How much information does an outcome carry, and how few bits can you get away with to describe a random source? Entropy answers both questions at once, and the answer turns out to be the quantity every language model is trained to estimate.

   7 min read
2. 02

   Cross-entropy and KL divergence: the language-model loss

   What does it cost to describe data with the wrong probability model, and why is that cost the loss nearly every language model is trained on? Cross-entropy measures that cost, and KL divergence isolates the part that comes from the model being wrong.

   6 min read
3. 03

   Prediction is compression

   Any model that assigns probabilities to data can be turned into a lossless compressor, and any lossless compressor implies a probability model. This equivalence is exact, not a metaphor. It explains why a language model's training loss is literally a file size, and why "compresses well" and "predicts well" are the same claim.

   8 min read