Probability & Inference

Reasoning under uncertainty: the rules of probability, Bayes' rule, why averages concentrate, and how maximum likelihood turns a model into a loss function.

4 lessons • 28 min read

1. 01

   Probability as the arithmetic of uncertainty

   Every modern AI system that outputs a prediction, from a spam filter to a large language model, is in the business of assigning numbers to uncertain outcomes. Probability is the small set of rules those numbers must obey to stay consistent, and nearly everything else in machine learning is built on top of that arithmetic.

   7 min read
2. 02

   Bayes' rule and the base-rate trap

   A detector that is right 95% of the time can still be wrong about most of the cases it flags. Bayes' rule says exactly when and by how much, and it is the single most useful formula for reading any classifier, test, verifier or evaluation result.

   7 min read
3. 03

   Expectation, variance and why averages work

   How many test questions do you need before a model's benchmark score means anything? Why does a minibatch of 512 examples give a usable gradient for a dataset of billions? Both questions have the same answer: averages of independent quantities concentrate around their expectation, at a rate you can calculate.

   7 min read
4. 04

   Maximum likelihood: where loss functions come from

   Squared error for regression, cross-entropy for classification, next-token log-loss for language models: these loss functions look like separate conventions, but they are one idea applied to different noise models. Maximum likelihood says to choose the parameters under which the data you actually observed would have been most probable.

   7 min read