Optimization & Generalization

How networks learn and why they work on data they have never seen: gradient descent, backpropagation, stochastic optimizers, and the surprises of overparameterized models.

4 lessons • 30 min read

1. 01

   Gradient descent and the shape of a loss surface

   Every modern neural network is trained by some variant of one idea: compute which way the loss decreases fastest and take a small step that way. This lesson covers why that works, how the step size and the curvature of the loss interact, and what the loss surfaces of large networks look like in practice.

   7 min read
2. 02

   Backpropagation is the chain rule, organised

   Gradient descent needs the gradient of one scalar loss with respect to every parameter, and a large model has billions of them. Backpropagation computes all of those derivatives for roughly the cost of a couple of extra forward passes. It does so by applying the ordinary chain rule in a particular order and saving the right intermediate results along the way.

   7 min read
3. 03

   Stochastic optimization: SGD, momentum and Adam

   No one computes the exact gradient of a language model's loss over its full training set: one gradient step would require a pass over trillions of tokens. Training instead uses cheap, noisy gradient estimates and adds two refinements, momentum and per-parameter step sizes. This lesson explains each piece, why it works, and the optimizer settings large models actually use.

   7 min read
4. 04

   Generalization: overfitting, regularization and the surprises of overparameterization

   A model is useful only if it performs well on data it was not trained on. Classical statistics gave a clear account of when that happens, and modern deep learning broke much of it. Networks with far more parameters than training examples fit their data perfectly and still generalize well. This lesson covers the classical picture, the evidence against it, and what partly replaces it.

   9 min read