Backpropagation is the chain rule, organised

Gradient descent needs the gradient of one scalar loss with respect to every parameter, and a large model has billions of them. Backpropagation computes all of those derivatives for roughly the cost of a couple of extra forward passes. It does so by applying the ordinary chain rule in a particular order and saving the right intermediate results along the way.

The computational graph

Any neural network computes its loss as a composition of simple operations: matrix multiplies, additions, elementwise nonlinearities, a softmax, a log. Write this as a directed acyclic graph. Inputs and parameters are the leaves, each node applies one operation to its parents, and a single scalar $L$ sits at the end.

The chain rule says that if $L$ depends on a variable $v$ only through the nodes $u_1,\dots,u_m$ that use $v$, then

Call $\bar v = \partial L/\partial v$ the adjoint of $v$. The formula says that a node's adjoint is the sum of its children's adjoints, each multiplied by a local derivative that the node's own operation determines. Backpropagation is the algorithm that evaluates this recursion efficiently:

1. Forward pass. Evaluate the graph from inputs to loss, storing every intermediate value.
2. Backward pass. Set $\bar L = 1$. Visit nodes in reverse order and, for each one, add its contribution $\bar u\cdot\partial u/\partial v$ into the adjoint of each parent $v$.

Each node is visited once in each direction, and each local derivative costs about as much as the operation itself. The whole gradient therefore costs a small constant multiple of one forward evaluation, no matter how many parameters there are. In the automatic-differentiation literature this is called reverse-mode differentiation, and backpropagation is its use for neural networks.^[1]

Worked example: a two-weight network

Take one input $x=2$, one target $y=1$, and the network

with $w_1 = 0.5$ and $w_2 = 3$.

Forward: $z = 1$, $h = 1$, $\hat y = 3$, $L = \tfrac12(2)^2 = 2$.

Backward, one local derivative at a time:

Check numerically. Nudge $w_1$ to $0.501$. Then $\hat y = 3.006$ and $L = \tfrac12(2.006)^2 = 2.012018$. The loss rises by $0.012018$ for a $0.001$ change, a slope of about $12.02$, which agrees with $\bar w_1 = 12$. Comparing against finite differences like this, called gradient checking, is the standard test for a hand-written backward pass.

Two things in the table generalise. First, $\bar w_2$ needed the forward value $h$ and $\bar w_1$ needed $x$. The backward pass reuses stored activations, which is why training uses far more memory than inference. Second, $\bar z$ passed through the ReLU's derivative, a gate that is 1 or 0. If $z$ had been negative, no gradient would have reached $w_1$ at all.

The matrix form

Real layers act on vectors. For a linear layer $y = Wx$ with incoming adjoint $\bar y = \partial L/\partial y$, the chain rule gives

The weight gradient is an outer product of "error arriving from above" and "activation from below". The error passed further down is multiplied by the transpose of the forward matrix. A minibatch replaces vectors with matrices, and both rules become matrix multiplications. That is why the backward pass costs about twice the forward pass (one multiply for $\bar W$, one for $\bar x$) and runs on the same hardware.

For the usual output layer, softmax followed by cross-entropy, the combined adjoint on the logits is simply $q - y$, prediction minus target, as derived in Cross-entropy and KL divergence.

Why reverse mode, and who found it

The chain rule can also be applied forward: push the derivative of every intermediate with respect to one input through the graph alongside the values. That costs one pass per input. Reverse mode costs one pass per output. Training has one scalar output and up to billions of inputs, so reverse mode wins by the same factor.

Reverse-mode differentiation was published by Seppo Linnainmaa in his 1970 master's thesis, in the setting of rounding-error analysis.^[2] Its use to train networks with hidden units was popularised by Rumelhart, Hinton and Williams in 1986. They showed that hidden units trained this way come to represent useful features of the task, something the earlier perceptron procedure could not do.^[3] Modern frameworks implement reverse mode generically ("autograd"). Each operation registers its local derivative rule, and the framework records the graph during the forward pass and replays it backwards.

Where backpropagation struggles

Memory. Storing every activation for the backward pass dominates memory in large models. Activation checkpointing stores only some activations and recomputes the rest during the backward pass. Chen et al. reduced memory for an $n$-layer network to $O(\sqrt n)$ at the cost of about one extra forward pass. One 1,000-layer network went from 48 GB to 7 GB at about 30% more run time.^[4] Large-model training uses variants of this routinely.

Vanishing and exploding gradients. Through $n$ layers, the adjoint is multiplied by $n$ Jacobians. If their typical gain is below 1, the signal shrinks exponentially; if it is above 1, it blows up. Bengio, Simard and Frasconi showed that this makes learning long-range dependencies with gradient descent increasingly hard as the range grows.^[5] Residual connections, $y = x + f(x)$, add an identity term to every layer's Jacobian, which gives gradients a path that does not shrink. This property helped residual networks train at depths of over a hundred layers,^[6] and every transformer uses it. Careful initialisation, normalisation layers and gradient clipping address the same problem.

Key takeaways

• Backpropagation is reverse-mode application of the chain rule over a computational graph: a forward pass stores values, and a backward pass accumulates adjoints from loss to leaves.
• The full gradient costs a small constant multiple of one forward pass, independent of the number of parameters. Reverse mode suits one scalar output with many inputs.
• For a linear layer, $\bar W = \bar y x^\top$ and $\bar x = W^\top\bar y$. Activations must be stored, which makes memory the main cost. Checkpointing trades compute for memory.
• Products of Jacobians can vanish or explode. Residual connections, normalisation and initialisation keep gradient flow usable at depth.

Sources