Gradient descent and the shape of a loss surface

Every modern neural network is trained by some variant of one idea: compute which way the loss decreases fastest and take a small step that way. This lesson covers why that works, how the step size and the curvature of the loss interact, and what the loss surfaces of large networks look like in practice.

The update

Given a differentiable loss $L(\theta)$ over parameters $\theta\in\mathbb{R}^d$, gradient descent repeats

where $\eta>0$ is the learning rate. Textbooks usually credit the method to Augustin-Louis Cauchy, who described it in a note to the Paris Académie des Sciences in 1847. Historians point out that his note is less clear-cut than the textbook version.^[1]

Why the negative gradient? A first-order Taylor expansion gives $L(\theta + \Delta) \approx L(\theta) + \nabla L^\top \Delta$. Among all steps of a fixed small length, $\nabla L^\top\Delta$ is most negative when $\Delta$ points along $-\nabla L$ (by the Cauchy–Schwarz inequality). The gradient is the direction of steepest local increase, and we go the other way.

Curvature sets the speed limit

The first-order picture says nothing about how large a step can be. Adding the second-order term shows what goes wrong:

where $g = \nabla L$ and $H$ is the Hessian, the matrix of second derivatives. The linear term always helps. The quadratic term hurts in proportion to the curvature along the step direction. If the step is too large, the curvature term wins and the loss goes up.

One dimension

Take $L(x) = \tfrac{\lambda}{2}x^2$, with curvature $\lambda>0$. Then $\nabla L = \lambda x$ and the update is

The iterate shrinks geometrically when $|1-\eta\lambda|<1$, that is, when $0<\eta<2/\lambda$. For $\eta<1/\lambda$ it approaches the minimum smoothly. For $1/\lambda<\eta<2/\lambda$ it overshoots and oscillates while still converging. For $\eta>2/\lambda$ it diverges. The highest curvature you encounter sets the largest stable learning rate.

Many dimensions and the condition number

Near a minimum, any smooth loss looks like a quadratic $\tfrac12\theta^\top H\theta$. In the eigenbasis of $H$, gradient descent splits into independent one-dimensional problems, one per eigenvalue $\lambda_i$, each contracting by $|1-\eta\lambda_i|$ per step. The learning rate must satisfy $\eta < 2/\lambda_{\max}$ for stability, but progress along the flattest direction is governed by $\lambda_{\min}$. The ratio $\kappa = \lambda_{\max}/\lambda_{\min}$ is the condition number. With the best fixed step, $\eta = 2/(\lambda_{\max}+\lambda_{\min})$, the error contracts by a factor of $(\kappa-1)/(\kappa+1)$ per step.

Worked example: a long, narrow valley

Let $L(x,y) = \tfrac12(x^2 + 100\,y^2)$. The curvatures are 1 and 100, so $\kappa = 100$.

• Stability requires $\eta < 2/100 = 0.02$.
• The best fixed step is $\eta = 2/101 \approx 0.0198$, and the contraction factor is $99/101 \approx 0.980$.
• Reducing the error by a factor of 100 then takes about $\ln 100 / (-\ln 0.980) \approx 4.6/0.020 \approx 230$ steps.

The steep $y$ direction forces a small step size, and the shallow $x$ direction then takes hundreds of steps to make progress. The iterates zig-zag across the valley while creeping along it. A perfectly round bowl ($\kappa=1$) would be solved in one step. Poor conditioning, not distance from the minimum, is what makes plain gradient descent slow. It is the main motivation for momentum and adaptive methods, covered in Stochastic optimization.

What neural-network loss surfaces look like

A quadratic bowl is a local picture. The full loss of a deep network is non-convex, high-dimensional and shaped by symmetries; for example, permuting the hidden units of a layer leaves the function unchanged. A few facts hold up across the literature:

• Saddle points outnumber bad local minima. At a critical point in very high dimension, it is unlikely that the curvature is positive in every one of millions of directions. Dauphin et al. argued, drawing on random-matrix theory and experiments, that the main obstacle is saddle points and the flat plateaus around them, which can look like local minima. High-error local minima are less of a problem.^[2] Gradient descent slows near saddles but is not trapped by them, and the noise of stochastic gradients helps it leave.
• Many minima, connected. Overparameterised networks have large sets of parameters that fit the training data equally well. Which one the optimiser lands in depends on initialisation, learning rate and batch size. This matters for generalization (see Generalization).
• Training runs at the edge of stability. The quadratic analysis predicts divergence once the sharpness $\lambda_{\max}$ exceeds $2/\eta$. Cohen et al. found that full-batch gradient descent on neural networks typically pushes sharpness up until it hovers just above $2/\eta$. The loss then decreases non-monotonically over short timescales while still falling over long ones.^[3] The network adapts its curvature to the learning rate rather than the other way round. The classical stability bound is still the right reference point, but it does not describe what happens next.

Practical consequences

Several standard training practices follow from the curvature picture:

• Learning-rate warmup. Early in training, before the parameters have settled into well-scaled values, a large step is more likely to diverge. Starting with a small step and raising it over the first few thousand updates avoids early divergence. Large-batch training relies on warmup explicitly,^[4] and large language models routinely use it. Llama 2, for instance, used 2,000 warmup steps followed by cosine decay.^[5]
• Decay. Lowering the step later lets the iterate settle into regions of higher curvature that a large step would bounce out of.
• Gradient clipping. Capping the gradient norm limits the damage from a single step taken where the local curvature or gradient spikes.
• Normalisation and residual connections improve conditioning, so a single learning rate suits more directions at once.

Key takeaways

• Gradient descent steps along $-\nabla L$, the direction of steepest local decrease. Its use for optimisation goes back to Cauchy in 1847.
• On a quadratic, stability requires $\eta < 2/\lambda_{\max}$, and the convergence rate is governed by the condition number $\kappa=\lambda_{\max}/\lambda_{\min}$. With $\kappa=100$, a 100-fold reduction takes about 230 steps.
• Deep-network losses are non-convex, with saddle points and plateaus as the main obstacles and many equally good minima.
• In practice sharpness tends to settle near $2/\eta$ (the edge of stability). Warmup, decay and clipping are ways of managing curvature.

Sources