Eigenvectors and the SVD: the directions that matter

Most matrices in a trained network are large, but their action is often dominated by a handful of directions. Eigenvectors and the singular value decomposition find those directions, say how much each one matters, and give the provably best way to throw the rest away.

Eigenvectors: directions a map only stretches

A square matrix $A$ generally rotates and stretches vectors. An eigenvector is a nonzero vector whose direction $A$ leaves unchanged:

The scalar $\lambda$ is its eigenvalue: the factor by which $A$ stretches (or, if negative, flips) that direction. Along an eigenvector a matrix acts like a number.

The most useful case is a symmetric matrix ($A = A^\top$), such as a covariance matrix or a Hessian of a loss. The spectral theorem guarantees that a real symmetric $n \times n$ matrix has $n$ real eigenvalues and an orthonormal set of eigenvectors, so

with the eigenvectors as the columns of $Q$ and the eigenvalues on the diagonal of $\Lambda$.^[1] In the eigenvector coordinate system, the matrix is just independent scaling along each axis.

Worked example

Let $A = \begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix}$. Try $v_1 = (1, 1)$: $Av_1 = (3, 3) = 3v_1$. Try $v_2 = (1, -1)$: $Av_2 = (1, -1) = 1 \cdot v_2$. So the eigenvalues are 3 and 1, with orthogonal eigenvectors along the diagonals. $A$ triples everything along $(1,1)$ and leaves $(1,-1)$ alone.

Repeated application: the dominant direction wins

Apply $A$ repeatedly to $x = (1, 0)$:

The vector swings towards $(1,1)$. Writing $x = \tfrac12 v_1 + \tfrac12 v_2$ shows why: $A^k x = \tfrac12 3^k v_1 + \tfrac12 v_2$, and the $v_1$ component outgrows the other by a factor of 3 per step. This is power iteration, the simplest algorithm for finding the top eigenvector.

The same arithmetic explains a classical failure of recurrent networks. Backpropagating through $T$ time steps multiplies gradients by the recurrent weight matrix (and activation derivatives) $T$ times. Pascanu, Mikolov and Bengio showed that, in the linear case, a largest eigenvalue below 1 is sufficient for long-range gradient components to vanish, and a largest eigenvalue above 1 is necessary for them to explode.^[2] Eigenvalues of size $0.9$ or $1.1$ look harmless until raised to the hundredth power: $0.9^{100} \approx 2.7 \times 10^{-5}$ and $1.1^{100} \approx 1.4 \times 10^{4}$.

The SVD: every matrix, rotate–scale–rotate

Eigenvectors require a square matrix, and for non-symmetric matrices they may be complex or fail to form a basis. The singular value decomposition has no such restrictions. Any real $m \times n$ matrix can be written

where $U$ ($m \times m$) and $V$ ($n \times n$) have orthonormal columns, and $\Sigma$ is diagonal with singular values $\sigma_1 \ge \sigma_2 \ge \cdots \ge \sigma_r > 0$, $r = \operatorname{rank}(W)$.^[1] Geometrically: rotate the input into the basis $\{v_i\}$, stretch the $i$-th axis by $\sigma_i$ (dropping dimensions or adding zeros to change size), then rotate into the output basis $\{u_i\}$. Every linear map, including every weight matrix in a network, is that sequence.

The singular values are the square roots of the eigenvalues of $W^\top W$, and the right singular vectors $v_i$ are its eigenvectors. For a symmetric matrix with non-negative eigenvalues, like $A$ above, the SVD and the eigendecomposition coincide.

The SVD reads off several quantities at once:

• Rank is the number of nonzero singular values.
• The spectral norm $\|W\|_2 = \sigma_1$ is the largest factor by which $W$ can stretch any vector. It bounds how much a layer can amplify a perturbation, which is why Miyato and colleagues stabilised GAN training by dividing each layer's weights by an estimate of $\sigma_1$ (spectral normalisation).^[3]
• The Frobenius norm $\|W\|_F = \sqrt{\sum_{ij} W_{ij}^2} = \sqrt{\sum_i \sigma_i^2}$ measures total energy across all directions.

Eckart–Young: the best low-rank approximation

Keep only the top $k$ terms of the sum:

Eckart and Young proved in 1936 that no rank-$k$ matrix is closer to $W$ in the Frobenius norm,^[4] and the error is exactly what was discarded:

(The analogous result holds in the spectral norm, with error $\sigma_{k+1}$.) If the singular values decay quickly, a small $k$ captures most of the matrix, and $W_k$ needs only $k(m + n + 1)$ numbers instead of $mn$.

Applied to a centred data matrix whose rows are examples, the top right singular vectors are the directions of greatest variance, which is principal component analysis. Pearson introduced the idea in 1901 as finding the lines and planes of closest fit to a cloud of points.^[5]

Worked example: compressing $A$

For $A = \begin{pmatrix} 2 & 1 \\ 1 & 2 \end{pmatrix}$, $\sigma_1 = 3$ with $u_1 = v_1 = \tfrac{1}{\sqrt 2}(1, 1)$, and $\sigma_2 = 1$. The best rank-1 approximation is

The error's Frobenius norm is $\sqrt{4 \times 0.25} = 1 = \sigma_2$, as the theorem says. The approximation keeps $\sigma_1^2 / (\sigma_1^2 + \sigma_2^2) = 9/10$ of the energy.

Low rank in modern models

These ideas explain why low-rank methods work for fine-tuning. Aghajanyan, Zettlemoyer and Gupta measured the intrinsic dimension of fine-tuning: optimising only 200 parameters, randomly projected into the full parameter space of a RoBERTa model, reached 90% of full fine-tuning performance on one paraphrase task (MRPC), and larger pretrained models tended to have lower intrinsic dimension.^[6] LoRA, introduced in the previous lesson, builds the low-rank constraint into the weight update itself, $\Delta W = BA$. Hu and colleagues report that on GPT-3, a rank as small as one sufficed for adapting the query and value projections on the tasks they examined; comparing the subspaces learned at rank 8 and rank 64, they found that the top singular directions overlapped substantially while the remaining directions did not.^[7] In SVD terms, the useful part of the update was concentrated in its first few singular values.

Low rank is an empirical property of particular updates and tasks, not a law, and nothing guarantees it for a new task. The SVD is the tool for checking: compute the singular values of a learned update, and see how fast they decay.

Key takeaways

• An eigenvector is a direction a square matrix only stretches; symmetric matrices have orthogonal eigenvectors and real eigenvalues.
• Repeated application amplifies the largest-eigenvalue direction; this is power iteration, and in recurrent networks it is the root of vanishing and exploding gradients.
• The SVD writes any matrix as rotate–scale–rotate. Singular values give the rank, the spectral norm (maximum stretch) and the Frobenius norm (total energy).
• By Eckart–Young, truncating the SVD gives the best rank-$k$ approximation, with error equal to the discarded singular values. PCA and low-rank adaptation such as LoRA rest on this.

Sources