Matrices as maps: layers, rank and composition

A neural network layer is, before its nonlinearity, a matrix multiplying a vector. Reading that matrix as a map that moves, stretches and collapses space explains why depth needs nonlinearities, why some information is irrecoverably lost inside a network, and why fine-tuning a large model can need only a small fraction of its parameters.

A matrix is a function

An $m \times n$ matrix $W$ takes a vector $x \in \mathbb R^n$ to $Wx \in \mathbb R^m$. Each output component is a dot product of one row of $W$ with $x$:

The map is linear: $W(ax + by) = a\,Wx + b\,Wy$. Linearity means the whole map is determined by what it does to the basis vectors $e_1, \dots, e_n$, and $We_j$ is simply the $j$-th column of $W$. So the columns are where the axes go, and $Wx$ is the combination of those columns weighted by the entries of $x$:

This column picture is the most useful single idea in applied linear algebra.^[1]

Worked example: three maps of the plane

The third matrix sends both axes onto the same line (the second column is twice the first), so every input lands somewhere on the line through $(1,2)$. For example, $(2, -1) \mapsto (2 - 2, 4 - 4) = (0, 0)$. A whole direction of inputs is mapped to zero and cannot be recovered from the output.

A layer is an affine map plus a nonlinearity

A fully connected layer computes $h = \sigma(Wx + b)$: a linear map, a shift by the bias $b$, and an elementwise nonlinearity $\sigma$ such as ReLU. A transformer is dominated by such maps: the query, key, value and output projections in attention and the two large matrices of each feed-forward block.^[2] Most of a large model's parameters and most of its arithmetic live in these matrices.

Composition is multiplication

Applying $W_1$ and then $W_2$ is the single map $W_2 W_1$: matrix multiplication is defined so that $(W_2 W_1)x = W_2(W_1 x)$. Shapes must chain: an $m \times k$ matrix times a $k \times n$ matrix gives $m \times n$.

This has an immediate consequence. Stack any number of linear layers without nonlinearities and the result is a single matrix:

Depth then adds nothing to what the network can represent; it can still only compute linear functions. The nonlinearity between layers is what breaks the collapse. With it, even a single hidden layer of sigmoidal units can approximate any continuous function on a bounded region to arbitrary accuracy given enough units, as Cybenko proved in 1989.^[3]

Order matters in two ways. Matrix multiplication is not commutative ($W_2 W_1 \ne W_1 W_2$ in general: rotate-then-stretch differs from stretch-then-rotate). And although $(AB)x = A(Bx)$, the costs differ: forming $AB$ first and then multiplying by $x$ can take far more arithmetic than two matrix–vector products. The adapter method below depends on choosing the cheap order.

Rank: how much of the space survives

The column space of $W$ is the set of all possible outputs $Wx$, all combinations of its columns. Its dimension is the rank of $W$. The null space is the set of inputs mapped to zero. For an $m \times n$ matrix, the rank–nullity theorem says

Every input dimension either survives into the output or is annihilated.^[1] In the worked example above, the flattening matrix has rank 1 and a one-dimensional null space (the line through $(2,-1)$).

Two facts follow for networks:

• Rank never exceeds the narrowest dimension. $\operatorname{rank}(W) \le \min(m, n)$, and for a product, $\operatorname{rank}(AB) \le \min(\operatorname{rank} A, \operatorname{rank} B)$. A composition of linear maps is at most as expressive as its narrowest bottleneck.
• Attention compares in a low-dimensional subspace. In the original transformer, $d_{\text{model}} = 512$ and each of 8 heads projects queries and keys down to $d_k = 64$ dimensions.^[2] The score $q \cdot k = x_i^\top W_Q W_K^\top x_j$ is a bilinear form whose $512 \times 512$ matrix $W_Q W_K^\top$ has rank at most 64. Each head can only measure similarity along 64 directions of the residual stream, which is one reason models use many heads.

Low-rank updates: LoRA

A rank-$r$ matrix of shape $d \times k$ can always be written as a product $BA$ with $B$ of shape $d \times r$ and $A$ of shape $r \times k$. Storing $B$ and $A$ takes $r(d + k)$ numbers instead of $dk$.

Hu and colleagues' LoRA (low-rank adaptation) uses exactly this to fine-tune large models cheaply. The pretrained weight $W_0$ is frozen, and the update is constrained to low rank:

$B$ starts at zero, so training begins from the pretrained model. The authors' hypothesis is that the weight change needed for adaptation has low intrinsic rank. They report that, compared with full fine-tuning of GPT-3 175B with Adam, LoRA reduced trainable parameters by a factor of 10,000 and GPU memory by a factor of 3, with quality on par or better on the models they tested.^[4] Because $BA$ is just a matrix, it can be added into $W_0$ after training, so the adapted model runs with no extra inference latency.

Worked example: counting parameters

Take one $4096 \times 4096$ projection matrix. Full fine-tuning updates $4096^2 = 16{,}777{,}216$ numbers. A rank-8 LoRA update trains $8 \times (4096 + 4096) = 65{,}536$, a factor of 256 fewer. During training the adapter is applied as $B(Ax)$: first an $8 \times 4096$ product, then a $4096 \times 8$ one, about $2 \times 8 \times 4096 = 65{,}536$ multiply-adds per input vector, rather than forming the full $4096 \times 4096$ matrix $BA$.

Which low-rank directions to choose, and why so few suffice, is a question about the singular value decomposition, the subject of the next lesson.

Key takeaways

• A matrix is a linear map; its columns are the images of the basis vectors, and $Wx$ is a weighted combination of those columns.
• Composing maps is multiplying matrices. Without nonlinearities, any stack of linear layers collapses to one matrix, so depth adds no expressive power.
• Rank is the dimension of what survives the map; it is capped by the narrowest dimension, and rank–nullity accounts for every lost direction.
• Attention heads compare tokens through low-rank bilinear forms, and LoRA fine-tunes by adding a low-rank product $BA$ to frozen weights.

Sources