The Geometry of Learning

Linear algebra as the language of neural networks: vectors and similarity, matrices as maps, and the eigenvectors and singular vectors that reveal what a layer does.

3 lessons • 21 min read

1. 01

   Vectors, dot products and similarity

   Neural networks represent words, images and whole documents as lists of numbers, and then decide what is related to what by multiplying those lists together. The dot product is that multiplication, and understanding its geometry explains embeddings, retrieval and attention.

   7 min read
2. 02

   Matrices as maps: layers, rank and composition

   A neural network layer is, before its nonlinearity, a matrix multiplying a vector. Reading that matrix as a map that moves, stretches and collapses space explains why depth needs nonlinearities, why some information is irrecoverably lost inside a network, and why fine-tuning a large model can need only a small fraction of its parameters.

   7 min read
3. 03

   Eigenvectors and the SVD: the directions that matter

   Most matrices in a trained network are large, but their action is often dominated by a handful of directions. Eigenvectors and the singular value decomposition find those directions, say how much each one matters, and give the provably best way to throw the rest away.

   7 min read