Vectors, dot products and similarity

Neural networks represent words, images and whole documents as lists of numbers, and then decide what is related to what by multiplying those lists together. The dot product is that multiplication, and understanding its geometry explains embeddings, retrieval and attention.

Vectors as points and arrows

A vector in $\mathbb R^d$ is an ordered list of $d$ real numbers, $a = (a_1, \dots, a_d)$. It can be read as a point in $d$-dimensional space or as an arrow from the origin to that point. Vectors add component-wise, which places arrows tip to tail, and scale by multiplying every component by a number.

The length (Euclidean norm) of a vector is

Pythagoras' theorem extended to $d$ dimensions. A vector of length 1 is a unit vector; dividing any nonzero vector by its length normalises it.

In machine learning $d$ is large. A token embedding in a mid-sized transformer might have $d = 768$ or $4096$ components. None of these dimensions need correspond to anything human-interpretable; what matters is the geometry, meaning the lengths of vectors and the angles between them.

The dot product

The dot product of two vectors is the sum of their component-wise products:

Algebraically this is a single multiply-accumulate loop, the operation that modern accelerators are built to do quickly. Geometrically it has a second meaning:

where $\theta$ is the angle between the arrows. (This follows from the law of cosines applied to the triangle with sides $a$, $b$ and $a - b$.^[1]) So the dot product is large and positive when the vectors point the same way, zero when they are perpendicular (orthogonal), and negative when they point in opposite directions. It also grows with the vectors' lengths.

Two related quantities fall out immediately.

Cosine similarity removes the length:

Projection: the component of $b$ along the unit vector $\hat a = a/\|a\|$ is $\hat a \cdot b$. A dot product with a unit vector measures "how much of $b$ points in this direction". Most linear-algebraic readings of neural networks, from a classifier's decision boundary to a probe for a concept, are projections.

Worked example

Let $a = (1, 2, 2)$ and $b = (2, 0, 1)$.

• $a \cdot b = 1\cdot 2 + 2 \cdot 0 + 2 \cdot 1 = 4$.
• $\|a\| = \sqrt{1 + 4 + 4} = 3$, $\quad \|b\| = \sqrt{4 + 0 + 1} = \sqrt 5 \approx 2.236$.
• $\cos\theta = 4 / (3 \times 2.236) \approx 0.596$, so $\theta \approx 53°$.

Now double $b$ to $2b = (4, 0, 2)$. The dot product doubles to 8, but the cosine stays 0.596: the direction has not changed. Whether length should count is a modelling decision, and different systems decide differently.

Distance and similarity agree on the sphere

Expanding the squared distance gives

For unit vectors this is $2 - 2\cos\theta$. Ranking neighbours by smallest Euclidean distance and ranking by largest cosine similarity therefore give the same order once vectors are normalised, which is why vector databases can offer either metric on normalised embeddings with identical results.

From documents to word vectors

Representing text as vectors and comparing them with cosines is older than neural networks. Salton, Wong and Yang's 1975 vector space model represented each document as a vector of term weights and retrieved documents by their similarity to a query vector.^[2]

Neural embeddings learn the vectors instead of counting words. Mikolov and colleagues' word2vec models (2013) trained shallow networks to predict a word from its neighbours (continuous bag-of-words) or the neighbours from a word (skip-gram), and reported learning high-quality vectors from a 1.6-billion-word dataset in under a day.^[3] Words used in similar contexts end up with high cosine similarity. A companion paper reported that relationships appeared as roughly consistent vector offsets: the vector $\text{king} - \text{man} + \text{woman}$ lies closest, by cosine similarity, to $\text{queen}$.^[4] Analogy arithmetic works only approximately and for some relations, but it shows that directions in embedding space can carry meaning.

Dot products inside attention

A transformer's attention layer is a soft lookup driven by dot products. Each position produces a query vector $q$, and every position produces a key $k_j$ and a value $v_j$. The score for attending from the query to position $j$ is $q \cdot k_j / \sqrt{d_k}$; a softmax turns the scores into weights, and the output is the weighted average of the values.^[5] The $\sqrt{d_k}$ keeps the scores' spread independent of dimension (the variance argument appears in Expectation and concentration).

Attention uses the raw dot product, not the cosine, so a key's length acts as a volume control. A query that is perpendicular to a key gives that position a score of 0, not $-\infty$, so the position still receives some weight after the softmax. Positions only drop out when other scores are much larger. The full mechanism is the subject of Attention from first principles.

High dimensions behave differently

Intuition from two and three dimensions misleads in $\mathbb R^{768}$. If two vectors have independent random components with mean zero, their cosine similarity has mean zero and a standard deviation of about $1/\sqrt d$: roughly 0.036 at $d = 768$. Random high-dimensional vectors are almost always nearly orthogonal. That is good news for capacity, since a space can hold far more nearly-orthogonal directions than it has dimensions, a fact interpretability research leans on (see Mechanistic interpretability).

Real learned embeddings are not random, however. Ethayarajh found that contextual representations in ELMo, BERT and GPT-2 are anisotropic, occupying a narrow cone rather than spreading in all directions; in GPT-2's last layer, two randomly chosen words had nearly perfect average cosine similarity.^[6] A cosine of 0.9 can therefore mean "closely related" in one space and "unremarkable" in another. Similarity scores are meaningful relative to the distribution of scores in that particular space, not on an absolute scale.

Key takeaways

• A dot product is a sum of products, and geometrically $\|a\|\|b\|\cos\theta$: it measures alignment, scaled by length.
• Cosine similarity discards length; for unit vectors it ranks neighbours in exactly the same order as Euclidean distance.
• Embeddings (from the vector space model to word2vec) place related items at small angles; attention scores are scaled dot products between queries and keys.
• Random high-dimensional vectors are nearly orthogonal, but learned embedding spaces are often anisotropic, so judge similarity scores against their own baseline.

Sources