Mechanistic interpretability: features, circuits and superposition

Mechanistic interpretability tries to explain a trained network the way one would reverse-engineer a compiled program: identify the variables it represents and the operations that combine them. This lesson covers the core vocabulary, the tools built on it, and the state of the debate about how far those tools reach, as of November 2026.

Features and circuits

The circuits programme, set out by Olah et al. in 2020 for vision models, made three working claims: features are the fundamental unit of a network's computation; features are connected by weights into circuits that implement interpretable algorithms; and analogous features and circuits recur across different models.^[1] In a transformer, features are sought in the residual stream, the vector each layer reads from and writes to. A feature is then a direction in that space, and "how much of feature $i$ is present" is a projection onto it. (See /academy/linear-algebra-for-learning/vectors-and-similarity.)

The obstacle is that individual neurons are usually polysemantic: one neuron responds to several unrelated things.

Superposition

Elhage et al. (2022) gave a toy model in which polysemanticity can be fully understood. Small ReLU networks were trained to reconstruct many sparse synthetic features through a bottleneck narrower than the number of features. When features were rarely active, the networks stored more features than dimensions, assigning them nearly-but-not-exactly orthogonal directions and tolerating the resulting interference. Superposition appeared through abrupt phase changes as sparsity varied, with directions forming regular geometric structures. The authors cautioned that it is unclear how far a model this simple generalises to real networks.^[2]

Worked example. Store five features in two dimensions, at angles $0^\circ, 72^\circ, 144^\circ, 216^\circ, 288^\circ$ (a pentagon). Activate only feature 0 with value 1; its projection onto each direction is the cosine of the angle between them:

A linear readout would report spurious activity of 0.309 on features 1 and 4. Add a bias of $-0.309$ and a ReLU: the readouts become $0.691, 0, 0, 0, 0$. Feature 0 is recovered cleanly, at a small cost in magnitude. This works only because features are sparse: if features 0 and 2 are active together, feature 1, which lies between them, receives $0.309 + 0.309 = 0.618$ of interference; the bias removes only 0.309, and a spurious activation of 0.309 survives the ReLU. Superposition is a bet that features seldom co-occur.

In high dimensions the bet gets better. Random unit vectors in $\mathbb{R}^d$ have pairwise cosine with standard deviation about $1/\sqrt{d}$; for $d = 4096$ that is about 0.016, so a residual stream can hold far more nearly-orthogonal directions than it has coordinates.

Dictionary learning with sparse autoencoders

If a layer's activations $x \in \mathbb{R}^d$ are a sparse combination of many feature directions, one can try to recover them with a sparse autoencoder (SAE): an overcomplete dictionary with $m \gg d$ latents,

The reconstruction term keeps the dictionary faithful; the $\ell_1$ penalty keeps few latents active per input.

Three results established the method. Cunningham et al. (2023) showed SAE features on language-model activations scored as more interpretable than neurons and other decompositions.^[3] Bricken et al. (2023) decomposed a one-layer transformer's 512-neuron MLP into dictionaries of up to 131,072 features, analysing a 4,096-feature run in detail, and found largely monosemantic features where the neurons were not.^[4] Templeton et al. (2024) scaled the approach to the middle-layer residual stream of a production model, Claude 3 Sonnet, with SAEs of up to 34 million features. Features were multilingual and multimodal, responded to both concrete instances and abstract discussion, included safety-relevant concepts such as deception and sycophancy, and could steer behaviour — clamping a Golden Gate Bridge feature high made the model talk as if it were the bridge. The authors noted that the feature set was incomplete and that they lacked rigorous ways to check whether features faithfully captured the model's computation.^[5]

Circuit tracing and attribution graphs

Features say what is represented; circuits say how it is used. In 2025 Anthropic replaced a model's MLPs with a cross-layer transcoder — features that read from the residual stream at one layer and write to all later MLP outputs — and built attribution graphs: for one prompt, a graph of linear effects between active features from input tokens to the output. Unexplained residuals are kept as explicit "error nodes", so a graph is a hypothesis about the computation, to be tested by intervention.^[6]

Applied to Claude 3.5 Haiku, the method produced several mechanistic case studies.^[7]

• Two-hop reasoning. For "the capital of the state containing Dallas", the graph showed an intermediate "Texas" representation feeding "Austin", alongside a direct shortcut; swapping in features for another state changed the answer to that state's capital.
• Planning. When writing rhyming couplets, the model appeared to select a candidate rhyme word before writing the line; injecting a different planned word redirected the line, ending it with the injected word in 70% of 25 test poems.
• Arithmetic. Addition combined a low-precision estimate of the sum with a separate exact computation of the last digit.

The authors estimated that attribution graphs gave satisfying insight for about a quarter of the prompts they tried, and the published cases are successes rather than a random sample.^[7] A later paper extended the graphs to attention's query–key interactions.^[8]

The open disagreement: how far do SAEs get you?

The strongest critiques of dictionary learning come from inside the field, and they are substantive.

• Downstream usefulness. In March 2025 the Google DeepMind interpretability team reported that SAE-based probes underperformed plain linear probes at detecting harmful intent on out-of-distribution jailbreaks, that probes on SAE reconstructions did worse than probes on raw activations, and that they were deprioritising fundamental SAE research — while noting SAEs had helped in dataset debugging.^[9] In December 2025 the same team described a broader pivot from ambitious reverse-engineering towards "pragmatic" interpretability judged on concrete proxy tasks.^[10]
• Non-uniqueness. Paulo and Belrose (2025) found that SAEs trained on the same model and data, differing only in random seed, shared only about 30% of their features in one large configuration.^[11] Leask et al. (2025) found that larger SAEs do not converge on a canonical set of units: latents in bigger dictionaries decompose into combinations of smaller ones (an "Einstein" latent into "scientist", "Germany" and "famous person").^[12]
• Faithfulness of replacements. Circuit tracing explains a replacement model that matches the original's output only part of the time; error nodes mark where the explanation is missing.^[6]

The defensible position, as of November 2026, is that SAE features are a useful decomposition of activation space rather than a list of the model's true variables, and that interpretability claims should be validated by intervention and by performance on tasks where the right answer is known. One alternative is to train models to be interpretable in the first place: OpenAI researchers reported that weight-sparse transformers yield task circuits roughly 16× smaller than dense models at matched loss, while noting difficulty scaling beyond tens of millions of non-zero parameters.^[13]

Where the frontier is in 2025–2026

In an auditing study, three of four blinded teams uncovered a model's deliberately hidden objective, using SAE features among other techniques.^[14] Lindsey (2025) injected concept vectors into Claude's activations and found the model sometimes reported the injected "thought" before it influenced its output — about 20% of trials at the best settings for Claude Opus 4.1 — while stressing the ability is highly unreliable.^[15] 2026 papers from Anthropic report emotion-concept representations in Claude Sonnet 4.5 that causally influence outputs,^[16] autoencoders that translate activations into natural language,^[17] and a small set of reportable, controllable representations behaving like a "global workspace".^[18] These are recent single-lab results; treat them as promising rather than settled.

Key takeaways

• Features are directions in activation space; circuits are weighted connections between them. Neurons are often polysemantic because networks store sparse features in superposition.
• Sparse autoencoders recover many interpretable, steerable features at production scale, but their dictionaries are incomplete, seed-dependent and not canonical.
• Attribution graphs give testable, prompt-level hypotheses about computation, with explicit error terms; they explained about a quarter of prompts in the original study.
• The live disagreement is about usefulness: several 2025 results found simple linear probes matched or beat SAE methods on downstream tasks, prompting a turn to pragmatic, task-validated interpretability.

Sources