Inside the transformer block

A large language model is mostly one block repeated dozens of times. This lesson opens the block up: what each part does, where the parameters and FLOPs actually go, and which parts 2025–2026 architectures have replaced.

The block in two lines

A modern decoder-only transformer layer, in the "pre-norm" arrangement most models use, takes a matrix $X$ of token vectors (one row per token, width $d$) and computes

The full model embeds the tokens and applies $L$ such layers. A final normalization and an "unembedding" matrix then map each row to logits over the vocabulary, and the softmax turns those into next-token probabilities. Attention, covered in the previous lesson, is the only place where tokens exchange information. Everything else acts on each token independently.

The residual stream

The additions $X + \dots$ are residual connections. Read the block that way and the picture changes: the vector for each token is a running sum, and each sublayer reads from it and adds an update. Elhage et al. call this sum the residual stream. Attention heads and MLP layers read from and write to subspaces of a shared channel.^[1] Three consequences follow.

• Gradients have a direct path. The derivative of $X'$ with respect to $X$ includes an identity term, so signal reaches early layers even through a deep stack. That is the backpropagation argument behind residual networks.
• Layers compose additively. A layer's contribution can be studied as a vector added to the stream. Much of mechanistic interpretability rests on this.
• Width is a shared budget. Every layer communicates through the same $d$ dimensions, so features compete for space.

Normalization: where and how

The original transformer applied layer normalization after each residual addition ("post-LN"). Xiong et al. showed that post-LN gives large gradients near the output at initialization, which is why it needed a careful learning-rate warm-up. Moving the norm inside the residual branch ("pre-LN", as written above) keeps gradients well behaved and can train without warm-up.^[2] Most current models also replace LayerNorm with RMSNorm:

RMSNorm drops LayerNorm's mean subtraction and bias. Its authors found comparable quality with 7–64% less running time across their models.^[3]

The feed-forward network

The FFN is a two-layer MLP applied to each token separately. The original used a ReLU between an up-projection to width $d_{ff}$ and a down-projection back to $d$. Most current models use the gated SwiGLU variant:

This uses three matrices instead of two. Shazeer found that GLU-family variants gave lower perplexity than ReLU or GELU in transformer pretraining.^[4] Geva et al. argued that FFN layers behave like key–value memories: the input-side weights detect patterns such as n-grams or topics, and the matching output-side weights write associated directions back into the stream.^[5] In a mixture-of-experts model, this FFN is the part that is split into experts.

Worked example: counting Llama 3 70B

Where do 70 billion parameters live? Llama 3 70B has $L=80$ layers, $d=8192$, $d_{ff}=28{,}672$, 64 query heads, 8 key/value heads (head dimension 128) and a 128,000-token vocabulary.^[6]

Attention, per layer. $W_Q$ and $W_O$ are $8192\times8192$ each. Because of grouped-query attention, $W_K$ and $W_V$ are only $8192\times1024$ each:

FFN, per layer. Three matrices of $8192\times28{,}672$:

Total. Each layer has about $855.6$M parameters (the norms are negligible), so 80 layers give $68.4$B. The input embedding and the output unembedding add $2\times128{,}000\times8192 \approx 2.1$B, for about $70.5$B in all.

Two lessons follow. First, about 82% of each layer's parameters are in the FFN, not attention. Second, a forward pass costs about 2 FLOPs per parameter per token (one multiply and one add per weight), so roughly $1.4\times10^{11}$ FLOPs per token, plus an attention term that grows with context length. That "$2N$" rule underlies the $C\approx6ND$ training-compute estimate in scaling laws. The extra factor of 3 covers the backward pass.

What the block does not contain

There is no recurrence or memory between forward passes, other than the KV cache, which is an exact record of past keys and values. All mixing across positions happens in attention. That is why attention's cost and cache size dominate long-context engineering, and it is the target of most recent architectural change.

What changed in 2025–2026

The pre-norm residual skeleton has proved durable. The sublayers inside it have been changing.

• State-space and recurrent layers. Mamba introduced selective state-space layers whose parameters depend on the input. They run in linear time with a constant-size state, and the authors reported about 5× higher inference throughput than transformers.^[7] Pure recurrent models struggle with exact recall over long contexts, so production designs are hybrids. NVIDIA's Nemotron-H replaces most attention layers with Mamba layers and reports up to 3× faster inference than similarly sized transformers at comparable accuracy.^[8] Qwen3-Next stacks three Gated DeltaNet layers for every gated-attention layer, with an MoE FFN in every block.^[9] Kimi Linear uses a 3:1 ratio of its delta-rule linear attention to MLA.^[10]
• Gates inside attention. A sigmoid gate on the attention output, introduced by Qwen researchers, improved stability and removed the attention-sink artefact. It won a NeurIPS 2025 best paper award.^[11][12]
• Sparse FFNs. Mixture-of-experts FFNs are now standard in large open-weight models (see sparsity and compression).

These hybrids matter for hardware as well as quality. A layer whose weights are fixed and whose state does not grow with context is far easier to accelerate than attention's growing cache. NeuroGenesys's cost-model study of analog in-memory compute found that the window policy of a hybrid's attention layers decides whether the hardware advantage survives at long context. The fraction of layers that use attention matters less.^[13]

Key takeaways

• A transformer layer is attention plus a per-token FFN, each wrapped as a normalized residual update to a shared residual stream.
• Pre-norm placement and RMSNorm make deep stacks trainable without fragile warm-up. SwiGLU is the common FFN.
• In a dense model most parameters sit in the FFN (about 82% per layer for Llama 3 70B). Forward cost is about 2 FLOPs per parameter per token.
• Recent architectures keep the residual skeleton but swap sublayers: linear-recurrent and state-space layers for most attention, gated attention, and mixture-of-experts FFNs.

Sources