Transformers & Language Models

What a language model is, how attention and the transformer block work, and what scaling laws do — and don't — tell us about training at frontier scale.

4 lessons • 30 min read

1. 01

   Next-token prediction: what a language model is

   A language model is a probability distribution over sequences of text. In practice it is built from one narrow skill: given everything so far, predict the next token. This lesson explains why that single objective is enough to train a chat assistant, a code generator and a reasoning model, and what it costs at inference time.

   8 min read
2. 02

   Attention from first principles

   Attention is how a transformer lets each token look up information from other tokens: a soft, differentiable lookup table built fresh for every input. This lesson derives the mechanism, explains its quadratic cost and its memory footprint at inference, and surveys how 2025–2026 models changed it to reach million-token contexts.

   8 min read
3. 03

   Inside the transformer block

   A large language model is mostly one block repeated dozens of times. This lesson opens the block up: what each part does, where the parameters and FLOPs actually go, and which parts 2025–2026 architectures have replaced.

   7 min read
4. 04

   Scaling laws and compute-optimal training

   Given a fixed compute budget, how large should a model be and how many tokens should it see? Scaling laws answer this with fitted power laws. They have steered billion-dollar training runs since 2020, and the right answer has shifted twice since then: first when Chinchilla corrected the allocation, and again when inference cost and test-time compute entered the budget.

   7 min read