Efficient AI: Inference, Sparsity & Hardware

Where the cost of AI actually goes — memory bandwidth, KV caches, batching — and the methods and hardware that cut it, including the lab's own analog and photonic results.

3 lessons • 27 min read

1. 01

   Inference economics: KV caches, memory bandwidth and batching

   What sets the cost of a generated token? For autoregressive language models the answer is usually bytes moved, not arithmetic performed. This lesson builds the roofline model that explains why, works through the numbers for a 70B model, and shows how batching, paged caches, speculative decoding and disaggregation follow from it. It also covers why reasoning models made these questions matter more in 2025–2026.

   8 min read
2. 02

   Mixture-of-experts, quantization and distillation

   There are three main ways to make a language model cheaper per token without making it proportionally worse. Activate only part of it for each token (mixture-of-experts). Store and move fewer bits per weight (quantization). Train a small model to imitate a large one (distillation). By late 2026 the leading open-weight models use all three at once. This lesson explains how each works, what it actually saves, and where it stops paying.

   9 min read
3. 03

   Beyond the GPU: analog, photonic and in-memory compute

   If moving data costs far more than computing on it, could a different physical substrate break the deadlock? Two candidates are analog crossbars that compute where the weights are stored, and light that multiplies matrices as it propagates. This lesson explains the physics behind both and the evidence for them as of November 2026, including a long series of NeuroGenesys falsification studies. Most of those studies came back negative or narrowly bounded, and they are reported that way here.

   10 min read