Inference economics: KV caches, memory bandwidth and batching

What sets the cost of a generated token? For autoregressive language models the answer is usually bytes moved, not arithmetic performed. This lesson builds the roofline model that explains why, works through the numbers for a 70B model, and shows how batching, paged caches, speculative decoding and disaggregation follow from it. It also covers why reasoning models made these questions matter more in 2025–2026.

Two phases with opposite bottlenecks

Serving a request has two phases.

• Prefill processes the whole prompt in one parallel pass and writes its keys and values to the KV cache. Each weight loaded from memory is used against thousands of tokens, so prefill is compute-bound.
• Decode generates one token per step for each sequence. Every step must read every weight (and the sequence's whole KV cache) to do a small amount of arithmetic. Decode is memory-bandwidth-bound.

Shazeer flagged the decode problem in 2019: incremental decoding is slow because of the memory bandwidth spent repeatedly loading keys and values.^[1] Pope et al. at Google formalized the trade-offs for serving 500B-parameter models. They showed how latency, batch size and chip partitioning interact, reporting 29 ms per token at small batch and 76% FLOP utilization for large-batch prompt processing on PaLM 540B.^[2]

The roofline in one ratio

A chip has peak compute $P$ (FLOP/s) and memory bandwidth $B$ (bytes/s). A kernel performing $F$ FLOPs on $M$ bytes has arithmetic intensity $I=F/M$. It is memory-bound whenever $I < P/B$, the chip's ridge point. An NVIDIA H100 SXM offers about 989 TFLOP/s of dense BF16 compute and 3.35 TB/s of HBM3 bandwidth,^[3] so

A decode step at batch size 1 does about 2 FLOPs per weight (see the transformer block) and reads each 2-byte BF16 weight once, so $I\approx1$. That is about 300× below the ridge. With a batch of $b$ sequences, each weight read serves $b$ tokens, so $I\approx b$ for the weight traffic. Batching is how decode climbs toward the ridge.

The gap is widening. Gholami et al. estimate that peak server FLOP/s grew about 3.0× every two years over the last two decades, against about 1.6× for DRAM bandwidth.^[4] NVIDIA's Rubin GPU, announced for 2026, pairs 50 PFLOP/s of NVFP4 inference compute with 22 TB/s of HBM4.^[5] That is roughly 2,300 low-precision FLOPs per byte.

Worked example: Llama 3 70B on eight H100s

Take a 70B model in BF16, so 140 GB of weights, sharded across 8 GPUs with an aggregate bandwidth of $8\times3.35 = 26.8$ TB/s. Its KV cache costs about 0.33 MB per token (computed in the attention lesson). Ignoring compute, which stays far below the ridge throughout, the time for one decode step is roughly

where $W$ is weight bytes, $b$ batch size, $n_{\text{ctx}}$ context length and $k$ KV bytes per token.

Batch 1. $t = 140\text{ GB}/26.8\text{ TB/s} \approx 5.2$ ms, so at most about 190 tokens/s. The GPUs are using about $1.4\times10^{11}/5.2\text{ ms}\approx 27$ TFLOP/s of their roughly 7,900 TFLOP/s, about 0.3%.

Batch 64, 8k context each. Each sequence's cache is $8192\times0.33\text{ MB}\approx2.7$ GB, so KV reads add $64\times2.7\approx172$ GB per step. Then $t\approx(140+172)/26.8\approx11.6$ ms, and throughput is $64/0.0116\approx5{,}500$ tokens/s, about 29× the batch-1 figure on the same hardware.

Two lessons follow. First, batching amortizes the weights but not the KV cache: each sequence reads its own. At long contexts, KV traffic overtakes weight traffic (172 GB against 140 GB here) and the benefit of batching flattens. Second, HBM capacity caps the batch. The cache must fit next to the weights.

Serving systems built on this arithmetic

Continuous batching. Requests finish at different times. The Orca system schedules at the granularity of a single model iteration, so new requests join the running batch as soon as any slot frees up. It reported up to 36.9× higher throughput than FasterTransformer on GPT-3 175B at equal latency.^[6]

Paged KV caches. Reserving a contiguous cache for each request's maximum length wastes memory and therefore batch size. vLLM's PagedAttention stores the cache in fixed-size blocks, like operating-system virtual memory, and reported 2–4× throughput gains over prior systems.^[7]

Prefill/decode disaggregation. Because the two phases bottleneck on different resources, DistServe runs them on separate GPUs and ships the KV cache between them. It reported up to 7.4× more requests served within latency targets.^[8] NVIDIA's Dynamo framework, released in 2025, implements the pattern. Its documentation notes that an aggregated deployment can still win at low concurrency, where the transfer overhead is not repaid.^[9] NeuroGenesys's review of interconnect measurements found that communication is a modest share of prefill time (measured at 9–23%) but becomes the binding constraint in multi-node decode. That is where faster interconnect, optical or otherwise, has its strongest case.^[10]

Speculative decoding: trading idle FLOPs for bytes

At batch 1 the GPU is about 99% idle on arithmetic, so extra computation is nearly free if it does not add memory traffic. Speculative decoding exploits this. A small draft model proposes $k$ tokens, and the large model scores all of them in one forward pass, which reads the weights once. A modified rejection-sampling rule accepts a prefix of the draft so that the output distribution is exactly the large model's.^[11][12] If each draft token is accepted with probability $\alpha$, the expected number of tokens per large-model pass is

which for $\alpha=0.8$, $k=4$ is about 3.4. Leviathan et al. reported 2–3× speedups on T5-XXL with identical outputs.^[11] EAGLE-3, which drafts from the target model's own hidden features, reported speedups of up to 6.5×.^[13] Multi-token prediction heads trained with the model (see next-token prediction) can serve as built-in drafters. The gain shrinks at large batch, where the FLOPs are no longer idle.

Why inference-time scaling raised the stakes

Reasoning models spend compute by generating long chains of tokens, and those are decode tokens: the expensive, bandwidth-bound kind. Usage data show the shift. In OpenRouter's study of more than 100 trillion tokens, models optimized for reasoning went from a negligible share of traffic in early 2025 to more than half of all tokens by late 2025. Average prompts grew about fourfold, to over 6,000 tokens, and completions nearly tripled.^[14] Prices at fixed capability have fallen fast. Epoch AI measured declines of 9× to 900× per year depending on the benchmark and threshold.^[15] But tokens consumed per task rose at the same time, so total serving demand still grew.

Model designers responded by attacking the KV cache and the attention FLOPs directly. DeepSeek-V4-Pro's model card reports that at a one-million-token context it needs 27% of the per-token FLOPs and 10% of the KV cache of its predecessor. It stores its expert weights in FP4.^[16] Sparsity and compression covers the weight side of the same bill.

Key takeaways

• Prefill is compute-bound. Decode is memory-bandwidth-bound, with arithmetic intensity of about 1 FLOP/byte at batch 1 against a ridge of about 300 on an H100.
• Batching amortizes weight reads, but each sequence's KV cache is read separately, so long contexts erode the benefit and HBM capacity caps the batch.
• Continuous batching, paged KV caches and disaggregation are systems responses to this arithmetic. Speculative decoding turns idle FLOPs into fewer weight reads.
• Reasoning models shifted traffic toward long, decode-heavy generations. As of November 2026, KV-cache and attention-FLOP reduction are first-class architectural goals.

Sources