6 — Why LLMs need all of this

Requirements: chapters 2 to 4, and basic matrix multiplication. Experiment 1 needs only a calculator. Experiments 2 and 3 need a GPU and vLLM.

Model inference has two phases: prefill, processing the prompt, and decode, generating tokens one at a time. One ratio explains how they differ.

The ridge point

The Roofline model compares a program with its hardware using operational intensity:

$I = \frac{\text{FLOPs executed}}{\text{bytes moved from memory}}$

(Practitioners often say “arithmetic intensity”, but the Roofline authors use operational intensity. It’s the same formula here.)

Now take an H100 SXM5:

• Dense FP16 peak: about 989 TFLOPS (FlashAttention-3)
• HBM3 bandwidth: 3.35 TB/s

Divide them:

$\frac{989 \times 10^{12}}{3.35 \times 10^{12}} \approx 295 \text{ FLOPs per byte}$

That’s the ridge point. Below 295, memory limits you. Above it, compute does.

The intensity of a linear layer

Take a weight matrix $W$ of size $D \times D$, multiplied by a batch $X$ of size $B \times D$, in FP16 (2 bytes per value).

1. FLOPs: each of the $B$ rows needs $D^2$ multiply-adds, or $2BD^2$ operations.
2. Bytes: reading $W$ costs $2D^2$. Reading $X$ costs $2BD$, and writing the result costs another $2BD$.

So:

$I = \frac{2BD^2}{2D^2 + 4BD} = \frac{BD}{D + 2B}$

Here’s how that plays out:

With $D = 8192$, you cross the ridge at about $B = 318$.

This is the key point. Being memory-bound depends on batch size, not on the word “decode”. With one active request, you get 1 FLOP per byte, so at most about 0.34% of peak FLOPS is reachable under this model. That’s a ceiling from arithmetic, not a measured idle percentage. Batching raises it.

A speed limit

Divide model size by memory bandwidth to get a bound on tokens per second, for one request, counting weights only:

These are model bounds, not benchmarks. Note that 140 GB doesn’t fit in one 80 GB H100, so that case needs sharding.

This is why quantization to FP8, INT4 or MXFP4 helps decode so much: fewer bytes per token. Peak FLOPS, on the other hand, mostly tells you about prefill.

Real kernels also stay below peak. FlashAttention-3 reaches 740 TFLOPs/s, or 75%, partly because softmax runs on a slower unit at 3.9 TFLOPS (FA3).

Why naive attention breaks

Self-attention, for sequence length $N$:

$\text{Attention}(Q, K, V) = \text{softmax}\left(\frac{QK^T}{\sqrt{d}}\right)V$

A naive implementation runs this as separate kernels, writing an $N \times N$ matrix to HBM between each. During prefill at $N = 128{,}000$, that matrix has $1.6 \times 10^{10}$ elements. It exhausts VRAM, and it saturates the bus with round trips. (A KV-cached decode step has only one query, so this applies to prefill.)

Avoiding HBM with FlashAttention

FlashAttention fuses the equation into a single kernel, using ideas from chapters 2 and 4:

1. Split $Q, K, V$ into tiles that fit in shared memory, which runs at 31 TB/s, about 9× faster than HBM.
2. Use the online softmax trick (Milakov and Gimelshein), keeping running maximums and scaling factors in registers.
3. Never write the $N \times N$ matrix to HBM.

FlashAttention-3 adds warp specialization on Hopper: one group fetches tiles with TMA while others compute. FlashAttention-4 reaches 1613 TFLOPs/s on B200, or 71%, partly emulating exponentials because that unit didn’t scale with Tensor Cores (Together AI, paper).

Keeping data high in the memory table is the entire idea.

The KV cache as virtual memory

To avoid recomputing old tokens, a server keeps their keys and values in the KV cache. Sequence lengths are unpredictable, so older systems reserved contiguous space for the maximum length. That wasted 61.8–79.6% of the KV cache, including reserved slots (PagedAttention paper).

If you’ve read the page fault chapter, you know the fix. PagedAttention (SOSP ‘23) splits the cache into fixed blocks, 16 tokens by default, allocated on demand. A block table maps logical blocks to physical ones.

What this actually gave:

• Throughput rose 2–4× (22× against FasterTransformer).
• External fragmentation goes away, while internal fragmentation stays, bounded by block size.
• Kernel latency actually rose 20–26%. The gain is at system level, through bigger batches.

The original custom kernel used a block table in global memory, with a computed index and one warp per block (v0.8.5 source). It’s now history: vLLM marks that document “historical” and defaults to FlashAttention versions (attention_backends.md). The paging idea lives on.

Past one GPU

When a model needs several GPUs, tensor parallelism splits each layer, so GPUs must combine partial results constantly.

Compare like with like: NVLink on Hopper gives 450 GB/s each way, against 64 GB/s each way on PCIe Gen5 (Hopper architecture). Blackwell’s NVLink doubles the per-GPU total to 1.8 TB/s.

With NCCL, each GPU can issue a multimem.ld_reduce to a shared address and get the reduced value back. multimem.st handles broadcasts, and SHARP engines in NVSwitch do the reduction in the network (PTX ISA, NVLink).

One token, end to end

1. The serving engine forms a batch and runs fused kernels.
2. The GPU spreads them across SMs. Copy engines stream tiles into shared memory, and warps issue tensor instructions.
3. The interconnect combines results across GPUs without involving the CPU.
4. Your tools watch it from outside, as in chapter 5.

The next chapter zooms out to how the whole machine is packaged.

Hands-on

Experiment 1: rebuild the Roofline table

Needs: a calculator or Python.

1. Compute 989 / 3.35.
2. Calculate $I = BD/(D+2B)$ for $D = 8192$ and $B$ = 1, 32, 256 and 4096.
3. Solve for the $B$ where $I = 295$.
4. Compare with the table above. Then repeat it using your own GPU’s numbers from chapter 2.

Experiment 2: observe prefill and decode

Needs: a GPU and Python 3.12.

1. Install vLLM and serve a small model (quickstart):

   uv venv --python 3.12 --seed
   uv pip install vllm --torch-backend=auto
   vllm serve Qwen/Qwen2.5-1.5B-Instruct

2. Save the metrics before and after each run:

   curl -s http://localhost:8000/metrics | grep -E 'vllm:(request_prefill_time_seconds|request_decode_time_seconds|time_to_first_token_seconds|kv_cache_usage_perc)'

3. Prefill sweep. Send one request at a time, with a fixed short max_tokens and longer and longer prompts. Expected: prefill time grows with prompt length.

4. Decode sweep. Use a short prompt with a long max_tokens. Expected: time per output token stays roughly flat.

5. Batch sweep. Raise concurrency. Expected: total throughput rises, while per-request speed doesn’t improve.

6. Expected: compute model bytes ÷ your bandwidth. Your measured time per token must be above that floor.

nvidia-smi dmon -s um -d 1 is useful alongside this, but its sm figure only says whether a kernel ran during the sample. It doesn’t show Tensor Core load, so don’t treat it as proof.

Experiment 3: watch the KV cache fill

Needs: a GPU and vLLM.

1. Before changing --block-size, check attention_backends.md. Some kernels want 128.
2. Raise the number of concurrent long prompts until vllm:kv_cache_usage_perc nears 1, and watch vllm:num_preemptions.
3. Expected: usage should grow with active tokens, not with reserved max_tokens. That’s the paging idea made visible.

multimem can’t be a home experiment, because it needs NVSwitch.