Prefill, decode, and the roofline that explains everything
An inference engine doesn't have one speed, it has two, and they obey opposite laws. The roofline model is the single most useful lens for reasoning about LLM performance.
People quote “tokens per second” as if an engine has one speed. It has two, and they’re governed by opposite limits. Once you see why, most of the surprising numbers in LLM inference stop being surprising. This is the lens I reach for first, and it’s worth installing properly.
If you need the gentle version of the two phases, the engine walkthrough has it. Here we go straight for the why.
Two ceilings
A GPU has two separate speed limits. How fast it can do maths (measured in TFLOP/s), and how fast it can move data in and out of memory (measured in TB/s). For any given piece of work, you hit one of them first, and that’s the one that caps you. The deciding factor is a single ratio.
Plot speed against that ratio and you get the famous “roofline”: a sloped line (when you’re memory-limited, more intensity buys more speed) that flattens into a ceiling (once you’re compute-limited, more intensity buys nothing).
Prefill lives on the right, decode on the left
Now place the two phases on that picture.
Prefill processes the whole prompt at once. Every weight it loads gets used against many tokens before being thrown away, so it does a lot of maths per byte: high arithmetic intensity. It sits on the flat part, pushing against the compute ceiling. Hand it more tokens at once and it gets more efficient.
Decode writes one token at a time. Each weight is loaded, used exactly once, and discarded. That’s very low arithmetic intensity (about 1 FLOP/byte in fp16), so decode is pinned to the sloped part, limited purely by memory bandwidth. The compute units spend most of their time idle, waiting for the next weight to arrive.
”Compute is free”
Put a hard number on it. For a typical decode step on a 5090 the arithmetic intensity is about 1 FLOP/byte, against a ridge point of roughly 60 to 120 FLOP/byte. The ridge is just peak FLOP/s divided by peak bandwidth: with the 5090’s 1.79 TB/s and roughly 105 TFLOP/s dense fp16 (fp32-accumulate) up to ~210 (fp16-accumulate), that lands around 60 to 120 FLOP/byte. Decode sits about two orders of magnitude to the left of it. The docs phrase it bluntly: compute is free. The chip is almost never doing maths; it is almost always waiting for bytes.
That bandwidth ceiling translates straight into a token rate. An 8B model at fp16 is about 16 GB of weights, and you stream all of them once per token, so 1.79 TB/s divided by 16 GB caps you near 110 tok/s (and roughly 220 tok/s at int8, which halves the bytes). The math ceiling never enters the picture.
That has a sharp consequence. For single-stream decode, the only lever that matters is moving fewer bytes, faster. It’s exactly why quantisation helps decode so much, and why a whole list of clever-sounding optimisations do nothing for it. In imp’s testing, all three were duds for decode:
- FP8 scores in attention. The compute is already idle, so making it cheaper changes nothing.
- Warp specialisation. All the warps are already streaming memory at peak; there’s no overlap left to win.
- Fancier memory instructions for the KV cache. The plain async copy is already near the bandwidth ceiling.
None of them move bytes, so none of them move decode.
Why this lens matters
This is the whole reason an engine that’s fast at prefill isn’t automatically fast at decode, and vice versa. They live under different ceilings and want different kernels. It’s why the hard-won kernel work splits cleanly along that line.
And it’s why batching changes everything. Serve many users at once and their decode streams share each loaded weight, dragging decode rightward into the compute region where the idle maths finally gets used. That’s the data centre’s trick. A single person running a model at home gets none of it, which is why, for batch-1 decode of the weights, bandwidth is the dominant lever, and why a card’s memory speed matters more than its raw compute. That holds for streaming the weights specifically; it leaves out prefill (compute-bound) and long-context attention, where the KV-cache traffic becomes its own bottleneck. The numbers all fall out of this one picture.