Speculative decoding: let a small model do the guessing

Decode is bandwidth-bound: generating one token means streaming the entire model’s weights through the GPU to produce a single word, and while all that memory traffic happens, the chip’s maths units sit mostly idle. That’s the roofline talking, and it’s the central frustration of single-stream inference. The weights cost the same to load whether you’re computing one token or ten; you’re paying full freight for the bandwidth and using a sliver of the compute.

Speculative decoding is the trick that spends that wasted compute. The output is identical to normal sampling, token for token in distribution, but it arrives faster. It’s one of the few free lunches in this field, and it falls straight out of the roofline once you see it.

The idea: guess cheap, verify in bulk

Two models. A small, fast draft model and the big target model you actually want answers from. The loop:

The draft model cheaply guesses the next handful of tokens, say 4, one after another. It’s small, so this is quick, and quick is the whole point.
The target model checks all 4 guesses in a single forward pass, in parallel, the same way prefill processes many tokens at once.
You keep the longest correct prefix, and the moment a guess is wrong, you discard it and the rest, taking the target’s own token there instead.

The magic is in step 2, and it’s pure roofline. Verifying 4 tokens loads the target’s weights once, exactly as loading them for 1 token does, because the bottleneck was bandwidth, not maths. So you check 4 guesses for nearly the price of generating 1. If the draft got 3 of them right, you just emitted 4 tokens (3 accepted + 1 from the target) for the bandwidth cost of one decode step. That’s the speedup, and it’s the idle compute from the roofline finally doing something.

One target pass, four tokens checked at once

The draft guessed "the cat sat quickly"; the target agreed up to "sat", rejected "quickly" and supplied "on" instead. Four tokens move forward for the bandwidth cost of one decode step. When the draft is wrong early, you fall back toward one token per pass, never worse than plain decode.

The part that makes it honest: no quality loss

The natural worry: doesn’t trusting a small model’s guesses degrade the output? It doesn’t, and that’s what separates this from “just use a smaller model.” The verification step isn’t a vibe check, it’s a precise piece of rejection sampling.

For each proposed token, you compare the draft’s probability for it against the target’s. If the target likes it at least as much, accept. If less, accept it only with probability p_target / p_draft, and on rejection, draw the replacement from a carefully corrected residual distribution. The maths works out so that the stream of tokens you emit is provably distributed exactly as if you’d sampled from the target model alone. The draft model only ever influences speed, never what gets produced. You get the big model’s answer; you just get it sooner.

That guarantee is the whole reason this is worth doing rather than a cheap approximation. The user-visible output is bit-for-bit the target’s distribution; the draft is invisible except on the clock.

What decides the speedup

It’s not free real estate; the gain hinges on two competing quantities.

Acceptance rate, α. How often the draft guesses right. A draft that mirrors the target well (same family, similar training) gets accepted often, so long runs of tokens fly through. A poorly matched draft gets rejected early and you barely beat plain decode.
Draft cost. The draft runs sequentially and adds its own latency. If it’s too big, its cost eats the savings; too small and it’s a poor guesser. There’s a sweet spot, often a draft roughly 10-20x smaller than the target.

The two pull against each other, which is why real-world speedups land around 2 to 3x for decode rather than the 4x you’d get from a perfect draft. Predictable text, boilerplate, code with obvious continuations, accepts more readily; genuinely surprising text accepts less. The number is workload-dependent, and any honest benchmark says which workload it measured.

Drafts without a second model

Running a separate draft model is the classic form, but it costs memory you may not have spare on a single consumer card. Several variants drop the second model entirely:

Prompt lookup / n-gram. For tasks with lots of repetition (summarising or editing a document, where chunks of the input reappear verbatim in the output), just guess the next tokens by copying from earlier text. No model, near-zero cost, surprisingly effective on the right workload.
Self-speculation (Medusa, EAGLE and kin). Bolt a few extra lightweight heads onto the target model itself so it predicts several tokens ahead in one go, then verify as usual. No separate weights to host; the draft is the target wearing extra heads.

All of them share the one structural idea: turn slow, bandwidth-bound, one-at-a-time decode into a few cheap guesses graded by one parallel, compute-friendly pass. It’s the roofline, exploited on purpose.

Why it belongs in the toolbox

Speculative decoding doesn’t make the GPU faster or the model smaller. It notices that single-stream decode leaves the maths units idle and puts them to work verifying instead of generating, converting spare compute into latency you actually feel. It’s the cleanest example of a recurring theme in this log: once you can name the bottleneck precisely, the optimisation often writes itself, and the best ones cost you nothing in quality. For a single card chasing 300 tokens a second, that’s exactly the kind of free lunch worth setting the table for.