# Speculative decoding: let a small model do the guessing
> Decode is bandwidth-bound, so the GPU's maths units sit half-idle one token at a time. Speculative decoding spends that idle compute to verify several guessed tokens at once, for the same output, faster.
Source: https://rfriedmann.de/blog/speculative-decoding/
Published: 2026-06-22 · Track: log · Level: Expert
[Decode is bandwidth-bound](/blog/prefill-vs-decode-roofline/): generating one token
means streaming the entire model's weights through the GPU to produce a single word,
and while all that memory traffic happens, the chip's maths units sit mostly idle.
That's the roofline talking, and it's the central frustration of single-stream
inference. The weights cost the same to load whether you're computing one token or
ten; you're paying full freight for the bandwidth and using a sliver of the compute.
Speculative decoding is the trick that spends that wasted compute. The output is
*identical* to normal sampling, token for token in distribution, but it arrives
faster. It's one of the few free lunches in this field, and it falls straight out of
the roofline once you see it.
## The idea: guess cheap, verify in bulk
Two models. A small, fast **draft** model and the big **target** model you actually
want answers from. The loop:
1. The draft model cheaply guesses the next handful of tokens, say 4, one after
another. It's small, so this is quick, and quick is the whole point.
2. The target model checks all 4 guesses **in a single forward pass**, in parallel,
the same way [prefill](/blog/prefill-vs-decode-roofline/) processes many tokens at
once.
3. You keep the longest correct prefix, and the moment a guess is wrong, you discard
it and the rest, taking the target's own token there instead.
The magic is in step 2, and it's pure roofline. Verifying 4 tokens loads the target's
weights *once*, exactly as loading them for 1 token does, because the bottleneck was
bandwidth, not maths. So you check 4 guesses for nearly the price of generating 1. If
the draft got 3 of them right, you just emitted 4 tokens (3 accepted + 1 from the
target) for the bandwidth cost of one decode step. That's the speedup, and it's the
idle compute from the roofline finally doing something.
One target pass, four tokens checked at once
[diagram omitted — see the page for the chart]
The draft guessed "the cat sat quickly"; the target agreed up to "sat", rejected "quickly" and supplied "on" instead. Four tokens move forward for the bandwidth cost of one decode step. When the draft is wrong early, you fall back toward one token per pass, never worse than plain decode.
## The part that makes it honest: no quality loss
The natural worry: doesn't trusting a small model's guesses degrade the output? It
doesn't, and that's what separates this from "just use a smaller model." The
verification step isn't a vibe check, it's a precise piece of rejection sampling.
For each proposed token, you compare the draft's probability for it against the
target's. If the target likes it at least as much, accept. If less, accept it only
with probability `p_target / p_draft`, and on rejection, draw the replacement from a
carefully corrected residual distribution. The maths works out so that the stream of
tokens you emit is **provably distributed exactly as if you'd sampled from the target
model alone**. The draft model only ever influences *speed*, never *what* gets
produced. You get the big model's answer; you just get it sooner.
That guarantee is the whole reason this is worth doing rather than a cheap
approximation. The user-visible output is bit-for-bit the target's distribution; the
draft is invisible except on the clock.
## What decides the speedup
It's not free real estate; the gain hinges on two competing quantities.
- **Acceptance rate, α.** How often the draft guesses right. A draft that mirrors the
target well (same family, similar training) gets accepted often, so long runs of
tokens fly through. A poorly matched draft gets rejected early and you barely beat
plain decode.
- **Draft cost.** The draft runs sequentially and adds its own latency. If it's too
big, its cost eats the savings; too small and it's a poor guesser. There's a sweet
spot, often a draft roughly 10-20x smaller than the target.
The two pull against each other, which is why real-world speedups land around **2 to
3x** for decode rather than the 4x you'd get from a perfect draft. Predictable text,
boilerplate, code with obvious continuations, accepts more readily; genuinely
surprising text accepts less. The number is workload-dependent, and any honest
benchmark says which workload it measured.
## Drafts without a second model
Running a separate draft model is the classic form, but it costs memory you may not
have spare on a [single consumer card](/blog/what-the-5090-lacks-vs-datacenter/).
Several variants drop the second model entirely:
- **Prompt lookup / n-gram.** For tasks with lots of repetition (summarising or
editing a document, where chunks of the input reappear verbatim in the output), just
guess the next tokens by copying from earlier text. No model, near-zero cost,
surprisingly effective on the right workload.
- **Self-speculation (Medusa, EAGLE and kin).** Bolt a few extra lightweight heads
onto the target model itself so it predicts several tokens ahead in one go, then
verify as usual. No separate weights to host; the draft *is* the target wearing
extra heads.
All of them share the one structural idea: turn slow, bandwidth-bound, one-at-a-time
decode into a few cheap guesses graded by one parallel, compute-friendly pass. It's
the roofline, exploited on purpose.
## Why it belongs in the toolbox
Speculative decoding doesn't make the GPU faster or the model smaller. It notices that
single-stream decode leaves the maths units idle and puts them to work *verifying*
instead of *generating*, converting spare compute into latency you actually feel. It's
the cleanest example of a recurring theme in this log: once you can [name the bottleneck
precisely](/blog/prefill-vs-decode-roofline/), the optimisation often writes itself,
and the best ones cost you nothing in quality. For a single card chasing
[300 tokens a second](/blog/serving-30b-models-rtx-5090/), that's exactly the kind of
free lunch worth setting the table for.