# Speculative decoding when the chats overlap
> Qwen3.8-27B ships its own draft head. On imp it sped up one chat by 35 %, and any second chat or UI side request switched it off. Nine changes later, four overlapping chats decode at about 140 tok/s instead of 84.
Source: https://rfriedmann.de/blog/mtp-when-chats-overlap/
Published: 2026-11-04 · Track: log · Level: Expert
Qwen3.8-27B comes with a small extra block, the MTP head, that guesses the token
after the next one. On a single stream imp has used it since August: the head
drafts, the main model checks the draft in the same step, and about 70 % of the
drafts are accepted. That is [speculative decoding](/blog/speculative-decoding/)
with the draft model built in.
Real chat traffic is not a single stream. A chat UI fires a side request between
two turns to name the conversation. A second person opens a second chat. Every one
of those cases switched the head off, for a different reason each time. This post
walks through the nine changes, all merged on 3 and 4 November 2026, that keep it
drafting when chats overlap. Every number is from the A/B table of the pull
request named next to it: Qwen3.8-27B-NVFP4 on one RTX 5090, a fresh server per
arm, three alternating pairs.
| Step | PR | what was wrong | kind |
|---:|---|---|---|
| 1 | [#3029](https://github.com/kekzl/imp/pull/3029) | head not loaded on a concurrent server | off |
| 2 | [#3030](https://github.com/kekzl/imp/pull/3030) | one head slot, the UI's side request took it | off |
| 3 | [#3031](https://github.com/kekzl/imp/pull/3031) | a batched step did not feed the head | off |
| 4 | [#3034](https://github.com/kekzl/imp/pull/3034) | a batched step could not check a draft | value |
| 5 | [#3037](https://github.com/kekzl/imp/pull/3037) | one draft per request | value |
| 6 | [#3039](https://github.com/kekzl/imp/pull/3039) | the guess read the FP8 LM head | price |
| 7 | [#3045](https://github.com/kekzl/imp/pull/3045) | the head pass was 56 launches | price |
| 8 | [#3047](https://github.com/kekzl/imp/pull/3047) | the draft kernel read one weight row per block | price |
| 9 | [#3048](https://github.com/kekzl/imp/pull/3048) | the pay-off guard ignored batched verifies | off |
"off" means the head was switched off without anyone noticing, "value" and
"price" mean it was on but paid less than it could.
## Why this model makes it hard
On a pure transformer, a rejected draft costs nothing to undo: the
[KV cache](/blog/kv-cache-and-long-context/) just drops its last row. Qwen3.8-27B
is a hybrid: 16 attention layers and 48 Gated DeltaNet layers, which carry a
[recurrent state](/blog/when-the-model-is-not-a-transformer/) instead of a cache.
That state is overwritten with every token, so there is nothing to truncate.
imp's answer is a spare state slot. A verify step writes the state after "last
token plus draft" into a spare and the state after "last token only" in place. If
the draft is accepted, the request's live slot becomes the spare. If not, it stays
where it was. No copies, but every request that verifies needs its own spare, and
on this model one slot is 79.5 MiB.
Draft accepted
2 tokens emitted
spare slot becomes the live slot
old live slot becomes the spare
Draft missed
1 token emitted
spare slot is discarded
live slot already holds the right state
The head has state of its own too: a KV cache of the hidden rows it has seen. A
request can only draft while that cache is in step with its context. Most of the
failures below were exactly that cache falling out of step.
## Step 1: the head was not even loaded
`speculative.mtp_k=auto` loaded the head only when the server ran with a maximum
batch of 1. A default `imp-server` allows 32 concurrent requests, so it served
this model without drafts. [#3029](https://github.com/kekzl/imp/pull/3029) loads
the head on a concurrent server when the model is a GDN hybrid. A prompt that
prefills alone binds the head and drafts while it decodes alone; a batch of
several decodes plain.
| Default server | before | after |
|---|---:|---:|
| 1 stream, tok/s | 96.87 | **131.28** |
| 4 concurrent bursts, tok/s | 341.3 | 339.7 |
| 32 concurrent bursts, tok/s | 2043.5 | 2042.0 |
| own peak VRAM, MiB | 22034 | 24807 |
One stream got 35.5 % faster with 2552 of 3644 drafts accepted. The price is
2773 MiB and a smaller KV pool: 15651 blocks down to 13230.
## Step 2: one slot, and the UI took it
The head had one KV slot. The chat UI's side request between two turns
overwrote it. The next turn resumed from the prefix cache with 464 to 2272
cached tokens, but the head needed hidden rows for those tokens that the cached
prefix never produced. So it unbound, and turns 3 to 6 decoded without drafts.
[#3030](https://github.com/kekzl/imp/pull/3030) gives the head 4 slots and picks
the one whose history covers the resume.
| 6-turn conversation | before | after | follow-up turns drafting |
|---|---:|---:|---|
| greedy chat + UI side request | 101.88 tok/s | **122.28** | 3/15 → 15/15 |
| two chats alternating | 101.52 tok/s | **117.43** | 6/30 → 30/30 |
| plain greedy chat (control) | 119.93 tok/s | 119.60 | 15/15 → 15/15 |
The head's K+V grew from 64 to 256 MiB.
## Step 3: a batched step left the head behind
When two chats decode at the same moment, they share one batched step, and that
step did not feed the head. The bound request's head cache fell behind its
context, so the next lone step unbound it. [#3031](https://github.com/kekzl/imp/pull/3031)
copies each bound request's hidden row out of every batched step into a carry
buffer (256 rows of 10 KiB per slot) and feeds the head from it later, in order.
| 2 overlapping chats, 6 turns each | before | after |
|---|---:|---:|
| turns drafting | 6/36 | 35/36 |
| mean tok/s | 93.78 | **98.38** |
The head now drafts on 35 of 36 turns. The gain is still small, because a batched
step can carry the hidden rows but cannot use a draft yet.
## Step 4: verify the drafts together
Decode on this card is [bound by memory bandwidth](/blog/prefill-vs-decode-roofline/).
At 32 streams, 12.06 ms of a 14.75 to 14.98 ms step is the time to stream the
weights, the LM head and the recurrent state from DRAM once. Adding a few rows to
a step that small barely changes its price.
[#3034](https://github.com/kekzl/imp/pull/3034) makes the step that runs 2 to 4
decoding requests forward each bound request's last token *plus* its draft.
| Step price | rows | time | tokens out |
|---|---:|---:|---:|
| plain decode, 4 requests | 4 | 11.72 ms | 4 |
| bound verify, 4 requests | 8 | 13.12-13.65 ms forward + 2.02-2.65 ms head | 7.15-7.20 |
One forward for everyone
Each bound request contributes its last token and its draft. Attention gets one block table per row, the GDN layers write the draft state into the request's spare slot.
One argmax over all rows
LM head and argmax run over every row, one copy to the host.
Accept per request
A request whose draft matched takes both tokens and swaps its live state slot to the spare. One that missed keeps one token and its old slot.
One head pass for everyone
The head is fed every new (token, hidden row) pair in one ragged pass and drafts the next token for each request.
| | before | after |
|---|---:|---:|
| 2 overlapping chats, tok/s per turn | 94.21 | **117.06** |
| 4 overlapping chats | 83.72 | **117.45** |
| turns drafting, 4 chats | 13/72 | 72/72 |
| 4 concurrent bursts | 337.0-338.1 | **457.2-461.8** |
| 32 concurrent bursts | 2020.2-2030.6 | 2029.8-2034.7 |
The same PR fixed two defects in the older opt-in path: its row cap read the
unresolved `max_batch_size` of 0, so every step declined, and its batched draft
needed an LM head this model's export does not build.
## Steps 5 to 8: make each verify worth more and cost less
With the batched verify in place, the rest is price per step.
| PR | change | 4 overlapping chats, tok/s |
|---|---|---|
| [#3037](https://github.com/kekzl/imp/pull/3037) | 2 drafts per request, 3 rows each | 121.59 → **134.17** |
| [#3039](https://github.com/kekzl/imp/pull/3039) | drafts read the 682 MiB NVFP4 head instead of the 1213 MiB FP8 head | 133.07 → **138.15** |
| [#3045](https://github.com/kekzl/imp/pull/3045) | the head pass replays as one CUDA graph | 140.40 → **143.47** |
| [#3047](https://github.com/kekzl/imp/pull/3047) | the draft LM head kernel reads 4 weight rows per block | 141.26 → **142.76** |
Each row is its own A/B, so a "before" does not equal the previous "after":
between runs on different hours the same build moves by a few per cent.
**Two drafts.** At 4 requests, k = 2 runs 12 rows in 13.87-13.93 ms plus a
3.79-3.80 ms head pass for 9.24-9.42 tokens. k = 1 ran 8 rows in 12.86-12.89 ms
plus 2.02-2.03 ms for 6.87-7.16 tokens. About a fifth more time for a third more tokens.
Each request now needs two spare slots, so the KV pool shrank from 12305 to 11380
blocks.
**A cheaper draft head.** One draft pass over the vocabulary, a
[248320 x 5120] matrix, costs 926.3 µs from the FP8 head and 425.3 µs from the
NVFP4 one at 1 row. Only the *guess* reads the 4-bit head. The check that decides
which tokens are emitted stays on FP8, so output quality does not depend on it.
Acceptance moved from 68.34 to 67.96 % at 2 chats.
**A CUDA graph.** The head pass was 25 + 31 separate launch calls. Replayed as one
graph, the GPU's idle time inside a pass dropped from 433.4 to 145.2 µs.
**Four rows per block.** The draft kernel gave each block one weight row and
re-read the activations for every one. At 4 rows it reached 64.82 % of DRAM
bandwidth. Four weight rows per block against activations held in registers:
86.47 %, and 659.8 → 456.6 µs per pass, bit-identical. 4 concurrent bursts went
from 534.17 to 564.33 tok/s.
4 overlapping chats, Qwen3.8-27B-NVFP4, tok/s (winning arm of each A/B)
[diagram omitted — see the page for the chart]
Each bar is the "after" arm of its own pull request (#3034, #3037, #3039, #3045), track = 150 tok/s. The grey bar is #3034's "before" arm, which already had steps 1 to 3.
## Step 9: the guard that judged half the evidence
imp unbinds a request whose drafts do not pay: if a verify emits fewer than 1.6
tokens on average, the head is not worth its price. That guard only counted lone
verifies. A request that drafted well inside batched steps all day got judged on
its last 8 lone verifies after its peer finished. Request 35 unbound at
`verifies=8 emitted=12`, while its response reported 659 of 1138 drafts accepted.
[#3048](https://github.com/kekzl/imp/pull/3048) adds the batched verifies to the
tally.
| 2 overlapping chats | before | after |
|---|---:|---:|
| unbinds per server | 1 / 2 / 2 | 0 / 0 / 0 |
| tok/s | 134.02 | **138.55** |
| request 35's last part, alone | 3.06-3.09 s | 2.23-2.24 s |
## Same text, faster
Greedy decoding gives one exact answer, so speed work must not change it. The
lone chat and the side-request chat produced byte-identical transcripts against
main in #3031, #3034 and #3047, 6 of 6 each time, and the plain chat did in #3030.
The degeneration suite passed 52 checks with 0 failures in #3029 and #3034.
## Honest limits
- **Greedy only, and small batches only.** A batch with a sampled row, or more than
4 decoding requests, takes the plain batched step. At 32 concurrent bursts
nothing binds: 2043.5 tok/s before, 2042.0 after.
- **GDN hybrids only.** Other models on a concurrent server keep n-gram drafting.
Nemotron's MTP head drafts at 39 % and loses decode rate, so it stays off there.
- **VRAM.** Every bound request pays for 2 spare state slots of 79.5 MiB, and the
head itself took 2773 MiB on a default server. That comes out of the KV pool.
- **Acceptance is the ceiling.** 67 to 70 % of drafts land on chat text. Everything
above only stopped the head from being switched off or made it cheaper; none of
it makes the guesses better.
Each fix was small. The hard part was seeing that the head was off: every one of
these failures still produced the right text, just without drafts.
Not every idea from those two days survived: [the kernel that ran zero
times](/blog/the-kernel-that-ran-zero-times/) collects the ones that didn't.