Raphael Friedmann
← The log

Speculative decoding when the chats overlap

Qwen3.8-27B ships its own draft head. On imp it sped up one chat by 35 %, and any second chat or UI side request switched it off. Nine changes later, four overlapping chats decode at about 140 tok/s instead of 84.

Qwen3.8-27B comes with a small extra block, the MTP head, that guesses the token after the next one. On a single stream imp has used it since August: the head drafts, the main model checks the draft in the same step, and about 70 % of the drafts are accepted. That is speculative decoding with the draft model built in.

Real chat traffic is not a single stream. A chat UI fires a side request between two turns to name the conversation. A second person opens a second chat. Every one of those cases switched the head off, for a different reason each time. This post walks through the nine changes, all merged on 3 and 4 November 2026, that keep it drafting when chats overlap. Every number is from the A/B table of the pull request named next to it: Qwen3.8-27B-NVFP4 on one RTX 5090, a fresh server per arm, three alternating pairs.

Step PR what was wrong kind
1 #3029 head not loaded on a concurrent server off
2 #3030 one head slot, the UI’s side request took it off
3 #3031 a batched step did not feed the head off
4 #3034 a batched step could not check a draft value
5 #3037 one draft per request value
6 #3039 the guess read the FP8 LM head price
7 #3045 the head pass was 56 launches price
8 #3047 the draft kernel read one weight row per block price
9 #3048 the pay-off guard ignored batched verifies off

“off” means the head was switched off without anyone noticing, “value” and “price” mean it was on but paid less than it could.

Why this model makes it hard#

On a pure transformer, a rejected draft costs nothing to undo: the KV cache just drops its last row. Qwen3.8-27B is a hybrid: 16 attention layers and 48 Gated DeltaNet layers, which carry a recurrent state instead of a cache. That state is overwritten with every token, so there is nothing to truncate.

imp’s answer is a spare state slot. A verify step writes the state after “last token plus draft” into a spare and the state after “last token only” in place. If the draft is accepted, the request’s live slot becomes the spare. If not, it stays where it was. No copies, but every request that verifies needs its own spare, and on this model one slot is 79.5 MiB.

Draft accepted

  • 2 tokens emitted
  • spare slot becomes the live slot
  • old live slot becomes the spare

Draft missed

  • 1 token emitted
  • spare slot is discarded
  • live slot already holds the right state

The head has state of its own too: a KV cache of the hidden rows it has seen. A request can only draft while that cache is in step with its context. Most of the failures below were exactly that cache falling out of step.

Step 1: the head was not even loaded#

speculative.mtp_k=auto loaded the head only when the server ran with a maximum batch of 1. A default imp-server allows 32 concurrent requests, so it served this model without drafts. #3029 loads the head on a concurrent server when the model is a GDN hybrid. A prompt that prefills alone binds the head and drafts while it decodes alone; a batch of several decodes plain.

Default server before after
1 stream, tok/s 96.87 131.28
4 concurrent bursts, tok/s 341.3 339.7
32 concurrent bursts, tok/s 2043.5 2042.0
own peak VRAM, MiB 22034 24807

One stream got 35.5 % faster with 2552 of 3644 drafts accepted. The price is 2773 MiB and a smaller KV pool: 15651 blocks down to 13230.

Step 2: one slot, and the UI took it#

The head had one KV slot. The chat UI’s side request between two turns overwrote it. The next turn resumed from the prefix cache with 464 to 2272 cached tokens, but the head needed hidden rows for those tokens that the cached prefix never produced. So it unbound, and turns 3 to 6 decoded without drafts. #3030 gives the head 4 slots and picks the one whose history covers the resume.

6-turn conversation before after follow-up turns drafting
greedy chat + UI side request 101.88 tok/s 122.28 3/15 → 15/15
two chats alternating 101.52 tok/s 117.43 6/30 → 30/30
plain greedy chat (control) 119.93 tok/s 119.60 15/15 → 15/15

The head’s K+V grew from 64 to 256 MiB.

Step 3: a batched step left the head behind#

When two chats decode at the same moment, they share one batched step, and that step did not feed the head. The bound request’s head cache fell behind its context, so the next lone step unbound it. #3031 copies each bound request’s hidden row out of every batched step into a carry buffer (256 rows of 10 KiB per slot) and feeds the head from it later, in order.

2 overlapping chats, 6 turns each before after
turns drafting 6/36 35/36
mean tok/s 93.78 98.38

The head now drafts on 35 of 36 turns. The gain is still small, because a batched step can carry the hidden rows but cannot use a draft yet.

Step 4: verify the drafts together#

Decode on this card is bound by memory bandwidth. At 32 streams, 12.06 ms of a 14.75 to 14.98 ms step is the time to stream the weights, the LM head and the recurrent state from DRAM once. Adding a few rows to a step that small barely changes its price. #3034 makes the step that runs 2 to 4 decoding requests forward each bound request’s last token plus its draft.

Step price rows time tokens out
plain decode, 4 requests 4 11.72 ms 4
bound verify, 4 requests 8 13.12-13.65 ms forward + 2.02-2.65 ms head 7.15-7.20
  1. One forward for everyone

    Each bound request contributes its last token and its draft. Attention gets one block table per row, the GDN layers write the draft state into the request's spare slot.

  2. One argmax over all rows

    LM head and argmax run over every row, one copy to the host.

  3. Accept per request

    A request whose draft matched takes both tokens and swaps its live state slot to the spare. One that missed keeps one token and its old slot.

  4. One head pass for everyone

    The head is fed every new (token, hidden row) pair in one ragged pass and drafts the next token for each request.

before after
2 overlapping chats, tok/s per turn 94.21 117.06
4 overlapping chats 83.72 117.45
turns drafting, 4 chats 13/72 72/72
4 concurrent bursts 337.0-338.1 457.2-461.8
32 concurrent bursts 2020.2-2030.6 2029.8-2034.7

The same PR fixed two defects in the older opt-in path: its row cap read the unresolved max_batch_size of 0, so every step declined, and its batched draft needed an LM head this model’s export does not build.

Steps 5 to 8: make each verify worth more and cost less#

With the batched verify in place, the rest is price per step.

PR change 4 overlapping chats, tok/s
#3037 2 drafts per request, 3 rows each 121.59 → 134.17
#3039 drafts read the 682 MiB NVFP4 head instead of the 1213 MiB FP8 head 133.07 → 138.15
#3045 the head pass replays as one CUDA graph 140.40 → 143.47
#3047 the draft LM head kernel reads 4 weight rows per block 141.26 → 142.76

Each row is its own A/B, so a “before” does not equal the previous “after”: between runs on different hours the same build moves by a few per cent.

Two drafts. At 4 requests, k = 2 runs 12 rows in 13.87-13.93 ms plus a 3.79-3.80 ms head pass for 9.24-9.42 tokens. k = 1 ran 8 rows in 12.86-12.89 ms plus 2.02-2.03 ms for 6.87-7.16 tokens. About a fifth more time for a third more tokens. Each request now needs two spare slots, so the KV pool shrank from 12305 to 11380 blocks.

A cheaper draft head. One draft pass over the vocabulary, a [248320 x 5120] matrix, costs 926.3 µs from the FP8 head and 425.3 µs from the NVFP4 one at 1 row. Only the guess reads the 4-bit head. The check that decides which tokens are emitted stays on FP8, so output quality does not depend on it. Acceptance moved from 68.34 to 67.96 % at 2 chats.

A CUDA graph. The head pass was 25 + 31 separate launch calls. Replayed as one graph, the GPU’s idle time inside a pass dropped from 433.4 to 145.2 µs.

Four rows per block. The draft kernel gave each block one weight row and re-read the activations for every one. At 4 rows it reached 64.82 % of DRAM bandwidth. Four weight rows per block against activations held in registers: 86.47 %, and 659.8 → 456.6 µs per pass, bit-identical. 4 concurrent bursts went from 534.17 to 564.33 tok/s.

4 overlapping chats, Qwen3.8-27B-NVFP4, tok/s (winning arm of each A/B)
Before batched verify83.72Batched verify117.45Two drafts per request134.17NVFP4 draft head138.15Head pass as CUDA graph143.47
Each bar is the "after" arm of its own pull request (#3034, #3037, #3039, #3045), track = 150 tok/s. The grey bar is #3034's "before" arm, which already had steps 1 to 3.

Step 9: the guard that judged half the evidence#

imp unbinds a request whose drafts do not pay: if a verify emits fewer than 1.6 tokens on average, the head is not worth its price. That guard only counted lone verifies. A request that drafted well inside batched steps all day got judged on its last 8 lone verifies after its peer finished. Request 35 unbound at verifies=8 emitted=12, while its response reported 659 of 1138 drafts accepted. #3048 adds the batched verifies to the tally.

2 overlapping chats before after
unbinds per server 1 / 2 / 2 0 / 0 / 0
tok/s 134.02 138.55
request 35’s last part, alone 3.06-3.09 s 2.23-2.24 s

Same text, faster#

Greedy decoding gives one exact answer, so speed work must not change it. The lone chat and the side-request chat produced byte-identical transcripts against main in #3031, #3034 and #3047, 6 of 6 each time, and the plain chat did in #3030. The degeneration suite passed 52 checks with 0 failures in #3029 and #3034.

Honest limits#

  • Greedy only, and small batches only. A batch with a sampled row, or more than 4 decoding requests, takes the plain batched step. At 32 concurrent bursts nothing binds: 2043.5 tok/s before, 2042.0 after.
  • GDN hybrids only. Other models on a concurrent server keep n-gram drafting. Nemotron’s MTP head drafts at 39 % and loses decode rate, so it stays off there.
  • VRAM. Every bound request pays for 2 spare state slots of 79.5 MiB, and the head itself took 2773 MiB on a default server. That comes out of the KV pool.
  • Acceptance is the ceiling. 67 to 70 % of drafts land on chat text. Everything above only stopped the head from being switched off or made it cheaper; none of it makes the guesses better.

Each fix was small. The hard part was seeing that the head was off: every one of these failures still produced the right text, just without drafts.

Not every idea from those two days survived: the kernel that ran zero times collects the ones that didn’t.