Raphael Friedmann
← The log

The kernel that ran zero times

A rewrite made an imp kernel up to 33 % faster, and serving never launches that kernel. Four results from one day that changed no engine code, and the ledger that stops the next audit from paying for them twice.

On 4 November 2026 imp’s main branch took 21 commits. Twelve were performance work, and several of those are in the post on overlapping chats. Four of the others touched nothing under src/. Each tested an idea or a suspicion, found it wrong, and committed the evidence.

This post is about those four. Every number comes from the evidence directory or ledger row of the pull request named next to it. The model is Qwen3.8-27B-NVFP4 unless a section says otherwise, on one RTX 5090.

PR what looked true what the data said lines added
#3049 a kernel rewrite is worth shipping the kernel does not run 795
#3054 draft less after a rejection slower in almost every regime 666
#3053 v0.63.0 lost 1.01 % of prefill does not reproduce 16
#3038 serving leaks about 585 MiB no memory behind it 415

1. A faster kernel nobody calls#

#3047 sped up the draft LM head by reading four weight rows per block. Its FP16 sibling, gemv_nvfp4_kpar_mb_fp16_kernel, was the obvious next target: imp’s own ledger said it served “every speculative verify chunk”. The same recipe went in, and the microbench agreed. Output stayed bit-identical at 24 of 24 points.

Shape N x K rows main, µs (2 runs) rewrite, µs (2 runs)
17408 x 5120 1 33.5 / 33.2 43.7 / 42.0
17408 x 5120 3 48.5 / 42.7 35.4 / 35.5
17408 x 5120 4 46.5 / 47.0 37.3 / 38.1
5120 x 17408, accumulating 4 54.9 / 50.0 37.5 / 36.7

Three and four rows came out 17 to 33 % faster, one row slower. The other half of the evidence is a profile of a real server, and it answers the question that comes first: how often does this kernel run?

Server load launches share of kernel time tok/s
default 1 stream 0 0.00 % 123.4
default 4 streams 0 0.00 % 504.5
default 32 streams 0 0.00 % 1943.2
33612-token prompt 1 stream 0 0.00 % 44.7
control: speculative.capture=false 1 stream 120400 67.25 % 89.5
control: speculative.capture=false 4 streams 10400 3.96 % 262.1

The control rows carry the argument. A zero only counts if the same instrument can read something else, and CUDA graphs are a good place to hide launches from a profiler. With graph nodes traced and capture switched off, the kernel takes 67.25 % of kernel time. So the tool sees it, and the default server does not run it.

The route explains why. #2802 sends every captured verify chunk to the small-M kernel. The batched verify of two or more requests has at least 6 rows, and this kernel stops at 4. Past 32768 tokens a chunk would run uncaptured, but MTP stopped drafting at 16384.

The rewrite did not ship. Its diff sits next to the microbench, with a note saying what would make it worth reviving: a path that runs uncaptured verify at 4 rows or fewer.

2. Drafting less after a miss#

The MTP head drafts two tokens per step. After a rejection, drafting one looks like the sensible reaction: the head is having a bad moment, so save the work. imp’s single-chunk verify already does this. The candidate brought the rule to the batched verify.

Traces priced it before any A/B.

Context second draft accepted after a miss depth 1 saves step estimate, fixed → adaptive
7696 45-52 % 1.60 ms 15.76 ms 159.63 → 156.84 tok/s
19773 55-57 % 2.10 ms 16.89 ms 150.05 → 146.17 tok/s
80043 45-73 % 2.31 ms 17.60 ms 146.15 → 142.55 tok/s

A miss does lower the odds, from 75 % after a full accept to 45-73 %, but not by enough. One more accepted token is worth 6.2 to 6.8 ms of step, and depth 1 saves at most 2.31 ms. The estimate said no, but an estimate is not a measurement, so the change was built and run anyway.

An A/B on one prompt then said yes, at least at 8k: three pairs went from 159.78 to 167.76 tok/s, 5.0 % faster. The same prompt at 80k went from 145.75 to 130.99, 10.1 % slower.

The reason is a trap. Padding a row in place of a draft changes the greedy text, by a mechanism nobody bisected. So the two arms no longer generate the same tokens, and an A/B on one prompt compares two different answers. Four prompts per context settled it.

Regime main adaptive
1 chat, 8k / 20k / 80k, 4 prompts x 2 pairs 166.68 / 147.16 / 145.60 165.04 / 143.48 / 139.50
2 overlapping chats, 20k / 40k / 80k, per stream 138.80 / 130.17 / 124.95 131.45 / 125.99 / 121.54
6 short chats, concurrency 1 154.61-155.07 148.07-148.38

The candidate won 3 of 12 prompt and context cells. Fixed depth 2 stays.

3. The regression with clean error bars#

The v0.63.0 release sweep flagged prefill on Nemotron-3-Nano-30B-A3B-NVFP4: pp512 down 1.01 %. Three pairs, the new build slower in all three, and the ranges did not even overlap: 34167 to 34477 tok/s old, 33965 to 34077 new. By most standards that is a regression, and the next step is a bisect.

The next step was more pairs, on the images actually published.

Run pairs old, median tok/s new, median tok/s new faster
release sweep, candidate image 3 34406.70 34058.33 0/3
published images, pp512 only 8 34277.40 34313.60 6/8
published images, release_ab.sh 4 34268.26 34411.18 3/4

No regression in v0.63.0. The delta belongs to the candidate image or to the host state during that sweep, and the log leaves it there. A coin flip loses three pairs in a row 12.5 % of the time, and a release sweep compares more than one number.

4. The memory that was not there#

Own VRAM on a default server grew 4098 MiB from the end of init to the end of a 32-stream wave. Most of it had names. About 585 MiB did not, and it only showed with the MTP head on, which made the head the suspect.

Own VRAM growth during serving, MiB (total 4098)
31 recurrent slots2480Batched-verify spares320KV grow-ahead312MTP workspace360Reading step585Residual41
20634 MiB at init, 24732 MiB after the fourth 32-stream wave, track = 2500 MiB. Grey: the reading step has no allocation behind it, the residual is unattributed. From breakdown.txt in #3038.

The hunt refuted eight candidates.

Candidate why not
captured verify graphs per recurrent slot speculative.capture=false grows the same
FFN prefill graphs per row bucket runtime.prefill_ffn_graph=false grows the same
cuBLASLt lazy kernel loads 4 MiB in total in a standalone probe
async pool high-water 11424 MiB at boot and after the wave
graph memory pool 0.0 MiB high-water since serving began
driver local memory at most 207.2 MiB, flat during the wave
MTP history slots, carry, batched verify all off: own VRAM still +2824 MiB
a cudaMalloc during serving interposer counted 0

Then the same window through a second instrument: own VRAM +2824 MiB, nvidia-smi +2240. A 50-line probe with no imp process on the card settled it. It only calls cudaMalloc in 256 MiB steps and compares the two readings. For 89 steps they move together. At step 90 nvidia-smi rises by the 256 MiB that were allocated, and cudaMemGetInfo rises by 841.

So on this WSL2 host cudaMemGetInfo, which imp’s own figure carries, jumps 585 MiB once the device passes about 24.5 GB in use. With the MTP head off, nvidia-smi peaks at 23850 MiB and the server never gets there. With it on, it reaches 25566 MiB. The head was only guilty of crossing a line.

The search still paid. The 360 MiB workspace was being taken after the capacity plan ran, and #3040 made it a planned row. And an A/B straddling that line now has a known 585 MiB of nothing in it.

Where the verdicts live#

The Nemotron run went into PERF_LOG.md, the reading step into MEMORY.md. The kernel and the draft depth went into docs/audit/SETTLED.md, 821 lines by now, which an audit reads before it generates hypotheses. It exists because the audit of 29 July spent most of its budget refuting itself: 8 of its 13 hypotheses described duplication that earlier work had already removed. The ledger’s stated reason:

a suspected problem that turns out to be clean is worth exactly as much as one that is real, and stops the next pass re-chasing it.

Each row has an anchor, the file that makes it settled, and a release check fails if an anchor stops resolving. On main today, 53 of 2928 commit subjects say “refuted” or “does not reproduce”.

The ledger is not an oracle, and the kernel from section 1 shows it. Its row first called the kernel a test oracle with no caller outside tests/. A correction on 17 October said it served every verify chunk. #2802 merged a day later and made that false for captured ones. That sentence stayed in the ledger until #3049 corrected the row again. An anchor proves that a file exists, never that a claim about it is still true.

What a refutation needs#

  • Proof the path ran. A launch count, a log line, a counter, plus a control where it reads non-zero. The ledger says it about a log line that never printed: “A gate that cannot be shown to fire has not been validated.”
  • More than one input, whenever a change can alter the output. One prompt gave +5.0 % for a change that loses.
  • More pairs before a bisect. Non-overlapping ranges from 3 pairs were not enough.
  • A second instrument. nvidia-smi next to cudaMemGetInfo ended a hunt that eight theories had not.
  • The diff, and a revival condition. A verdict without them gets paid for again.

Honest limits#

  • A verdict has a date. #3049 merged at 14:22. #3050 merged at 16:34 the same day and changed two of its premises: MTP now drafts past 16384 tokens, and a lone request verifies in the batched step with 3 rows. Reading the code, the kernel stays off, because those steps are captured too and take the small-M route. That is a reading. Nobody has re-run the launch count.
  • Each verdict covers one model on one card. Adaptive depth lost on Qwen3.8-27B-NVFP4 and says nothing about other acceptance rates.
  • Three causes stay open. Why a pad row changes the greedy text, what moved the v0.63.0 sweep, and why the driver’s reading jumps.
  • It costs. 1892 added lines of evidence and tools across the four, and none of them made a token faster.

The next session that looks at that kernel reads one table row first. That row has already been wrong twice, so it also gets a launch count.