Decoding GGUF faster than llama.cpp on a 5090

GGUF is llama.cpp’s own format, its home turf. So it’s a slightly cheeky claim that imp decodes dense GGUF models at least 37% faster (and up to 72% on the best cases) than llama.cpp on a 5090. That figure is the decode-throughput delta measured in imp’s own single-stream, batch 1 runs at short context (tg128, 128 generated tokens). The two ends are Qwen3-14B Q6_K (155.4 vs 113.7 tok/s, +37%) and Qwen3-8B Q8_0 (274.4 vs 159.7 tok/s, +72%), measured 2026-05-29 on the same GGUF files against llama.cpp build 19e92c3 (CUDA 12.8, -fa 1 -ngl 999 -r 5), imp at 8 reps. The driver version was not logged for these runs. Here’s exactly why, the honest version, and the models where the claim flips the other way.

Decode is a bandwidth game

The whole thing rests on one fact from the roofline post: single-stream, batch 1 decode is bandwidth-bound. The chip waits on weights more than it does maths. So two things decide who wins: how many bytes you move per weight, and how well your kernels use this particular chip. imp wins on both, and it’s worth being precise, because the easy version of this story is wrong.

What llama.cpp actually does

The easy version is “llama.cpp widens everything to FP16 and imp doesn’t.” That’s not true on the decode path. llama.cpp’s CUDA decode runs quantised integer kernels (its mul_mat_vec_q family): it quantises the activation to INT8 and uses dp4a dot products straight against the quantised weights, no detour through FP16. It is genuinely well-built. So both engines already skip FP16 for the weights, and the real difference is narrower and more specific than the marketing makes it sound.

Weight bits streamed per token, Q8_0 model (decode)

Both engines decode with quantised integer kernels, not FP16. imp's edge on an 8-bit source is its 4-bit NVFP4 decode cache, half the weight bytes on a bandwidth-bound path. On sources that are already 4-bit, the win comes from kernels tuned for this exact chip instead. The 8 and 4 are the nominal element widths; both formats carry block-scale overhead, so effective bits are roughly 8.5 (Q8_0, per-32 fp16 scale) versus about 4.5 (NVFP4, per-16 E4M3 scale), not exactly 8 versus 4.

Where imp’s edge actually comes from

Two places, and only the first is about bytes.

A 4-bit decode cache. At load time, imp converts the GGUF weights (Q8_0, Q6_K) once into 4-bit NVFP4 and stores them contiguously. This is a lossy requantisation of the decode-path weights, not a free repacking. For an 8-bit source, that halves the weight bytes the decode hot path streams, and on a bandwidth-bound path, half the bytes is most of a win. Build the small format once, spend it on every one of the thousands of decode steps. Prefill keeps the full-precision source, so the requant only ever touches the decode path; its output quality is not separately gated here.

Kernels built for one chip. The less glamorous half: imp’s matrix-vector kernels are written for sm_120 specifically, the NVFP4 path on the 5090’s FP4 tensor cores, a per-quant cascade that picks the right kernel for each format. On a source that’s already 4-bit, a Q4_K model (here an MoE, Gemma-4-26B-A4B), there’s no byte advantage left to take, and yet imp still wins. That points at the kernels carrying it there rather than the bandwidth, though with an MoE example it’s a softer signal than a clean dense-model proof would be.

Add the two together and you get the +37 to 72% across dense GGUF, in BENCHMARKS.md. The prefill comparisons there are commit-anchored; the decode-vs-llama.cpp figure is currently summary-only.

A note on the formats

Worth knowing why there are so many quant names. Q8_0 and Q4_0 use simple 32-element blocks with one scale each. The Q*_K formats use 256-element “superblocks” carrying eight finer sub-scales, which buys better quality at the same bit-width. imp’s integer path handles both; the superblocks just cost a more involved unpack.

Where it loses, honestly

The pattern holds for the dense models, not for every MoE. In the same 2026-05-29 run Gemma-4-26B won, but Qwen3-30B-A3B Q4_K_M decoded 15% slower than llama.cpp (276.5 vs 316.7 tok/s), and Qwen3.6-35B, a Gated DeltaNet hybrid, about 31% slower (157.9 vs 228.6). On Qwen3.6 imp ran the recurrent projections as FP16 through cuBLAS every token. That was imp’s own default, not a floor: an opt-in flag that moves them into the 4-bit decode cache took the same file to 248 tok/s with flat perplexity (PR #486). Leaving the default numbers in the table is the honest version.

And credit where it’s due: GGUF, the quant formats, the whole practical playbook for running models locally came from llama.cpp. imp gets to specialise for one chip only because llama.cpp built the road everyone drives on. Being faster on a single GPU is a narrow win against a project doing something far harder, supporting everything, everywhere.

The broader lesson is the one that motivates the whole engine: on consumer Blackwell, “fast everywhere” loses to “built for exactly this chip.” Same format, same model, a sharper bet about which bytes to move and which kernels to run.