Decoding GGUF faster than llama.cpp on a 5090

GGUF is llama.cpp’s own format, its home turf. So it’s a slightly cheeky claim that imp decodes dense GGUF models at least 37% faster (and up to 72% on the best cases) than llama.cpp on a 5090. That figure is the decode-throughput delta measured in imp’s own single-stream, batch 1 runs at short context (tg128, 128 generated tokens). Here’s exactly why, the honest version, and the one model where the claim flips the other way.

Decode is a bandwidth game

The whole thing rests on one fact from the roofline post: single-stream, batch 1 decode is bandwidth-bound. The chip waits on weights more than it does maths. So two things decide who wins: how many bytes you move per weight, and how well your kernels use this particular chip. imp wins on both, and it’s worth being precise, because the easy version of this story is wrong.

What llama.cpp actually does

The easy version is “llama.cpp widens everything to FP16 and imp doesn’t.” That’s not true on the decode path. llama.cpp’s CUDA decode runs quantised integer kernels (its mul_mat_vec_q family): it quantises the activation to INT8 and uses dp4a dot products straight against the quantised weights, no detour through FP16. It is genuinely well-built. So both engines already skip FP16 for the weights, and the real difference is narrower and more specific than the marketing makes it sound.

Weight bits streamed per token, Q8_0 model (decode)

Both engines decode with quantised integer kernels, not FP16. imp's edge on an 8-bit source is its 4-bit NVFP4 decode cache, half the weight bytes on a bandwidth-bound path. On sources that are already 4-bit, the win comes from kernels tuned for this exact chip instead. The 8 and 4 are the nominal element widths; both formats carry block-scale overhead, so effective bits are roughly 8.5 (Q8_0, per-32 fp16 scale) versus about 4.5 (NVFP4, per-16 E4M3 scale), not exactly 8 versus 4.

Where imp’s edge actually comes from

Two places, and only the first is about bytes.

A 4-bit decode cache. At load time, imp converts the GGUF weights (Q8_0, Q6_K) once into 4-bit NVFP4 and stores them contiguously. This is a lossy requantisation of the decode-path weights, not a free repacking. For an 8-bit source, that halves the weight bytes the decode hot path streams, and on a bandwidth-bound path, half the bytes is most of a win. Build the small format once, spend it on every one of the thousands of decode steps. Prefill keeps the full-precision source, so the requant only ever touches the decode path; its output quality is not separately gated here.

Kernels built for one chip. The less glamorous half: imp’s matrix-vector kernels are written for sm_120 specifically, the NVFP4 path on the 5090’s FP4 tensor cores, a per-quant cascade that picks the right kernel for each format. On a source that’s already 4-bit, a Q4_K model (here an MoE, Gemma-4-26B-A4B), there’s no byte advantage left to take, and yet imp still wins. That points at the kernels carrying it there rather than the bandwidth, though with an MoE example it’s a softer signal than a clean dense-model proof would be.

Add the two together and you get the +37 to 72% across dense GGUF, in BENCHMARKS.md. The prefill comparisons there are commit-anchored; the decode-vs-llama.cpp figure is currently summary-only.

A note on the formats

Worth knowing why there are so many quant names. Q8_0 and Q4_0 use simple 32-element blocks with one scale each. The Q*_K formats use 256-element “superblocks” carrying eight finer sub-scales, which buys better quality at the same bit-width. imp’s integer path handles both; the superblocks just cost a more involved unpack.

Where it loses, honestly

One model breaks the pattern. Qwen3.6-35B decodes about 31% slower than llama.cpp, because it’s a Gated DeltaNet hybrid whose recurrent projections must stay FP16, the one path the 4-bit cache can’t touch. That’s a structural quality floor, not a tuning gap, and pretending otherwise would be the kind of benchmark dishonesty these posts try to avoid.

And credit where it’s due: GGUF, the quant formats, the whole practical playbook for running models locally came from llama.cpp. imp gets to specialise for one chip only because llama.cpp built the road everyone drives on. Being faster on a single GPU is a narrow win against a project doing something far harder, supporting everything, everywhere.

The broader lesson is the one that motivates the whole engine: on consumer Blackwell, “fast everywhere” loses to “built for exactly this chip.” Same format, same model, a sharper bet about which bytes to move and which kernels to run.