When the model isn't a transformer: GDN and Mamba2

Almost everything people call an LLM is a transformer: attention, end to end. But imp also runs models that are not. Qwen3.6-35B uses Gated DeltaNet for most of its layers; Nemotron-3-Nano uses Mamba2. These are linear-attention and state-space architectures, and they are different enough to be worth a proper look, not least because one of them is the model where imp’s GGUF decode lost to llama.cpp by the widest margin.

Attention’s bill, and the alternative

Softmax attention compares every token with every other one. That’s O(n squared) work, and it leaves behind a KV cache that grows with context. Linear-attention and state-space models make a different bargain: replace the all-pairs comparison with a recurrence, a fixed-size state that gets updated one token at a time.

Gated DeltaNet: the delta rule

GDN keeps a state matrix H and, for each token, applies the “delta rule”: it writes a gated correction into H (a learned decay scales the old state, a learned rate scales the new write), then reads the output as H times the query. No scores, no softmax, just a running state that’s continuously edited.

imp runs this as a register-resident scan: in the recurrent decode form the state lives in registers and never round-trips to global memory. The chunked “WY representation” tensor-core path for the parallel form does materialize chunk states to shared memory (following Yang et al. 2024). It is a real piece of engineering precisely because the recurrence is sequential and the difficulty is making it parallel without changing the answer.

Mamba2: a selective state space

Mamba2 has the same shape from a distance: a discrete-time state-space scan. A state h is updated as h = a*h + b*x per token and the output is read as y = C*h. imp parallelizes it across the state dimension with a transposed, coalesced layout so neighbouring threads touch neighbouring memory. Same idea as GDN: a fixed state, updated in a scan, instead of a matrix of comparisons.

The long-context superpower

Here is the payoff. Because the state is a fixed size, a 128-token context costs exactly what a 128,000-token context costs: nothing extra. There is no KV cache to grow and no limit to hit. For very long sequences, the per-step memory cost stays constant where a transformer’s keeps growing, which is why these architectures keep appearing in long-context models. The tradeoff is that a bounded recurrent state has weaker long-range exact recall than full attention, which is why production models tend to be hybrids.

The tax: where imp loses most

Now the catch, and the reason Qwen3.6-35B is the model where imp’s GGUF decode trailed llama.cpp furthest, by about 31%: on 2026-05-29, Q4_K_M, a 512-token prompt and 128 generated tokens, imp ran 157.9 tok/s against 228.6 for llama.cpp build 19e92c3 (-fa 1 -ngl 999). Batch size and sequence-length limit were not recorded. A re-run on 2026-06-07 read 165 against 192 (imp over 256 generated tokens, llama.cpp over 32).

On the NVFP4 path imp keeps the recurrent projections in FP16. Quantizing them to 4-bit looks free, but the error does not cancel; it accumulates in the running state H token after token, and imp’s code notes that 4-bit degrades quality on 9B-plus models. So the FP16 line items above add up to about 44% of decode running as FP16 matrix-vector work. Batch-1 decode is memory-bandwidth-bound, so keeping these projections in FP16 means moving 2 bytes per weight instead of 0.5; what you forgo is the 4-bit bandwidth saving, not tensor-core acceleration. On the GGUF file, though, the FP16 step was imp’s own choice, not something the architecture forces on every engine: imp dequantized the GDN in/out projections from the file’s quantized weights to FP16 and ran them through cuBLAS every token. An opt-in flag that routed them into imp’s 4-bit decode cache instead took the same Q4_K_M file from 161 to 248 tok/s with flat perplexity, past llama.cpp’s 228.6 (PR #486, 2026-05-30).

So a non-transformer is genuinely a different machine. Constant memory instead of a growing cache, a continuously-edited state instead of a matrix of scores, and a precision question on the one path that carries the state forward: imp keeps it FP16 on the NVFP4 path for quality, yet on the GGUF file a 4-bit route held perplexity flat. The honest-loss line in the benchmark tables had a real architecture behind it, and part of it was imp’s own choice.