When the model isn't a transformer: GDN and Mamba2

Almost everything people call an LLM is a transformer: attention, end to end. But imp also runs models that are not. Qwen3.6-35B uses Gated DeltaNet for most of its layers; Nemotron-3-Nano uses Mamba2. These are linear-attention and state-space architectures, and they are different enough to be worth a proper look, not least because one of them is the only model where imp’s GGUF decode actually loses to llama.cpp.

Attention’s bill, and the alternative

Softmax attention compares every token with every other one. That’s O(n squared) work, and it leaves behind a KV cache that grows with context. Linear-attention and state-space models make a different bargain: replace the all-pairs comparison with a recurrence, a fixed-size state that gets updated one token at a time.

Gated DeltaNet: the delta rule

GDN keeps a state matrix H and, for each token, applies the “delta rule”: it writes a gated correction into H (a learned decay scales the old state, a learned rate scales the new write), then reads the output as H times the query. No scores, no softmax, just a running state that’s continuously edited.

imp runs this as a register-resident scan: in the recurrent decode form the state lives in registers and never round-trips to global memory. The chunked “WY representation” tensor-core path for the parallel form does materialize chunk states to shared memory (following Yang et al. 2024). It is a real piece of engineering precisely because the recurrence is sequential and the difficulty is making it parallel without changing the answer.

Mamba2: a selective state space

Mamba2 has the same shape from a distance: a discrete-time state-space scan. A state h is updated as h = a*h + b*x per token and the output is read as y = C*h. imp parallelizes it across the state dimension with a transposed, coalesced layout so neighbouring threads touch neighbouring memory. Same idea as GDN: a fixed state, updated in a scan, instead of a matrix of comparisons.

The long-context superpower

Here is the payoff. Because the state is a fixed size, a 128-token context costs exactly what a 128,000-token context costs: nothing extra. There is no KV cache to grow and no limit to hit. For very long sequences, the per-step memory cost stays constant where a transformer’s keeps growing, which is why these architectures keep appearing in long-context models. The tradeoff is that a bounded recurrent state has weaker long-range exact recall than full attention, which is why production models tend to be hybrids.

The tax: the one place imp loses

Now the catch, and the reason Qwen3.6-35B is the single model where imp’s GGUF decode trails llama.cpp by about 31%.

The recurrent projections have to stay FP16. Quantising them to 4-bit looks free, but the error does not cancel; it accumulates in the running state H token after token, and on a 9B-plus model it silently corrupts the output. So the FP16 line items above add up to roughly 46% of decode running as FP16 matrix-vector work. Batch-1 decode is memory-bandwidth-bound, so keeping these projections in FP16 means moving 2 bytes per weight instead of 0.5; what you forgo is the 4-bit bandwidth saving, not tensor-core acceleration. Much of that is a quality decision rather than missing tuning: closing it needs a quality-preserving low-precision recurrence, which is open research, and I am not aware of one that holds up. But it does not fully account for the gap to llama.cpp specifically, which runs the same architecture under the same FP16-recurrence necessity, so part of the ~31% is likely kernel and tuning on our side.

So a non-transformer is genuinely a different machine. Constant memory instead of a growing cache, a continuously-edited state instead of a matrix of scores, and a hard precision floor on the one path that carries the state forward. The honest-loss line in the benchmark tables has a real architecture behind it.