# When the model isn't a transformer: GDN and Mamba2 > Not every LLM is built on attention. Gated DeltaNet and Mamba2 replace the score matrix with a recurrence, which gives constant memory, and one stubborn precision floor. Source: https://rfriedmann.de/blog/when-the-model-is-not-a-transformer/ Published: 2026-06-13 · Track: log · Level: Expert Almost everything people call an LLM is a transformer: attention, end to end. But [imp](https://github.com/kekzl/imp) also runs models that are not. Qwen3.6-35B uses **Gated DeltaNet** for most of its layers; Nemotron-3-Nano uses **Mamba2**. These are linear-attention and state-space architectures, and they are different enough to be worth a proper look, not least because one of them is the only model where imp's GGUF decode actually loses to llama.cpp. ## Attention's bill, and the alternative Softmax attention compares every token with every other one. That's O(n squared) work, and it leaves behind a [KV cache that grows with context](/blog/kv-cache-and-long-context/). Linear-attention and state-space models make a different bargain: replace the all-pairs comparison with a **recurrence**, a fixed-size state that gets updated one token at a time.

Transformer (attention)

GDN / Mamba2 (recurrence)

## Gated DeltaNet: the delta rule GDN keeps a state matrix `H` and, for each token, applies the "delta rule": it writes a gated correction into `H` (a learned decay scales the old state, a learned rate scales the new write), then reads the output as `H` times the query. No scores, no softmax, just a running state that's continuously edited. imp runs this as a register-resident scan: in the recurrent decode form the state lives in registers and never round-trips to global memory. The chunked "WY representation" tensor-core path for the parallel form does materialize chunk states to shared memory (following Yang et al. 2024). It is a real piece of engineering precisely because the recurrence is sequential and the difficulty is making it parallel without changing the answer. ## Mamba2: a selective state space Mamba2 has the same shape from a distance: a discrete-time state-space scan. A state `h` is updated as `h = a*h + b*x` per token and the output is read as `y = C*h`. imp parallelizes it across the state dimension with a transposed, coalesced layout so neighbouring threads touch neighbouring memory. Same idea as GDN: a fixed state, updated in a scan, instead of a matrix of comparisons. ## The long-context superpower Here is the payoff. Because the state is a fixed size, a 128-token context costs exactly what a 128,000-token context costs: nothing extra. There is no [KV cache](/blog/kv-cache-and-long-context/) to grow and no limit to hit. For very long sequences, the per-step memory cost stays constant where a transformer's keeps growing, which is why these architectures keep appearing in long-context models. The tradeoff is that a bounded recurrent state has weaker long-range exact recall than full attention, which is why production models tend to be hybrids. ## The tax: the one place imp loses Now the catch, and the reason Qwen3.6-35B is the single model where imp's GGUF decode trails llama.cpp by about 31%.
Where Qwen3.6-35B decode time goes
[diagram omitted — see the page for the chart]
Roughly 46% of decode is locked in FP16. Quantizing the recurrent projections to 4-bit makes error pile up in the running state and wrecks quality, so they stay full-precision. That tax accounts for much of the ~31% gap to llama.cpp, though some of it is likely kernel and tuning on our side.
The recurrent projections **have to stay FP16**. Quantising them to 4-bit looks free, but the error does not cancel; it accumulates in the running state `H` token after token, and on a 9B-plus model it silently corrupts the output. So the FP16 line items above add up to roughly 46% of decode running as FP16 matrix-vector work. Batch-1 decode is memory-bandwidth-bound, so keeping these projections in FP16 means moving 2 bytes per weight instead of 0.5; what you forgo is the [4-bit](/blog/nvfp4-at-the-bit-level/) bandwidth saving, not tensor-core acceleration. Much of that is a quality decision rather than missing tuning: closing it needs a quality-preserving low-precision recurrence, which is open research, and I am not aware of one that holds up. But it does not fully account for the gap to llama.cpp specifically, which runs the same architecture under the same FP16-recurrence necessity, so part of the ~31% is likely kernel and tuning on our side. So a non-transformer is genuinely a different machine. Constant memory instead of a growing cache, a continuously-edited state instead of a matrix of scores, and a hard precision floor on the one path that carries the state forward. The [honest-loss line](/blog/gguf-decode-beats-llama-cpp/) in the benchmark tables has a real architecture behind it.