# The KV cache, and what really limits long context > The beginner version calls it short-term memory. The engineering reality: the KV cache, not the weights, is what decides how long your context can get, and what runs you out of memory. Source: https://rfriedmann.de/blog/kv-cache-and-long-context/ Published: 2026-05-19 · Track: log · Level: Advanced The [beginner version](/blog/how-imp-works-model-to-token/) calls the KV cache the model's "short-term memory" and leaves it there. The engineering reality is sharper: the KV cache is the thing that decides how long your context can get, and for long conversations it, not the model's weights, is what runs you out of memory. Worth understanding properly. ## What's actually in it Each time the model reads a token, attention computes a **Key** and a **Value** for it, in every attention head, in every layer. Rather than recompute all of that for the whole history on every new token, the model stores it. That store is the KV cache. The size adds up fast. The total cache is: ``` 2 (K and V) x layers x seq_len x kv_heads x head_dim x bytes_per_number x batch ``` Drop seq_len and batch and you get the bytes per token, which is the useful unit here: ``` bytes per token = 2 (K and V) x layers x kv_heads x head_dim x bytes_per_number ``` Take a representative 30B GQA model: 64 layers, 8 KV heads, head_dim 128. In FP16 that is: ``` 2 x 64 layers x 8 kv_heads x 128 head_dim x 2 bytes = 262,144 bytes = 256 KB per token ``` So roughly **256 KB per token** at FP16 (a 60-layer model lands near 240 KB). Not much for one token. A lot for a hundred thousand of them. ## Why it's the long-context bottleneck Here's the crucial asymmetry. The model's weights are a *fixed* cost: they depend on the model, not on how long your conversation is. The KV cache is not. It grows straight, linearly, with the number of tokens in context.
The KV cache grows with every token (one 30B model)
[diagram omitted — see the page for the chart]
Model weights are a fixed cost; the KV cache is not. It grows straight with context length, so long chats are limited by it, not by the weights. Storing it in 4-bit cuts the slope to a quarter, which buys roughly 4x the context before you hit the wall.
At short context the weights dominate and the cache is a rounding error. But the cache keeps climbing, and on a 5090 with roughly 12 GB free after the weights are loaded, an FP16 cache at 256 KB per token runs you into the wall somewhere around 45-50k tokens. Past that, you're not out of model, you're out of cache. ## Paging: don't reserve the worst case The naive approach reserves memory for the maximum possible context up front, which wastes most of it on short chats. imp uses a **paged** KV cache instead, the approach introduced by PagedAttention in vLLM (Kwon et al., SOSP 2023): memory is carved into small blocks (16 tokens each, the vLLM default) and handed out as the conversation actually grows. A short exchange uses a handful; a long one grows into thousands. Because blocks are reference-counted, identical prefixes can be *shared* rather than copied. A system prompt that's the same across a thousand requests can live in one set of blocks instead of a thousand, which is what makes prefix caching work. ## Shrinking the cache The same [quantisation trick that shrinks the weights](/blog/how-a-big-model-fits-on-one-card/) works on the cache itself. Store the Keys and Values in fewer bits and you fit far more of them. | KV precision | Size vs FP16 | Roughly | |---|---|---| | FP16 | 1x | the baseline | | FP8 | 1/2 | 2x the context, around 1% quality cost | | NVFP4 (4-bit) | 1/4 | 4x the context, modest degradation | | INT4 | 1/4 | 4x, but loses decode speed and more quality | The quality numbers are not constants: they depend on the model and the task, and FP8's "around 1%" or NVFP4's "modest, benchmarked degradation" can be better or worse on yours. Treat them as a rough guide, not a guarantee. NVFP4 is the sweet spot on Blackwell: a quarter of the size for roughly four times the context, and because the chip has hardware 4-bit conversion, reading it back stays cheap. That's the difference between a model that forgets after a long document and one that holds the whole thing. ## The hidden tax: bandwidth One last thing the capacity story misses. During decode, every new token reads the *entire* KV cache back to do attention. So a long context isn't only a memory-size problem, it's a [bandwidth](/blog/prefill-vs-decode-roofline/) one: the longer the context, the more cache bytes get streamed per token, and the slower each token gets. At short context the weight reads dominate the bandwidth budget and the slowdown is far below proportional; the slope approaches proportional only once KV reads dominate. imp even tags those cache reads as "evict first" in the L2 cache, so they don't push the weights out of the way. Shrinking the cache, then, buys you two things at once: more context, and faster tokens within it.