The KV cache, and what really limits long context

The beginner version calls the KV cache the model’s “short-term memory” and leaves it there. The engineering reality is sharper: the KV cache is the thing that decides how long your context can get, and for long conversations it, not the model’s weights, is what runs you out of memory. Worth understanding properly.

What’s actually in it

Each time the model reads a token, attention computes a Key and a Value for it, in every attention head, in every layer. Rather than recompute all of that for the whole history on every new token, the model stores it. That store is the KV cache.

The size adds up fast. The total cache is:

2 (K and V)  x  layers  x  seq_len  x  kv_heads  x  head_dim  x  bytes_per_number  x  batch

Drop seq_len and batch and you get the bytes per token, which is the useful unit here:

bytes per token  =  2 (K and V)  x  layers  x  kv_heads  x  head_dim  x  bytes_per_number

Take a representative 30B GQA model: 64 layers, 8 KV heads, head_dim 128. In FP16 that is:

2  x  64 layers  x  8 kv_heads  x  128 head_dim  x  2 bytes  =  262,144 bytes  =  256 KB per token

So roughly 256 KB per token at FP16 (a 60-layer model lands near 240 KB). Not much for one token. A lot for a hundred thousand of them.

Why it’s the long-context bottleneck

Here’s the crucial asymmetry. The model’s weights are a fixed cost: they depend on the model, not on how long your conversation is. The KV cache is not. It grows straight, linearly, with the number of tokens in context.

The KV cache grows with every token (one 30B model)

At short context the weights dominate and the cache is a rounding error. But the cache keeps climbing, and on a 5090 with roughly 12 GB free after the weights are loaded, an FP16 cache at 256 KB per token runs you into the wall somewhere around 45-50k tokens. Past that, you’re not out of model, you’re out of cache.

Paging: don’t reserve the worst case

The naive approach reserves memory for the maximum possible context up front, which wastes most of it on short chats. imp uses a paged KV cache instead, the approach introduced by PagedAttention in vLLM (Kwon et al., SOSP 2023): memory is carved into small blocks (16 tokens each, the vLLM default) and handed out as the conversation actually grows. A short exchange uses a handful; a long one grows into thousands.

Because blocks are reference-counted, identical prefixes can be shared rather than copied. A system prompt that’s the same across a thousand requests can live in one set of blocks instead of a thousand, which is what makes prefix caching work.

Shrinking the cache

The same quantisation trick that shrinks the weights works on the cache itself. Store the Keys and Values in fewer bits and you fit far more of them.

KV precision	Size vs FP16	Roughly
FP16	1x	the baseline
FP8	1/2	2x the context, around 1% quality cost
NVFP4 (4-bit)	1/4	4x the context, modest degradation
INT4	1/4	4x, but loses decode speed and more quality

The quality numbers are not constants: they depend on the model and the task, and FP8’s “around 1%” or NVFP4’s “modest, benchmarked degradation” can be better or worse on yours. Treat them as a rough guide, not a guarantee.

NVFP4 is the sweet spot on Blackwell: a quarter of the size for roughly four times the context, and because the chip has hardware 4-bit conversion, reading it back stays cheap. That’s the difference between a model that forgets after a long document and one that holds the whole thing.

The hidden tax: bandwidth

One last thing the capacity story misses. During decode, every new token reads the entire KV cache back to do attention. So a long context isn’t only a memory-size problem, it’s a bandwidth one: the longer the context, the more cache bytes get streamed per token, and the slower each token gets. At short context the weight reads dominate the bandwidth budget and the slowdown is far below proportional; the slope approaches proportional only once KV reads dominate. imp even tags those cache reads as “evict first” in the L2 cache, so they don’t push the weights out of the way. Shrinking the cache, then, buys you two things at once: more context, and faster tokens within it.