How a 30-billion-parameter model fits on one card
Quantisation, explained for normal people: how shrinking each number in a model lets a giant fit on a desktop graphics card, and what it costs.
A model like Qwen3-30B has 30 billion numbers inside it (its weights). Stored the normal way, that is about 60 gigabytes. An RTX 5090 graphics card holds 32. The numbers don’t add up, and yet the model runs on that card just fine. The trick that makes it possible has an ugly name and a simple idea.
A number doesn’t need 16 bits
By default, each weight is stored using 16 bits, which is 2 bytes. That’s roomy: it can represent a number very precisely. But a model’s weights don’t need that much precision. You can store roughly the same value in 8 bits, or even 4.
Fewer bits means a rougher version of each number, but also a much smaller model. And it turns out “rough but close” is good enough for the billions of weights in a language model.
Why it matters: it fits, and it’s faster
Shrink every number from 16 bits to 4 and the whole model shrinks with it.
That buys you two things at once. First, the model actually fits in the card’s memory, with room to spare for the working memory it needs while it runs. Second, it runs faster: generating text means hauling all those weights through the chip for every word, and fewer bytes means less hauling. (That speed story has its own post.)
The catch: precision
There’s no free lunch. Fewer bits means each number is less exact, and push it too far and the model gets noticeably dumber.
The honest way to picture it is a JPEG photo. Save it smaller and it loses a little detail you’ll probably never notice; crush it too hard and you see ugly blocks. Quantisation is the same trade. Crude 4-bit hurts; but modern 4-bit formats are clever about it (they keep a little extra scaling information for each small group of numbers), and the result stays remarkably close to the full-size model.
That last trick is exactly what imp leans on: it converts a model to 4-bit when it loads, which is the whole reason a 30-billion parameter model runs on a desktop card at all. If you want the hardware side of why that card can and can’t do certain things, see what a 5090 is missing next to a data centre GPU.