What a GPU is, and why AI needs one

To run AI at home, everyone tells you to get a “GPU”, a graphics card. Which is a little odd, because you’re not playing a video game. So why does generating text need the same chip that draws explosions in games?

A few big workers, or thousands of small ones

The chip you already have, the CPU, has a handful of very powerful cores. It’s brilliant at doing one complicated thing after another, quickly. A GPU is the opposite bet: thousands of much simpler cores that all work at the same time.

Why AI loves a GPU

AI is the same simple sum done across billions of numbers. A CPU does a few at a time, fast; a GPU does thousands at once. That parallelism is the whole reason GPUs run AI.

That suits AI perfectly. Underneath all the cleverness, running a model is the same humble sum, multiply two numbers and add, repeated across billions of weights for every word. A CPU grinds through those a few at a time. A GPU does thousands at once. That’s the whole reason AI moved onto graphics cards.

The number that really matters: memory

Raw speed isn’t the catch, though. The catch is memory.

An RTX 5090 has 32 gigabytes of VRAM. The model’s weights, plus the working memory it needs while it runs, all have to live inside that budget.

What fills a 32 GB card (one RTX 5090)

The model has to fit in here. Quantization is what shrinks it enough to leave room for the working memory it needs while it runs.

So when people compare graphics cards for AI, two numbers do most of the talking. How much memory (that decides how big a model you can run) and how fast that memory is (that decides how quickly it writes each word). On a 5090 those are 32 GB and about 1.79 terabytes per second.

This is also why quantisation, the trick of shrinking the model, matters so much: it’s what gets a big model under that 32 GB ceiling. And if you want to know what this particular card can and can’t do compared to the giant ones in data centres, that’s its own story.