How imp turns a model file into words

If you’ve used ChatGPT or Claude, you’ve watched an answer appear on screen word by word. Something has to do that work: read your message, then produce a reply one piece at a time. That something is called an inference engine (a fancy name for “the program that runs a trained AI model and produces its answers”).

imp is one I built to run on a single gaming graphics card instead of a room full of data centre hardware. This is a tour of what happens inside it, written for people who do not write GPU code. I will explain the jargon as we go.

The engine really does just two things: a slow setup that happens once, and a fast answer that happens every time you send a message.

Part one: loading the model (happens once)

When imp starts, it reads the model file off your disk and copies those weights onto the graphics card, where the maths can run fast.

While it’s at it, imp loads the tokenizer (more on that in a second) and sets aside some scratch space on the card for your conversation. Two small but important tricks happen here. It shrinks every weight to a smaller size so there’s less data to shuffle around later, which is the biggest single lever for the memory-bound writing (decode) phase (that trick has its own post). And it records the sequence of graphics-card operations for the per-word steps once, so it can replay that same sequence quickly for every word it writes afterwards. The maths still runs each time on new data; it’s the launch sequence that’s captured, not the results.

By the end of part one, the model is loaded and waiting.

Tokens: how the model “sees” your text

Computers don’t read letters, they read numbers. So before anything else, your text is chopped into tokens and each token is swapped for an ID number.

Here is that very first step made visible:

From here on, the model only ever deals in numbers. When it answers, it produces numbers too, which get turned back into readable text at the very end.

Part two: answering your message, in four steps

Every message you send runs through the same four steps.

1. Chop it up (tokenize)

   Your text is turned into token numbers, the step you just saw above. This is a fast, CPU-only step (no graphics card needed yet).

2. Read the question (prefill)

   The model reads your whole message in one go and jots down notes it will need. This part is fast because it can read everything at once, in parallel.

3. Write the answer (decode)

   Now it writes the reply one token at a time. Each round it glances at its notes and works out a probability for every possible next token (a whole distribution, not a single pick). A part called the sampler then draws one token from that distribution, usually not the single most likely one, which is why the same prompt can give you different answers. It writes that token down and repeats. This loop runs once per word until the model decides the answer is finished.

4. Back to text (detokenize)

   The token numbers are turned back into words and sent to your screen as they're produced. That's why you see the answer type itself out.

Why reading and writing are different jobs

Steps 2 and 3, “read the question” and “write the answer”, sound similar but they push on the graphics card in opposite ways. Reading is one big burst of maths. Writing is the same tiny step repeated hundreds of times.

This is the one idea that makes everything else click: an engine that is fast at reading is not automatically fast at writing. They need different tricks, and most of the hard work in imp is being good at both.

The notes it keeps: a short-term memory

I keep saying the model “jots down notes”. Those notes have a name.

Picture a parking garage split into small numbered bays. Each bit of the conversation gets parked in a bay, and the model keeps a little permit listing which bays are its own. When it needs to look back at earlier words, it reads from those bays instead of starting over.

That’s the trick that lets a single card hold a very long conversation: it only uses as much memory as the conversation actually needs.

How your app talks to it

All of this sits behind a normal-looking connection. imp runs as a small server, and your program sends it a request in the same format it would send to OpenAI or Anthropic. So a tool built for the OpenAI or Anthropic API can use imp by pointing it at imp’s URL (the two formats differ in their details, so it’s not quite “no changes at all”, but the shape is the same): they ask a question, imp runs the four steps, and the words stream back.

That’s the whole engine at this height. Underneath sits the genuinely hard part, the hand-written programs (called kernels) that do the actual maths on the graphics card and are tuned to run efficiently on the chip. Those, and everything else in imp, were written with Claude Code; my job was deciding how the pieces fit together. If you want to see the real thing, it’s all at github.com/kekzl/imp.