How a model sees a picture
You can hand a modern AI a photo and ask about it. But a language model only understands tokens, so what happens to the image? A plain-words look at how vision gets bolted onto a model that only ever knew words.
Drop a photo into a modern chatbot and ask “what’s wrong with this diagram?” or “read the handwriting” or “is this safe to eat?”, and it answers. That’s genuinely new, and a little strange, because everything we’ve said so far is about a model that works on text, chopped into tokens. A picture isn’t text. So what’s actually going on?
The short answer: the image gets translated into the same kind of numbers the model already uses for words, and from there it’s treated as just more “tokens” to read. The clever part is the translator.
A reminder: inside, it’s all numbers
From what an LLM is: a word never enters the model as letters. It’s turned into a list of numbers, a vector, that captures its meaning, and the model only ever works on those vectors. “Cat” and “kitten” become nearby vectors; “cat” and “lorry” become distant ones. The model’s whole skill is manipulating these meaning-vectors.
That’s the loophole vision walks through. If you can turn an image into the same kind of meaning-vectors, the model doesn’t care that they didn’t come from words. To it, they’re just more numbers in the stream to be reasoned about.
The translator: a vision encoder
So the job is: picture in, meaning-vectors out. That’s done by a separate component, a vision encoder, sitting in front of the language model.
Two things make this work. First, the image is broken into a grid of small patches (think of a mosaic), and each patch becomes one vector, so a picture turns into a handful, or a few hundred, image “tokens.” Second, that encoder was trained on huge numbers of image-and-caption pairs, so it learned to map visual content onto vectors that line up with the language meaning. A patch of orange fur ends up near the region where the word “cat” lives. That alignment is the whole game: it’s what lets the model talk about what it sees.
Once the image tokens are in the stream, there’s no special “looking” step. The model just continues the combined sequence of word-tokens and image-tokens with its usual next-token guessing, which is why it can answer questions, describe, compare, even read text out of a photo, all with the same machinery.
What it’s good at, and where it stumbles
Vision is genuinely useful, and genuinely uneven. Knowing the shape helps you trust it in the right places.
- Strong at the gist. What’s in the scene, describing it, classifying it, drafting alt-text, explaining a chart’s overall message. The broad “what am I looking at” is often excellent.
- Shaky on the precise. Exact counts (“how many people?”), fine spatial detail, reading tiny or messy text, precise measurements. The patch grid blurs fine detail, and the model is guessing the plausible answer, which for “count the objects” can be confidently wrong.
- OCR is decent, not perfect. It’ll often read clear printed text well and trip on handwriting, low contrast, or unusual layouts. Treat anything it transcribes from an image as a draft to check, not a scan.
- It can be misled by what it expects. Show it a clock and it may “read” a typical time rather than the one actually shown, because plausible beats literal. The same pattern-over-truth tendency as everywhere else.
The same idea, other senses
Nothing here is specific to pictures. Audio works the same way: a sound encoder turns speech into vectors, and the model reads them, which is how voice assistants understand you and how a model can summarise a meeting recording. Some models go the other direction too, generating images or speech, by running the translation in reverse. It’s all the same recipe: convert the modality into the model’s number language, or convert the number language back out.
This is also exactly how the non-LLM AI worlds of vision and speech, long separate fields, got folded into one conversational model. The unifying move was realising that if everything becomes meaning-vectors, one model can reason across all of it at once.
The takeaway
A multimodal model hasn’t gained a new sense so much as a new translator. The image gets converted into the same meaning-vectors the model already used for words, and then it’s business as usual: read the combined stream, guess what comes next. That’s why its vision feels so fluent and occasionally so oddly literal-blind at the same time, it’s the familiar pattern machine, now fed pictures. Powerful for the gist, worth double-checking on the precise, exactly like its way with words.