How a model sees a picture
You can hand a modern AI a photo and ask about it. A language model only understands tokens; this is how vision is added to a model that only knew words.
Drop a photo into a modern chatbot and ask “what’s wrong with this diagram?” or “read the handwriting” or “is this safe to eat?”, and it answers. That is new, because everything so far has been about a model that works on text, chopped into tokens. A picture isn’t text.
The short answer: the image gets translated into the same kind of numbers the model already uses for words, and from there it’s treated as just more “tokens” to read. The key component is the translator.
A reminder: inside, it’s all numbers
From what an LLM is: a word never enters the model as letters. It’s turned into a list of numbers, a vector, that captures its meaning, and the model only ever works on those vectors. “Cat” and “kitten” become nearby vectors; “cat” and “lorry” become distant ones. The model’s whole skill is manipulating these meaning-vectors.
This is how vision fits in. If you can turn an image into the same kind of meaning-vectors, the model doesn’t care that they didn’t come from words. To it, they’re just more numbers in the stream to be reasoned about.
The translator: a vision encoder
So the job is: picture in, meaning-vectors out. That’s done by a separate component, a vision encoder, sitting in front of the language model.
Two things make this work. First, the image is broken into a grid of small patches (think of a mosaic), and each patch becomes one vector, so a picture turns into a handful, or a few hundred, image “tokens”. Second, that encoder was trained on huge numbers of image-and-caption pairs, so it learned to map visual content onto vectors that line up with the language meaning. A patch of orange fur ends up near the region where the word “cat” lives. That alignment is what lets the model talk about what it sees.
Once the image tokens are in the stream, there’s no special “looking” step. The model just continues the combined sequence of word-tokens and image-tokens with its usual next-token guessing, which is why it can answer questions, describe, compare, even read text out of a photo, all with the same machinery.
Strengths and limits
Vision is useful, but its quality is uneven. Knowing the shape helps you trust it in the right places.
- Strong at the gist. What’s in the scene, describing it, classifying it, drafting alt-text, explaining a chart’s overall message. The broad “what am I looking at” is often very good.
- Shaky on the precise. Exact counts (“how many people?”), fine spatial detail, reading tiny or messy text, precise measurements. The patch grid blurs fine detail, and the model is guessing the plausible answer, which for “count the objects” can be confidently wrong.
- OCR is decent, not perfect. It’ll often read clear printed text well and make errors on handwriting, low contrast, or unusual layouts. Treat anything it transcribes from an image as a draft to check, not a scan.
- It can be misled by what it expects. Show it a clock and it may “read” a typical time rather than the one actually shown, because it favours the plausible over the literal, the same tendency it shows with text.
The same idea, other senses
Nothing here is specific to pictures. Audio works the same way: a sound encoder turns speech into vectors, and the model reads them, which is how voice assistants understand you and how a model can summarise a meeting recording. Some models go the other direction too, generating images or speech, by running the translation in reverse. It’s all the same recipe: convert the modality into the model’s number language, or convert the number language back out. Image generators take a different road from there, covered in how an image generator creates a picture from noise.
This is also exactly how the non-LLM AI worlds of vision and speech, long separate fields, got folded into one conversational model. The unifying move was realising that if everything becomes meaning-vectors, one model can reason across all of it at once.
The takeaway
A multimodal model hasn’t gained a new sense so much as a new translator. The image gets converted into the same meaning-vectors the model already used for words, and then it’s business as usual: read the combined stream, guess what comes next. That’s why its vision is fluent and at the same time sometimes misses literal detail: it is the same pattern machine, now fed pictures. Reliable for the gist, worth double-checking on precise detail, as with words.