AI that isn't an LLM

Almost everything on this site is about large language models, because that’s what I build for. But an LLM is one family of AI, not the whole field, and reaching for one when your problem isn’t “produce text” is a common and expensive mistake. Here is a tour of the other big families and what each is actually good at.

AI is bigger than LLMs

First, one thing this post is not about. My post on models that aren’t transformers covers LLMs built on a different internal design. Those are still LLMs doing the LLM job. This post is about systems that do an entirely different job.

Diffusion models: making images and video

When you generate an image from a prompt, there’s no LLM doing it. That’s a diffusion model, and it works in a completely different way: start with pure visual noise, then repeatedly clean it up, step by step, until an image matching your prompt emerges from the noise.

It’s the opposite shape from an LLM’s one-word-at-a-time loop: the whole image is refined at once, over and over, rather than built left to right.

Embedding models: turning meaning into numbers

Some of the most useful AI never generates anything. An embedding model reads a piece of text (or an image) and outputs a list of numbers that captures its meaning, so that similar things land near each other.

This is the quiet workhorse behind semantic search, recommendations, deduplication, and the “give the agent a search tool” trick from how agents use memory. It doesn’t write, it measures similarity, and that’s exactly what a lot of real products need.

Speech: from sound to text and back

Two more specialised families handle audio, and neither is an LLM:

A talking voice assistant is usually three models in a row: speech-to-text, then an LLM to decide what to say, then text-to-speech. The LLM is just the middle link.

Vision and classical models: the unglamorous majority

Long before the generative wave, AI was mostly about classifying things, and a huge amount of it still is.

• Vision models look at an image and answer a fixed question: is there a tumour in this scan, a defect on this part, a pedestrian in this frame. They detect and label, they don’t paint.
• Classical machine learning (decision trees, gradient boosting, regression) works on the spreadsheet-shaped data that runs most businesses: will this customer churn, is this transaction fraud, what will demand be next week. For tabular data, these often beat anything fancier, and they’re cheap and explainable.
• Recommender systems decide what to show you next on a shop or a streaming service, learned from what people like you clicked.

None of these produce sentences, and for their jobs an LLM would be slower, costlier, and usually worse.

So when is an LLM the right tool?

The honest test is about the shape of your problem, not the marketing.

In practice the strongest systems combine them. An agent might use an LLM to plan, call an embedding model to search, and hand off to a diffusion model to make a picture, each family doing the part it’s best at. The LLM is a remarkable tool. It just isn’t the only one in the box, and knowing the rest of the box is what keeps you from using a chatbot to do a spreadsheet’s job.