Why models make things up

You ask a model for a quote, a court case, a library function, a date. It gives you one, fluently, in exactly the right format, with the confident tone of something read straight off a reference shelf. Except it’s wrong. The quote was never said, the case doesn’t exist, the function isn’t real. The unsettling part isn’t the mistake, it’s the confidence. It sounds exactly as sure as when it’s right.

This is usually called hallucination, and it’s the single most important thing to understand before trusting an AI with anything that matters. The good news: once you see why it happens, it stops being mysterious, and you can work around it.

It was never trying to be right

Here’s the root of it, and it’s worth sitting with because it reframes everything. An LLM was not built to retrieve facts. It was built to continue text plausibly, one token at a time, by predicting the most likely next word (see what an LLM is). That’s the whole job. Truth was never the target; plausibility was.

Most of the time those two line up. The most plausible continuation of “the capital of France is” really is “Paris”, because that pairing appears constantly in its training text, so the patterns are strong and the answer is right. The trouble starts when you ask something where the plausible-sounding answer and the true answer come apart.

A made-up citation is a perfect example. The model has seen thousands of real citations, so it knows exactly what one looks like: author, year, plausible title, journal, page numbers. Asked for a source it doesn’t have, the most plausible continuation is a string that has all those features, a citation-shaped object. It isn’t copying a real one and getting it wrong; it’s generating something that fits the pattern of “citation” beautifully and happens to correspond to nothing.

Why it sounds so sure

The confidence trips everyone up, so it’s worth being precise about it. A human who doesn’t know something usually sounds unsure, they hedge, they pause. The model has no such tell, for a simple reason: fluency and accuracy are produced by the same machinery, and it’s only ever good at one of them.

The smoothness of the sentence is a measure of how well the words fit together, not how well they fit reality. A false answer can be just as grammatically smooth, just as well-structured, as a true one, so it comes out sounding identical. The model isn’t suppressing doubt; from the inside there’s often nothing that flags the fabrication as different from a fact. Confidence in the prose is not evidence about the content, and that decoupling is the trap.

When it’s most likely to happen

Hallucinations aren’t random. They cluster in predictable places, and knowing them tells you exactly when to be on guard.

• Specific facts at the edges of common knowledge. Exact dates, statistics, prices, who-said-what, the population of a small town. The more precise and obscure, the thinner the training signal, and thin signal is where it fills in.
• Anything you ask it to cite. Sources, page numbers, case law, study titles, quotes. Citation-shaped fabrication is a classic, precisely because the shape is so well learned.
• APIs, functions and config. It’ll happily invent a method name or a flag that should exist by the look of the rest of the library, because plausible-by-pattern is exactly what it does.
• The very recent, and the very niche. Anything after its training cut-off, or too rare to have appeared much, it has no real basis for, so it improvises.
• When you lead it. Ask “what’s the name of the law that bans X?” and it may conjure a law to satisfy the question, even if none exists. A confident question invites a confident answer. Leave it room to say “there isn’t one.”

What actually helps

You can’t switch it off, but you can shrink it a lot. None of this is exotic.

The single biggest lever is give it the facts rather than asking it to recall them. A model handed the actual document and told to answer from that text is far more reliable than one asked to remember, because now the plausible continuation and the true answer are forced to coincide, the right answer is sitting right there in front of it. That’s the whole idea behind connecting a model to your own documents, and it’s why serious AI products lean on it so heavily. It turns “recall a fact” into “read and quote”, which is a job the machine is genuinely good at.

The rest is judgement. Modern models, especially the large hosted ones with web search, hallucinate far less than they did a couple of years ago, and many will now tell you when they’re unsure or when a search came up empty. But “far less” is not “never”, and the failure mode is invisible by design. So the rule that’s served me well: the more specific, consequential and checkable a claim is, the more it’s worth checking. Use the model to draft, to explain, to find the shape of an answer, then verify the load-bearing facts yourself. That’s not distrust, it’s just using the tool for what it is, a brilliant generator of plausible text, which is a different thing from an oracle.

If you’ve read this, the different-answers post and the one on tokens, you’ve now got the three things that surprise people most about how these models behave, and most of what you need to use one without being caught out.