Why you get a different answer every time

Type the same question into a chatbot twice and you’ll often get two different answers, sometimes worded differently, sometimes genuinely different in substance. A calculator never does this. So is the model broken, or making it up as it goes?

Neither. The variation is on purpose, and once you see where it comes from, a lot of otherwise-baffling AI behaviour clicks into place. It comes down to one small step at the very end, and one knob that controls it.

A reminder: the model ranks, then picks

From what an LLM is: the model writes one word at a time, and for each next word it doesn’t produce a single answer. It produces a score for every word in its vocabulary, a ranked list of how likely each one is to come next.

"The weather today is ___" - the model's scored options

Not one answer, a whole spread. "sunny" leads, but "cold", "lovely" and the rest are live options too. The question is what to do with this list, and that's where the choice happens. Numbers illustrative.

Now there’s a decision. You’ve got this ranked list, how do you turn it into one actual word? There are two honest strategies, and the difference between them is the whole story.

Two ways to pick

Always take the top one. Simple, predictable. Pick “sunny” every time, no exceptions. This is called greedy decoding, and with the same input it gives the exact same output, every run. Repeatable, but it tends to be flat and a little dull, and it can get stuck in loops, repeating a phrase because the safest next word keeps pointing back into the rut.

Roll weighted dice. Most of the time you’ll still land on “sunny”, but now and then you’ll get “cold” or “lovely”, in rough proportion to their scores. Do this at every word and the answers fan out: different openings, different phrasings, sometimes a different conclusion. That’s the variation you see, and it’s why the text feels written rather than stamped out. Almost every chatbot samples by default, because greedy output reads as robotic.

So the same question gives different answers for the same reason rolling dice gives different numbers: there’s deliberate randomness in the final pick. Not confusion, not memory, just a weighted draw repeated hundreds of times.

The knob: temperature

How loaded are those dice? That’s set by a single dial called temperature, and it’s worth understanding because it’s the one sampling setting you’ll actually run into, on APIs, in playground sliders, in tools that let you tune a model.

Think of it as how much the model is allowed to wander from the safest path.

The same five options, reshaped by temperature

Same scores, two different settings. Low temperature (top) pushes nearly all the weight onto "sunny", so output is focused and repeatable. High temperature (bottom) levels the field, so the underdogs win often enough to make things creative, or to wander off. Illustrative shapes.

There’s no universally “right” setting; it depends on the job:

Low (near 0): when you want consistency and correctness, factual answers, extracting data, code, anything where one right answer beats five colourful ones.
Middle (~0.7, the common default): natural, varied conversation that still mostly behaves. Where most chatbots sit.
High (1+): brainstorming, fiction, jokes, when you want surprise and don’t mind the odd swerve into nonsense.

There’s a related knob you might see, top-p (also called nucleus sampling), which trims the list to only the most likely words before rolling. Same goal: keep the dice from ever landing on a genuinely silly choice while still allowing variety. You rarely need to touch either as a casual user; they’re there for when you do.

Why this is worth knowing

Two practical things fall out of this.

First, “give me the same answer every time” is a setting, not a wish. If you need reproducibility, ask for low temperature (or a tool that lets you set it), and the randomness mostly goes away. Some setups also expose a seed that fixes the dice entirely. Variation is a default, not a law.

Second, and more important: the variation is not the model knowing several truths. When you ask a factual question and get two different answers, that’s not the model weighing up evidence, it’s the dice landing differently on the wording, and sometimes dragging the substance along with them. A confident, fluent answer is just the most probable path through the words, and “most probable” is not the same as “true”. That gap is exactly where models make things up, which is the next thing worth understanding, and it’s also why the whole input matters more than people expect, right down to how words are chopped into tokens before any of this even starts.