What the model remembers: the context window

People assume a chatbot remembers them the way a person would: you told it something earlier, so it knows it now. That’s not what’s happening, and the real mechanism explains a surprising amount of everyday behaviour, why it “forgets” the start of a long chat, why pasting a huge document sometimes fails, why starting a fresh conversation can fix a model that’s gone confused.

The thing doing all of this is the context window, and it’s worth understanding properly because almost everything about how a chatbot feels to use comes back to it.

The model has no memory. It just re-reads.

Here’s the part that reframes everything. An LLM does not remember your previous messages. Between one reply and the next it keeps nothing. What actually happens is that every time you hit send, the chat software bundles up the entire conversation so far, your messages and its own, and feeds the whole thing back in as one long piece of text. The model reads it all fresh, start to finish, and continues it (the guess-the-next-word loop you already know).

Every turn, the whole conversation goes back in

It "knows" your name because the line where you said it is sitting right there in the text it just re-read, not because it stored a memory. Drop that line out of the window and the knowledge is simply gone.

So when a chatbot “remembers” your name from earlier, it’s not recalling, it’s re-reading. The information is present because it’s physically still in the text being fed in. This one fact is the key to the window: the model only knows what’s in the window right now. Not what you said last week, not what fell off the top of a long chat. If it’s not in the text in front of it, for the model it never happened.

The window has a hard edge

That bundle can’t grow forever. Every model has a maximum context size, the most tokens it can take in at once. Older models had a few thousand; today’s range from tens of thousands to, on the big hosted models, hundreds of thousands or even past a million tokens. Big, but always finite.

When a conversation gets long enough to bump that ceiling, something has to give, and typically the oldest text gets pushed out to make room, like a conveyor belt where the start falls off the end.

Long chat: the oldest turns slide out of the window

Once full, the window slides forward. The earliest messages drop off and, as far as the model is concerned, cease to exist. That's why a very long chat seems to "forget" how it began: it literally can't see that part any more.

That’s the honest mechanism behind “the AI forgot what we agreed at the start.” It didn’t forget in the human sense; that text fell out of the window. (Some products paper over this with separate long-term “memory” features that quietly re-insert saved facts, but that’s a layer on top, not the model remembering by itself.)

Why this explains so much

Once you hold “it only knows what’s in the window,” a lot of chatbot quirks stop being mysterious:

Long chats drift or forget. The beginning has slid out, or there’s now so much text that the relevant bit is buried. Models also tend to attend best to the very start and very end of a long window, and lose things stranded in the middle.
“Start a new chat” genuinely helps. A confused, cluttered conversation carries all its confusion forward every turn, because it’s all re-read every turn. A fresh chat is an empty window: a clean slate, no baggage.
Pasting a giant document can fail or cost a lot. It has to fit in the window alongside your question, and on paid APIs you pay per token for everything in there, every turn.
Context is shared, not free. Document, instructions, conversation and the answer being written all draw on the same fixed budget. Fill it with a huge pasted file and there’s less room for a long reply.

The practical takeaways

You don’t need to count tokens to use this well. A few habits follow directly:

Works with the window

New chat for a new topic, keep it focused
Put the important stuff near your latest message
Paste only the relevant part of a long document
Restate a key constraint if the chat's gone long

Fights the window

One endless mega-thread for everything
Assuming it recalls last week's conversation
Dumping a whole book and asking about page 1
Expecting buried mid-context detail to stick

The context window is the model’s whole world for the length of one reply: wide, but bounded, and wiped clean the moment you open a new chat. Almost everything that feels like memory, or the lack of it, is really about what’s inside that window right now.

If you want the engineering version of why the window has a ceiling and what it actually costs in memory, that’s the KV cache, the real bottleneck behind long context. And the reason a model can confidently answer about text that’s not in its window, by making it up, is its own story.