A chat forgets once it outgrows its context window

Start here

A conversation that ran smoothly for an hour can suddenly stop referencing something said near the start. That is not the assistant losing interest; it is the context window filling up. Anthropic's context window documentation defines the term as all the text a model can reference when generating a response, and describes it as the model's working memory rather than its trained knowledge. Google's long context documentation uses the same analogy directly: the window behaves like short-term memory, holding what is currently in view rather than everything the model has ever learned.

What the documents say

Vendors now advertise very large windows. Anthropic's page states several current Claude models hold up to 1 million tokens, and OpenAI's model documentation lists roughly 1.05 million tokens for its latest flagship models, each with its own output-token ceiling. But size is not the whole story: Anthropic's documentation names a phenomenon it calls context rot, stating that as token count grows, accuracy and recall degrade even within a window that has not overflowed. Google's documentation makes a related claim: when a prompt asks the model to locate several pieces of information scattered through a long context, the model does not perform with the same accuracy as with a single needle. A bigger window changes what fits; it does not change how reliably the model uses everything inside it.

Check this

A reader who suspects a long chat is degrading can test this directly: ask the assistant to act on a detail established early in the conversation, after many turns have passed, and compare that against asking the same question in a fresh chat that states the detail once. Google's documentation also recommends putting a query at the end of a long prompt rather than the beginning. If a request fails outright, the behavior differs by vendor: Anthropic's documentation states that some newer models stop generation with a named error when output plus input would exceed the window, rather than silently truncating.

What holds and what fails

The working-memory description holds as an explanation for why old instructions seem forgotten partway through a long session: they have not been deleted, but now compete with everything generated since. It fails as a guarantee of accuracy merely because a document technically fits; the vendors' own language about context rot and reduced accuracy on scattered facts says a fitting prompt is not automatically a well-understood one. Whether a task needs a fresh conversation instead of a longer one is a judgment call the documentation leaves to the user, not a rule it states.

• Restate a key fact or instruction partway through a very long conversation rather than assuming it still carries weight.
• Put the actual question at the end of a long pasted document, after the source material.
• Start a new conversation for a new task instead of extending one that has grown very long.

A context window is generous shelf space, not a guarantee that everything on it gets read with equal care. Knowing the mechanism turns a vague frustration with a long chat into a specific choice about when to start over.