2172 words | Esment
The context window.
Why a model forgets the start of a long conversation.
╌╌╌╌
Have a long enough conversation with a chatbot and something odd happens. Details you gave it at the start stop being used. It asks for something you already told it, or confidently contradicts a decision the two of you made an hour ago. It has not grown tired, and nothing has been deleted. The beginning of the conversation has simply fallen outside the only thing the model can see.
That thing is the context window3: the stretch of text a model reads before it writes each token of its reply. Everything a model knows about you, right now, in this conversation, has to fit inside it. Understanding the window, its size, its cost and its blind spots, explains most of the strange ways assistants forget.
What the model actually sees
The previous chapter ended with a loop: read all the text, predict one token10, append it, read everything again. The context window is the limit on “all the text”. It is measured in tokens, and it has to hold several things at once.
- The system prompt9: instructions written by whoever built the app, often thousands of tokens long, which you usually never see.
- Anything added on your behalf: uploaded files, search results, saved memories, descriptions of the tools the model may use.
- The conversation so far, every message from you and every reply, in order.
- And the answer being written, which takes up room too, token by token, as it grows.
Here is the part that surprises most people. A model is stateless8. It does not keep the conversation anywhere between turns. Each time you press send, the app gathers all of the above into one long piece of text and sends the whole thing again, from the very first word. The conversation “exists” only because it is re-read, in full, every single time.
How big is a window?
Window sizes have grown astonishingly fast. GPT-2, in 2019, could see 1,024 tokens, roughly two pages. When ChatGPT launched at the end of 2022, its window was about 4,000 tokens. In 2023 models arrived that could read 100,000 tokens, and in 2024 Google showed one that could take in a million.
A token is about three quarters of an English word, so 128,000 tokens is roughly 96,000 words, the length of a typical novel. A million tokens is a small shelf of them. These are impressive numbers, and they have led many people to assume the problem of forgetting is solved. It is not, for three reasons this chapter walks through: windows still run out, they are expensive to fill, and models do not read them evenly.
What happens when it runs out
A model cannot read past the edge of its window. If the text is longer, something has to go, and the app, not the model, decides what. The simplest strategy is to keep the system prompt and drop the oldest turns of the conversation until the rest fits. This is called truncation11.
This is why long conversations go wrong in such a particular way. Nothing fades gently. A turn is inside the window, fully available, and then it is outside, completely gone, and the model has no idea it ever existed. If that turn contained your name, the constraint you set, or the decision you reached together, the model will carry on cheerfully without it.
Better apps do something smarter than cutting. Some ask a model to write a short summary of the oldest part of the conversation and put the summary in its place, a process often called compaction2. It saves room, but a summary is a choice about what mattered, and details that seemed unimportant at the time are the ones that vanish.
Why bigger is expensive
If windows are so useful, why not make them infinite? Because of attention1. Every token is compared with every token before it, so the work grows with the square of the length. Four tokens make 16 pairs; eight make 64; sixteen make 256.
At the scale of real windows this matters a great deal. A 100,000-token prompt means billions of comparisons in every attention head of every layer, before the model writes a single word of its answer. Engineers have found many clever ways to soften this cost, and windows keep growing because of them, but the basic shape remains: more context is more work, and you pay for it in time and in money. Most providers charge by the token, for what you send as well as what the model writes.
The memory that grows
There is a second cost, and it is the one that sets hard limits. Recall that attention gives every token a key and a value. When the model writes its next token, the keys and values of all the earlier tokens are still needed, and they have not changed. Recomputing them every step would be absurdly wasteful, so they are stored in the KV cache4.
The cache makes generation fast, but it lives in the GPU’s memory, and it grows with every token. For a large model, each token’s keys and values, across every layer, can take up hundreds of kilobytes. A long conversation can need many gigabytes of cache, for one user, on top of the model itself. This is the practical reason windows have edges: not that the model cannot be shown more text, but that holding it all ready is expensive hardware.
Lost in the middle
Suppose the text fits and cost is no object. Does the model use all of it equally well? Not quite. In 2023 a group of researchers led by Nelson Liu hid a single relevant document among many irrelevant ones and moved it around. Models were best at using it when it sat near the beginning or the end of the context, and noticeably worse when it was buried in the middle. They called the effect lost in the middle5.
Newer models have improved a great deal, and they pass simple tests of finding one fact in a huge document. But the general lesson has held up: a model’s attention is not spread evenly, and a sea of loosely related text dilutes the parts that matter. More context is not free even when it fits. It is noise as well as signal.
Choosing what goes in
Put these limits together and a clear strategy emerges. The window is not a filing cabinet to be stuffed. It is a desk, and the skill is in deciding what to put on it for the task at hand.
Instead of pasting every note, file and old chat into the conversation, a well-built system keeps them outside the model and, for each question, fetches the few pieces that are actually relevant. This is called retrieval6, and it usually works by the semantic search7 described in the embeddings chapter: turn the question into a vector, find the stored pieces nearest to it, and place only those in the window.
Retrieval turns the window from a limit into a budget. A small slice goes to instructions, a small slice to the handful of things worth remembering right now, and the rest is left for the conversation and the answer. The hard questions move elsewhere: what to store in the first place, how to find it again, and when to let it go.
That gives us the full picture of the foundations. Text becomes tokens, tokens become vectors, attention lets the vectors shape each other, and the window decides how many of them can take part. What none of it does, on its own, is remember. How a machine can keep something from one conversation to the next, and what it means for it to do so, is where the next part of this manual begins.
Glossary
1Attention — The mechanism that lets each word in a passage build a new version of itself from the other words, weighting each one by how relevant it is.
2Compaction — Replacing the oldest part of a long conversation with a shorter summary written by a model, so that the rest still fits in the context window.
3Context window — The maximum amount of text, measured in tokens, that a model can read at once. Everything the model uses to write its reply has to fit inside it.
4KV cache — The stored keys and values of every token read so far, kept in GPU memory so they do not have to be recomputed for each new token. It grows with the length of the context.
5Lost in the middle — The tendency, described in a 2023 study, for models to use information near the start or end of a long context better than information in the middle.
6Retrieval — Fetching only the stored pieces of text relevant to the current question and placing them in the context window, instead of including everything.
7Semantic search — Finding stored text by meaning rather than by matching words: embed the question, then return the stored items whose embeddings are nearest to it.
8Stateless — Keeping nothing between requests. A language model does not hold a conversation in between turns; the whole conversation is sent again each time.
9System prompt — Instructions written by whoever built an app, placed at the start of the context window before the conversation. Users usually do not see it.
10Token — One of the pieces a tokenizer cuts text into: a whole common word, a fragment of a rarer one, or a single character. Context windows are measured in tokens.
11Truncation — Dropping the oldest parts of a conversation so that the rest fits in the context window.
Get the next chapter by email.
One email per new chapter. Nothing else.