What Is a Context Window?
One token budget covering your prompt, the conversation history, retrieved documents, and the answer being generated. Everything competes for the same space.
On this page
The context window is the maximum number of tokens a model can consider at once. A “200K context window” means 200,000 tokens.
The part people miss is that this single budget covers everything at once:
- the system prompt
- the full conversation history so far
- any documents or code you pasted
- the current question
- the answer being generated
That last item is the one that catches people. Output shares the same budget as input. Fill 199,000 tokens with input on a 200,000-token model and there is room for roughly a thousand tokens of reply.
Why there is a limit at all
Two reasons, and they are independent.
Attention cost grows with the square of length. Every token compares against every token, so doubling the context quadruples the attention work.
And KV cache memory grows linearly with length, multiplied across every layer and head. For a provider serving many users concurrently, cache memory is usually the binding constraint.
The limit is a resource decision, not an arbitrary product tier.
What happens when you exceed it
Behavior varies by system, and knowing which one you are dealing with matters.
Hard rejection. Raw APIs typically return an error. Clean, and you know immediately.
Truncation. The oldest content gets dropped. Common in chat frameworks and dangerous because it is silent — your carefully written system prompt can quietly fall off the front while the conversation appears to continue normally.
Sliding window. Only the most recent n tokens are retained, continuously. The conversation runs indefinitely but the model genuinely forgets earlier turns.
Summarization. Older history is compressed into a summary that replaces it. Preserves gist, loses specifics, and can silently drop the one detail that mattered.
That is what “why did it forget what I told it earlier” usually is: not a model defect, but a context management strategy operating without telling you.
Advertised length is not usable length
A model that accepts 200,000 tokens may use the middle of them poorly.
Two independent causes. Some architectures use approximate attention that genuinely does not look everywhere. And even with exact attention, the model must have learned to use distant positions — positional patterns at extreme distance were rare in training. The capacity exists; the practiced skill may not.
The observable result is a well-documented pattern: information at the start and end of a long context is used more reliably than information buried in the middle.
The practical response is to place what matters at the edges. If one document is critical, do not bury it at position 40,000 of 80,000.
Managing the budget
Retrieve instead of stuffing. Selecting the right 2,000 tokens usually beats sending 100,000 and hoping. Cheaper and more accurate at once — that is the case for RAG.
Cap conversation history deliberately. Decide what to keep rather than letting a framework truncate silently. Keeping the system prompt plus the last several turns is often better than keeping everything until it breaks.
Put stable content first. Not only for attention reasons — a fixed prefix enables prefix caching, which cuts cost and latency.
Watch the squared growth. Each turn resends the whole transcript, so total tokens across an n-turn conversation grow with n². Long chats get expensive faster than they feel like they should.
Leave headroom for output. If you need a 2,000-token answer, budget for it.
Counting before you send
Every provider publishes a tokenizer, and the rough English heuristics are order-of-magnitude only — they mislead on code, on non-English text, and on anything with unusual formatting.
For anything where hitting the limit matters, count with the actual tokenizer rather than estimating.
What to remember
- One budget covers system prompt, history, documents, question, and the generated answer.
- The limit exists because attention cost is quadratic and KV cache memory is linear-but-large.
- Overflow behavior varies: hard error, silent truncation, sliding window, or summarization — silent truncation explains most “it forgot” complaints.
- Advertised length does not guarantee good use of the middle; put important content at the edges.
- Retrieval beats stuffing; conversation cost grows with the square of length.