TL;DR: The context window is a model's working memory: the maximum number of tokens it can consider in one request. The system prompt, conversation history, retrieved documents, tool results, and the answer being generated all share that one budget. When it runs out, something gets dropped - and model behavior degrades before the hard limit is even reached.
How it works
A model has no memory between requests. Each API call must carry everything the model should know: the system prompt, every prior turn of the conversation, any documents or tool output, and the user's latest message. The context window is the ceiling on that total, measured in tokens. A "long conversation" with a chatbot is an illusion maintained by resending the transcript on every turn - which is also why long chats get slower and more expensive as they grow.
The limit exists because of how transformers work: attention compares every token against every other token, so cost climbs steeply with input length. Modern windows are large enough to hold whole codebases or several books, but two practical problems remain. First, cost - you pay per token processed, so stuffing the window on every request is the most expensive way to use a model, and prompt caching exists precisely to soften that. Second, attention quality - models demonstrably use information at the start and end of a long context better than material buried in the middle, a failure mode nicknamed "lost in the middle". A fact the model was given can still go unused if it sits in the wrong place.
This is why "just paste everything in" loses to selection. RAG retrieves only the passages relevant to the question rather than shipping the whole knowledge base. Agents summarize or discard old turns to keep room for new work. Coding assistants practice deliberate context management, choosing which files enter the window. A concrete example: a support bot with a 50-page product manual answers better retrieving the three relevant sections per question than pasting all 50 pages - cheaper, faster, and the model actually attends to what matters.
When you hit the ceiling anyway, the options are triage: truncate oldest history, summarize it, or split the task. Every production LLM application eventually implements one of these.
Where it sits in the AI stack
The context window is the funnel every input competes to get through before the model sees anything:
Key tools and implementations
-
Retrieval (RAG)
Selects only the relevant slices of your data so the window holds signal, not bulk.
-
Prompt caching
Reuses the processed prefix of a long prompt across requests to cut cost and latency.
-
Conversation summarization
Compacts old turns into a short summary to keep long sessions inside the budget.
-
Token counters
Libraries and API endpoints that measure a payload before you send it over the limit.
Related entries
- Token The chunk of text - roughly three-quarters of a word - that a language model reads and writes one at a time.
- RAG (retrieval augmented generation) Fetching relevant documents at query time so a language model can answer from your data instead of memory alone.
- Prompt caching Reusing the computed state of a repeated prompt prefix so later requests skip that work.
- Context management Keeping an AI agent's working context relevant and small through compaction, memory files, and delegating work to subagents.