Context Windows

Why This Matters

The context window explains the most common practical frustrations with LLMs: why a chatbot "forgets" the start of a long conversation, why you cannot just paste an entire codebase, why RAG exists at all, and why long-document features cost so much. Understanding it turns mysterious limitations into simple arithmetic.

Everyday Analogy

A context window is a whiteboard of fixed size. Everything the model knows about your conversation, including instructions, documents, chat history, and the reply it is writing, must physically fit on that whiteboard. When it fills up, something must be erased before anything new fits. The model's training knowledge is the library it memorized long ago; the whiteboard is the only place where today's information lives.

What Counts Against the Window?

Everything in the current request draws from one shared budget, not separate pools:

• The system prompt (instructions, persona, rules)
• Retrieved context (RAG chunks, pasted documents)
• The conversation history (every prior turn you resend)
• The user's current message
• The response being generated (output tokens share the same window)

A "200K context" model does not give you 200K tokens just for your documents; it gives you 200K for all of the above combined.

How Big Are Context Windows?

Sizes have grown a thousandfold in a few years:

Rules of thumb: 100K tokens ≈ 75,000 words ≈ a 300-page book. A million tokens ≈ roughly 10 novels or a mid-sized code repository.

Bigger Is Not Automatically Better

Three costs arrive with long context:

1. Money. You pay per input token on every call. Resending a 150K-token document on each of 20 conversation turns bills you for 3 million input tokens.
2. Latency. Processing 200K tokens before the first output token takes noticeably longer than processing 2K.
3. Attention quality, known as "lost in the middle." Research shows models recall information at the start and end of a long context far better than the middle. Stuffing 500 pages in and asking about page 250 is measurably less reliable than retrieving the right page and sending only that.

This is precisely why RAG (AI-016) still matters in the million-token era: retrieving the right 2K tokens usually beats stuffing 200K and hoping.

Worked Example: Budgeting a Support Bot

Your model has a 128K window, but you design to a much smaller working budget for cost and quality:

The remaining headroom is a safety margin, not free space to fill. Production systems that "design to the max" have the worst cost and latency profiles.

Strategies When Content Exceeds the Window

• Retrieval (RAG): index the corpus and fetch only relevant chunks per query. This is the default answer.
• Summarization / compaction: replace old conversation turns with a running summary; chatbots do this invisibly.
• Chunked map-reduce: summarize a huge document section-by-section, then summarize the summaries.
• Sliding window: keep the system prompt plus the most recent N turns; drop the middle.
• Prompt caching: providers can cache a long, unchanging prefix (like a big system prompt) so you stop paying full price to resend it every call.

Real-World Showcase

• Claude's 200K window made "paste an entire annual report and ask questions" a real workflow. Analysts now do in one prompt what used to require a RAG pipeline.
• Gemini's 1M window demoed ingesting a full codebase and a 44-minute silent film, showing that long context itself can be a product differentiator.
• ChatGPT's conversation memory quietly summarizes older turns as chats grow, so you have experienced context compaction without noticing it.
• Cursor and Claude Code spend significant engineering on deciding which files enter the window, because context selection, not context size, is their competitive edge.

Try It Yourself

1. Take a long article (5,000+ words). Paste the whole thing into a chatbot and ask a question whose answer sits in the middle. Note the quality.
2. Now paste only the relevant section and ask again. Compare precision. You have just demonstrated why retrieval beats stuffing.
3. Estimate your own budget: using the Token Counter tool on this site, measure your typical system prompt, one conversation turn, and a typical document chunk. Fill in the budgeting table from this lesson for your own use case.

Common Mistakes

• Confusing the context window with model memory: nothing persists between API calls unless you resend it.
• Designing to the maximum window instead of a working budget, then discovering the cost and latency in production.
• Ignoring "lost in the middle" and burying the critical instruction on page 200 of pasted context.
• Forgetting output tokens share the window, so a full input leaves no room for the answer.
• Assuming a bigger window makes RAG obsolete, when retrieval still wins on cost, latency, and mid-context accuracy.

Key Takeaways

• The context window is the model's entire working memory, measured in tokens, shared by input and output.
• Every turn resends everything, so cost scales with context length times conversation length.
• Models recall the start and end of long contexts best; the middle is least reliable.
• Budget slots deliberately: system prompt, retrieval, history, user input, reserved output.
• Retrieval, summarization, and caching are how real products live within the budget.