Solutions for
Please select
Try the Web App
ki-konzepte

The Context Window of LLMs — What Fits and What Doesn't

Redaktion ·

The context window is one of the most important and most misunderstood properties of a language model. In short: it’s the model’s working memory for a single request — the maximum number of tokens it can consider at once. Anything that fits can be processed. Anything beyond it has to be solved differently. This article explains what the window is, how big it is today, where its limits lie, and when you need RAG instead.

What the context window is

An LLM has no memory in the human sense. With each request it receives a block of text — system prompt, conversation history, your question, embedded documents — and produces an answer from it. The context window is the upper limit for that block plus the answer, measured in tokens.

Important: the window covers input and output together. If a model has a window of 200,000 tokens and your input already uses 195,000, only 5,000 are left for the answer. Stuff the whole space with context and you leave the model no room to respond.

A token is not a word but a text fragment — as a rough rule, about 0.75 words in English. 200,000 tokens correspond roughly to a thick book.

How big modern windows are

The sizes have exploded over the past few years. A snapshot (as of 2026):

  • Early LLMs had windows of a few thousand tokens (4k, 8k).
  • The middle generation sat at 32k to 128k tokens.
  • Current top models reach up to 1 million tokens and beyond — enough for entire codebases, multiple long PDFs, or extensive conversation histories.

These numbers shift quickly, so any concrete figure is a snapshot. But the trend is clear: bigger. Still, “it fits” doesn’t automatically mean “it gets used well” — and that’s where the next phenomenon comes in.

Lost in the middle

A large window is no guarantee that the model uses everything in it equally well. The study Lost in the Middle (Liu et al., 2023) revealed a robust pattern: models use information at the beginning and end of a long context far better than information in the middle. Retrieval performance follows a U-curve — relevant facts placed in the middle of a long document are found less reliably than the same facts at the edges.

Practical consequence: just because you throw in 500,000 tokens doesn’t mean the model finds the crucial sentence on page 300. The most important information belongs at the start or the end of your prompt, not buried in the middle of a wall of text. Newer models have mitigated this problem, but it hasn’t disappeared.

Cost and latency

Long contexts aren’t free. Two effects:

Cost. You pay per input token. A prompt with 800,000 tokens costs a hundred times more per request than one with 8,000. Dump half the wiki into every prompt out of convenience and you burn money on context the model often doesn’t even need.

Latency. More input means more computation before the first answer token appears. Long contexts make the response noticeably slower. For interactive applications (chat, voice) that’s a real problem.

A common countermeasure is prompt caching: the stable part of a prompt (such as a long system prompt or a fixed document) is cached by the provider and processed more cheaply and quickly on follow-up requests. This significantly reduces cost and latency for recurring contexts.

The distinction: context vs. long-term memory and RAG

Here lies the core misconception. The context window is ephemeral: it exists only for the duration of one request. After the answer it’s gone. The model “remembers” nothing — for every new request you have to send the relevant context again.

If your body of knowledge is larger than the window (an entire document collection, a database, a company wiki), you can’t simply pack it all in. The solution is RAG (Retrieval-Augmented Generation): instead of forcing all knowledge into the window, only the relevant part is fetched from an external store for each request and inserted into the context.

So the practical rule is:

  • The context window holds what’s needed for this specific task: the current question, the relevant conversation history, the few documents that matter right now.
  • RAG fetches what’s too large, too volatile, or too rarely needed to carry permanently: the knowledge base, from which only matching excerpts are retrieved.

A larger window doesn’t make RAG obsolete — it just shifts the threshold at which you need it. Even a 1-million-token window is finite against a growing document collection, expensive, and subject to the lost-in-the-middle effect.

FAQ

Is a larger context window always better? Not automatically. It allows more input, but long contexts cost more, are slower, and suffer from lost in the middle. A precise, short context often beats a huge, stuffed one. Bigger is an option, not an end in itself.

What does it mean that input and output share the window? The tokens of your prompt and the tokens of the answer count together against the limit. Fill the window almost entirely with input and there’s barely room for a detailed answer — the model gets cut off mid-response.

Does the model remember earlier conversations? No, not on its own. The context window is ephemeral and exists only per request. “Memory” across sessions only arises if the application resends earlier content or supplies it via RAG/external storage.

When do I need RAG instead of a large window? As soon as your body of knowledge is larger than the window, changes frequently, or would be too expensive to send in full every time. RAG fetches only the relevant excerpts per request — cheaper, faster, and it sidesteps the lost-in-the-middle effect.

What exactly is lost in the middle? An empirically documented pattern: models use information at the beginning and end of a long context better than in the middle. In practice: don’t bury important content in the middle of a long input — put it at the edges.

Themenuebersicht

KI-KonzepteLLM-Grundlagen