Large Language Model (LLM) Context Windows

Introduction to Context Windows

When interacting with a Large Language Model (LLM), it is easy to anthropomorphize the system and assume it has a continuous, human-like memory. In reality, LLMs are entirely stateless. Every time you send a message to an AI, the system must process the entire conversation history from scratch.

The physical limit of how much text an LLM can process in a single transaction is known as the Context Window.

If the LLM is the brain, the context window is its short-term working memory. It dictates how large of a document you can ask the model to summarize, how much historical chat it can remember, and how much external data can be injected via Retrieval-Augmented Generation (RAG). Understanding the mechanics, limitations, and costs associated with context windows is a fundamental requirement for anyone engineering AI applications.

Tokens: The Currency of Context

To understand context windows, we must first understand how LLMs read text. LLMs do not read words; they read Tokens.

Before text is fed into a neural network, a “Tokenizer” chops the string into sub-word pieces.

• A short, common word like apple is usually one token.
• A longer or less common word like anthropomorphize might be split into 3 or 4 tokens (e.g., anthro, pomorph, ize).
• As a general rule of thumb in English, 100 tokens equates to roughly 75 words.

When an AI provider advertises a “128,000 token context window,” they mean the model can ingest approximately 96,000 words (the length of a short novel) in a single prompt.

Input vs. Output Tokens

The context window is a shared space. It includes the system instructions, the user’s prompt, the injected RAG data (the Input Tokens), plus the space required for the model to generate its response (the Output Tokens).

If a model has an 8,000 token limit, and you provide a 7,500 token prompt, the model will simply stop generating its answer after 500 tokens, resulting in a cut-off, incomplete response.

The Engineering Challenge: The Attention Mechanism

Why can’t we simply build models with infinite context windows? The limitation lies in the core architecture of the Transformer neural network: the Self-Attention Mechanism.

When a Transformer processes text, it compares every single token in the sequence to every other token in the sequence to understand context and relationships. This mathematical operation scales quadratically ($O(N^2)$).

• If you double the number of tokens in the context window, the computational power and VRAM (GPU Memory) required to process it increases by a factor of four.
• A 1-million token context window requires immense clusters of synchronized GPUs just to hold the attention matrix in memory for a single query.

While recent architectural breakthroughs (like Ring Attention and state-space models like Mamba) have pushed boundaries—allowing models like Gemini 1.5 Pro to achieve context windows of 1 to 2 million tokens—massive context windows introduce secondary challenges.

The “Lost in the Middle” Phenomenon

Research has shown that LLMs do not pay equal attention to all tokens within a massive context window. They exhibit a U-shaped attention curve.

• The model perfectly remembers facts located at the very beginning of the prompt (the system instructions).
• The model perfectly remembers facts at the very end of the prompt (the user’s final question).
• However, the model severely degrades in accuracy when trying to retrieve facts buried in the vast “middle” of a long document.

Therefore, simply stuffing a 500-page PDF into a massive context window is often an anti-pattern. The model will likely hallucinate or skip over critical details buried on page 250.

Strategies for Managing Context Limits

Because context is computationally expensive and prone to degradation, AI engineers must employ strategies to optimize it.

1. Retrieval-Augmented Generation (RAG)

Instead of feeding an entire database into the context window, RAG uses semantic search to find only the 3 or 4 paragraphs highly relevant to the user’s question. This keeps the prompt short, cheap, and highly concentrated with relevant facts, drastically reducing the “Lost in the Middle” effect.

2. Conversation Summarization (Memory Rolling)

In a long-running chatbot application, the conversation history will eventually breach the context limit. To prevent the app from crashing, engineers implement “Memory Rolling.” When the chat reaches 80% of the context limit, a background process takes the oldest messages and asks a cheaper, faster LLM to summarize them. The raw messages are deleted from the array and replaced with a dense 100-token summary (e.g., “User and Assistant discussed setting up a Python environment”). This frees up room in the context window while preserving the gist of the historical conversation.

3. Prompt Compression

There are algorithmic tools designed to compress prompts before sending them to the LLM. They strip out stop words, unnecessary whitespace, and redundant phrasing, compacting the semantic meaning into fewer tokens to save costs and space.

Conclusion

The context window is the defining operational constraint of generative AI. While the industry is aggressively pushing toward million-token capacities, the fundamental laws of compute cost and attention degradation remain. Mastery of Prompt Engineering and AI architecture ultimately comes down to treating the context window not as an infinite dumping ground, but as a highly valuable, constrained real estate where only the most relevant, compressed information should reside.