Long Context & Lost-in-the-Middle
Modern models advertise context windows of 128K to 1M tokens - but advertising and reality diverge. Simply stuffing a large context window with documents and hoping the model finds the right answer is a recipe for unreliable outputs. Understanding how models actually handle long contexts shapes better system design.
Nominal vs. Effective Context Length
The nominal context length is the maximum number of tokens a model can accept as input. The effective context length is how much of that the model can reliably attend to when answering a specific question.
The gap is real
A model may accept 200K tokens but only reliably recall facts from the first and last 20K tokens of the prompt. Performance benchmarks measuring "can the model find a needle in a haystack" show significant drops for facts placed in the middle of very long contexts - even for models that claim full 200K support.
Lost-in-the-Middle Failure
The "lost-in-the-middle" phenomenon describes a consistent finding: models attend most strongly to the beginning and end of the context window, with degraded recall for content in the middle. Placing critical information in the middle of a 100K-token prompt significantly reduces the probability the model will use it correctly.
Start of context
Highest attention - place system instructions here
Middle of context
Lowest attention - avoid placing critical facts here alone
End of context
High attention - place the immediate task / question here
Practical Limits by Model (2025–2026)
| Model | Nominal window | Notes |
|---|---|---|
| Claude 3.5 / 4 | 200K tokens | Strong long-context recall; extended thinking adds more budget |
| GPT-4o | 128K tokens | Good retrieval but cost scales linearly; use prompt caching |
| Gemini 2.5 Pro | 1M tokens | Best-in-class long context; suited for full codebase or book-length analysis |
| Local models (7B–70B) | 4K–32K typical | Practical limit much lower; long context degrades faster than frontier models |
RAG vs. Long Context
Both approaches aim to give the model relevant information. They have different tradeoffs:
Long context (stuff everything in)
- ✓ Simpler - no retrieval infrastructure
- ✓ Model sees relationships across all documents
- ✗ Expensive at high token counts
- ✗ Lost-in-middle degrades precision
- ✗ Poor fit when corpus is large (>100K tokens)
RAG (retrieve relevant chunks)
- ✓ Works for any corpus size
- ✓ Only relevant content is in context
- ✗ Retrieval quality is the ceiling for answer quality
- ✗ Misses cross-document relationships not in top-k
- ✗ Requires vector DB and embedding infrastructure
The practical heuristic: use long context for small corpora (<50K tokens) where you want full coverage; use RAG for large corpora where selective retrieval is more efficient and cost-effective.
Mitigations for Long Context Issues
- Place the most critical documents at the start or end of the context, not in the middle
- Chunk and re-order retrieved documents by relevance so the top-ranked chunk appears last
- Use prompt caching for the stable portion of long prompts to reduce cost
- For very long contexts, use a smaller model to pre-filter, then the main model on the filtered set
- Test long-context recall with a needle-in-a-haystack eval before deploying