Context Engineering Overview
Context engineering is the discipline of deliberately designing what information goes into an LLM's context window - and what stays out. The term emerged in 2025 as practitioners discovered that the quality of the context often matters more than the wording of the prompt. A perfectly phrased question with the wrong context will fail; a mediocre prompt with the right context will succeed.
What Is the Context Window?
The context window is everything the model can "see" when generating a response - system prompt, conversation history, retrieved documents, tool call results, examples, and the current user message. It is the model's working memory: finite, sequential, and the only information it has access to.
Andrej Karpathy's framing: the LLM is the CPU; the context window is RAM. Everything the model knows for this request must fit in RAM. Stuff on disk (training data) can inform general skills but cannot be directly accessed.
The Context Stack
A well-designed context window has layers, each with a distinct purpose:
System Prompt
Role, constraints, output format, tone, and business rules. Changes rarely. The foundation that everything else builds on.
Few-Shot Examples
Representative input/output pairs that demonstrate the expected behaviour. More effective than describing behaviour in prose.
Retrieved Documents
External knowledge fetched at query time via RAG. The most powerful lever for grounding responses in accurate, current information.
Conversation History
Prior turns. Grows with each exchange and consumes budget fast. Summarisation or selective retention is usually necessary in long conversations.
Tool Results & State
Outputs from API calls, code execution, or database queries that the agent has performed in this session. Often verbose - needs pruning.
Current User Message
What the user actually asked. Should be last - recency bias means the model pays most attention to recent tokens.
Core Principles
Only include relevant information
Noise in the context degrades signal. Irrelevant documents retrieved by RAG can mislead the model as much as no retrieval. Filter aggressively.
Position matters
Models pay more attention to the beginning and end of the context. Place critical instructions at the start (system prompt) and the immediate task at the end.
Compress proactively
Long contexts cost more and degrade recall. Summarise conversation history, trim tool results to relevant excerpts, and remove stale information.
Measure token budget
Know how many tokens each context layer consumes. Set hard limits on history and retrieved docs so you never exceed the model's context window.
Context Engineering vs Prompt Engineering
Prompt engineering asks "how should I phrase this?" - context engineering asks "what should the model know when it answers?". The latter has a larger ceiling. A mediocre question with a perfectly curated context will outperform an expert-crafted question with irrelevant or missing context. Both matter, but context is the higher-leverage investment for production systems.