Advanced

Context Compression Techniques

Every token in the context window costs money and latency. As conversations grow and RAG retrieves more documents, the context fills up - and stuffing everything in degrades model performance. Context compression is the set of techniques for keeping what matters and discarding what does not.

Conversation Summarisation

The simplest compression technique: when the conversation history exceeds a threshold (say, 8K tokens), call the model to summarise everything up to that point into 500–1000 tokens, and replace the history with that summary. The summary becomes the new "history" that gets prepended to each subsequent prompt.

// Pseudocode: rolling summary
if (historyTokens > 8000) {
  const summary = await llm.complete({
    system: "Summarise the following conversation. Preserve key decisions, user preferences, and open questions. Be concise.",
    user: conversationHistory.join("\n")
  });
  history = [{ role: "system", content: "Conversation summary: " + summary }];
}

Rolling summaries work well for chat applications. The tradeoff is that the model may lose specific details - mitigate by explicitly listing "facts to preserve" in the summary prompt (e.g. user name, decisions made, current task state).

Selective Retention

Rather than summarising everything, selectively retain specific types of content:

Decisions - what did the user or system decide? Retained verbatim.
Current task state - what step are we on? What is left?
User preferences - expressed choices that affect future outputs.
Errors and corrections - what went wrong and was fixed, to avoid repeating.

Transient content - exploratory ideas the user abandoned, tool call details for completed steps, verbose API responses - can be dropped entirely.

Compressing Retrieved Documents

RAG pipelines often retrieve 3–10 document chunks, each potentially several hundred tokens. Techniques for compressing them before they enter the context:

Abstractive compression

Pass each retrieved chunk to a small model and ask it to extract only the sentences relevant to the query. Can reduce chunk size by 60–80% while keeping the information that matters.

Extractive compression

Score each sentence in the chunk for relevance to the query using a cross-encoder, then keep only the top-scoring sentences. Fast, cheap, but may miss context that only makes sense with surrounding sentences.

Prompt Caching

The largest token efficiency win for repeated requests is prompt caching. When the prefix of a prompt (system prompt + examples + static context) is identical across requests, the model provider caches the KV computation for that prefix and charges a fraction of the normal token price for cache hits.

Provider	Cache write cost	Cache hit cost
Anthropic (Claude)	1.25× normal	0.1× normal (90% saving)
OpenAI	Normal	0.5× normal (50% saving)

Design prompts so that the static prefix (system prompt, examples, any fixed context) comes first and the variable part (user query, conversation history, retrieved docs) comes last. The longer the cacheable prefix, the greater the savings.

Token Budgeting

Enforce a token budget for each context layer before building the final prompt:

Example budget for a 128K model

• System prompt: ≤ 2,000 tokens (fixed)
• Few-shot examples: ≤ 3,000 tokens (cacheable prefix)
• Retrieved documents: ≤ 8,000 tokens (compressed)
• Conversation history / summary: ≤ 4,000 tokens
• Current user message: ≤ 1,000 tokens
• Reserve for model response: ≤ 4,000 tokens
Total: ~22,000 tokens (17% of window - leaves headroom)

Compression Checklist

Summarise conversation history when it exceeds 8K tokens
Extract only query-relevant sentences from retrieved chunks before inserting into context
Place the static portion of the prompt first to maximise prompt cache hits
Set hard token limits per context layer; enforce them before building the final prompt
Drop tool call details for completed steps; retain only outcomes and decisions

Context Compression & Summarisation