[SL_CORE]
< CD ..||6 min read

Context Window Management: Strategies for 200K+ Tokens

# Large context windows don't mean you should fill them. Learn the strategies for managing context effectively: selective inclusion, compression, sliding windows, and retrieval-based context building.

[llm][context-window][architecture][performance]

Claude Sonnet supports 200K tokens; Gemini 2.5 Flash goes to 1M. Having a large context window doesn't mean you should fill it. Model performance degrades with noisy context, costs scale linearly with tokens, and the "lost in the middle" problem is real. Here's how to manage context strategically.

The Lost in the Middle Problem

Research confirms: when relevant information is in the middle of a long context, models perform significantly worse than when it's at the beginning or end.

Performance by position in context:
Beginning  ████████████████  90%
End        ██████████████    85%
Middle     █████████         65%

Practical implication: if you're building a RAG system, put the most relevant chunks at the start of the context, not buried in the middle.

Context Budget Allocation

Treat context like a budget. Allocate it explicitly:

interface ContextBudget {
  systemPrompt: number;       // 1,000–5,000 tokens: instructions
  longTermMemory: number;     // 500–2,000: retrieved memories
  retrievedDocs: number;      // 2,000–10,000: RAG chunks
  conversationHistory: number; // 2,000–20,000: recent turns
  currentTask: number;        // 500–5,000: current user input
  outputBuffer: number;       // 1,000–4,000: reserved for response
}

const BUDGET_200K: ContextBudget = {
  systemPrompt: 3_000,
  longTermMemory: 2_000,
  retrievedDocs: 8_000,
  conversationHistory: 20_000,
  currentTask: 5_000,
  outputBuffer: 4_000,
}; // total: ~42,000 of 200,000 available

// Don't use all 200K just because you have it
// Use it for: truly long documents, complex codebases, book analysis

Strategy 1: Selective Context Inclusion

Don't dump everything — include only what's relevant to the current turn:

async function buildOptimalContext(
  query: string,
  conversation: Message[],
  availableChunks: string[]
): Promise<string[]> {
  // 1. Always include: recent conversation (last 5 turns)
  const recentHistory = conversation.slice(-10);

  // 2. Selectively include: relevant chunks only
  const relevantChunks = await rankChunksByRelevance(query, availableChunks, topK = 5);

  // 3. Conditionally include: older history if referenced
  const olderHistory = await getReferencedHistory(query, conversation.slice(0, -10));

  return buildOrderedContext({
    priority1: relevantChunks.slice(0, 2), // most relevant first
    priority2: recentHistory,
    priority3: relevantChunks.slice(2),
    priority4: olderHistory,
  });
}

async function rankChunksByRelevance(
  query: string,
  chunks: string[],
  topK: number
): Promise<string[]> {
  const [queryEmbedding] = await embedTexts([query]);
  const chunkEmbeddings = await embedTexts(chunks);

  return chunks
    .map((chunk, i) => ({
      chunk,
      score: cosineSimilarity(queryEmbedding, chunkEmbeddings[i]),
    }))
    .sort((a, b) => b.score - a.score)
    .slice(0, topK)
    .map(({ chunk }) => chunk);
}

Strategy 2: Conversation Summarization

For long-running conversations, compress history progressively:

interface ConversationManager {
  recent: Message[];      // last 10 turns — kept verbatim
  summary: string;        // older turns — compressed
  keyFacts: string[];     // extracted facts — always kept
}

async function compressOldHistory(messages: Message[]): Promise<string> {
  const response = await anthropic.messages.create({
    model: "claude-haiku-4-5-20251001", // cheap model for compression
    max_tokens: 500,
    messages: [{
      role: "user",
      content: `Summarize this conversation for context. Keep: decisions made, key facts, open questions. 
Discard: pleasantries, repetition, resolved clarifications.
Return a 3-5 sentence summary.

Conversation:
${messages.map((m) => `${m.role}: ${m.content}`).join("\n")}`,
    }],
  });
  return response.content[0].text;
}

// In the agent loop: compress when history exceeds threshold
async function manageHistory(
  manager: ConversationManager,
  newMessage: Message
): Promise<ConversationManager> {
  manager.recent.push(newMessage);

  if (manager.recent.length > 10 || estimateTokens(manager.recent) > 8000) {
    const toCompress = manager.recent.slice(0, -5); // keep last 5 verbatim
    const newSummary = await compressOldHistory(toCompress);

    manager.summary = manager.summary
      ? `${manager.summary}\n\n${newSummary}`
      : newSummary;
    manager.recent = manager.recent.slice(-5);
  }

  return manager;
}

function buildContextFromManager(manager: ConversationManager): string {
  const parts = [];
  if (manager.summary) parts.push(`[Earlier conversation summary]: ${manager.summary}`);
  if (manager.keyFacts.length) parts.push(`[Key facts]: ${manager.keyFacts.join("; ")}`);
  parts.push(...manager.recent.map((m) => `${m.role}: ${m.content}`));
  return parts.join("\n\n");
}

Strategy 3: Sliding Window for Long Documents

For processing very long documents (books, large codebases), use a sliding window:

async function processLongDocument(
  document: string,
  query: string,
  windowSize = 50_000, // tokens
  overlap = 5_000
): Promise<string> {
  const chunks = splitByTokens(document, windowSize, overlap);
  const partialAnswers: string[] = [];

  for (const chunk of chunks) {
    const response = await anthropic.messages.create({
      model: "claude-sonnet-4-6",
      max_tokens: 500,
      messages: [{
        role: "user",
        content: `Document section:\n${chunk}\n\nQuery: ${query}\n\nExtract any information relevant to the query. If none, say "No relevant content."`,
      }],
    });
    const answer = response.content[0].text;
    if (!answer.includes("No relevant content")) {
      partialAnswers.push(answer);
    }
  }

  if (partialAnswers.length === 0) return "No relevant information found in document.";

  // Synthesize partial answers
  const synthesis = await anthropic.messages.create({
    model: "claude-sonnet-4-6",
    max_tokens: 1000,
    messages: [{
      role: "user",
      content: `Combine these partial answers into a complete response:\n${partialAnswers.join("\n\n")}`,
    }],
  });
  return synthesis.content[0].text;
}

Strategy 4: Dynamic Context Assembly

Build context dynamically based on what the model actually needs:

async function dynamicContextAssembly(
  query: string,
  availableSources: Record<string, () => Promise<string>>
): Promise<string> {
  // 1. Ask the model what context it needs
  const planResponse = await anthropic.messages.create({
    model: "claude-haiku-4-5-20251001",
    max_tokens: 200,
    messages: [{
      role: "user",
      content: `To answer: "${query}"
Available sources: ${Object.keys(availableSources).join(", ")}
Which sources are needed? Reply JSON: {"needed": ["source1", ...], "reason": "brief"}`,
    }],
  });

  const { needed } = JSON.parse(planResponse.content[0].text);

  // 2. Fetch only needed sources in parallel
  const contextParts = await Promise.all(
    (needed as string[])
      .filter((s) => s in availableSources)
      .map(async (source) => ({
        source,
        content: await availableSources[source](),
      }))
  );

  return contextParts
    .map(({ source, content }) => `[${source}]\n${content}`)
    .join("\n\n");
}

When Large Context IS the Right Tool

Sometimes you should use the full context window:

| Use case | Why large context wins | |----------|----------------------| | Code review of entire repo | Cross-file dependencies need full context | | Contract analysis | Legal documents have cross-references | | Book summarization | Narrative arcs span the whole document | | Debugging complex traces | Full log context needed for root cause |

For these cases, use prompt caching to avoid re-processing the large context on repeated queries. See Prompt Caching Mastery.

FAQ

Does model quality really degrade with more context? Yes — for current transformer architectures, attention becomes more diffuse as context grows. The effect is pronounced when relevant content is <20% of the total context. Relevance filtering is always worth doing.

What's the actual token cost of a 200K context call? Claude Sonnet at $3/M input: 200K tokens = $0.60 per call. With prompt caching (90% hit rate): $0.60 → $0.06 for repeated calls on the same context.

Is it faster to use a 1M context vs. multiple 50K context calls? Single large call: linear cost scaling. Multiple focused calls: parallel execution possible but more API overhead. For batch processing, multiple focused calls often finish faster due to parallelism.

How do I estimate token count before sending? Use tiktoken (OpenAI) or Anthropic's token counting API endpoint. Rule of thumb: ~1 token per 4 English characters, ~1 token per 3 characters for code.

What's the difference between context window and effective context length? Context window = maximum tokens the model can receive. Effective context length = the portion where the model reliably attends to content. For current models, effective length is often 50–80% of the maximum for complex reasoning tasks.