Prompt Caching Mastery: 90% Cost Reduction in Practice

Prompt caching is the highest-leverage cost optimization available for LLM workloads with repeated context. When you send the same system prompt or document on every request, you're paying to process it every time. Caching processes it once and reuses the result — cutting input token costs by up to 90% on the cached portion.

How Prompt Caching Works

Anthropic's prompt caching stores processed KV (key-value) activations from transformer layers. When the same prefix is sent again, the model skips recomputation and loads from cache.

Request 1: [System (2000 tokens)] + [User message]
  → Process all 2000 tokens + user message
  → Cache the system prefix (TTL: 5 min)
  → Cost: full input price

Request 2: [System (2000 tokens)] + [Different user message]
  → Load system prefix from cache ✓
  → Process only user message
  → Cost: 10% of normal input price (90% discount)

Cache hits are billed at 10% of the base input token price on Claude. Cache writes (first request) cost 25% more than normal.

Marking Content for Caching

You must explicitly mark cache breakpoints with cache_control:

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

// Large document loaded once, cached across requests
const largeDocument = await fs.readFile("knowledge-base.txt", "utf8");

async function queryDocument(question: string) {
  return client.messages.create({
    model: "claude-sonnet-4-6",
    max_tokens: 1024,
    system: [
      {
        type: "text",
        text: "You are a helpful assistant. Answer questions based on the provided document.",
      },
      {
        type: "text",
        text: largeDocument,
        cache_control: { type: "ephemeral" }, // mark for caching
      },
    ],
    messages: [{ role: "user", content: question }],
  });
}

The cache breakpoint must be at a stable prefix — content before the marker is cached, content after is not. Always put dynamic content (user messages, timestamps) after the last cache_control marker.

Cache Architecture Patterns

Pattern 1: System Prompt Caching (most common)

const SYSTEM = `You are an AI assistant for Shahriar Labs...
[2000 tokens of instructions and context]`;

// All requests reuse this cache after the first
const response = await client.messages.create({
  model: "claude-sonnet-4-6",
  system: [{ type: "text", text: SYSTEM, cache_control: { type: "ephemeral" } }],
  messages: [{ role: "user", content: userQuery }],
  max_tokens: 512,
});

console.log("Cache read tokens:", response.usage.cache_read_input_tokens);
console.log("Cache write tokens:", response.usage.cache_write_input_tokens);

Pattern 2: Document RAG Caching

When doing RAG over a fixed document set (e.g., a product manual), cache the documents:

async function ragWithCache(documents: string[], query: string) {
  const docBlocks = documents.map((doc, i) => ({
    type: "text" as const,
    text: doc,
    // Only cache the last block — everything before it is also cached
    ...(i === documents.length - 1 ? { cache_control: { type: "ephemeral" as const } } : {}),
  }));

  return client.messages.create({
    model: "claude-sonnet-4-6",
    system: [
      { type: "text", text: "Answer based on these documents:" },
      ...docBlocks,
    ],
    messages: [{ role: "user", content: query }],
    max_tokens: 512,
  });
}

Pattern 3: Multi-Turn Conversation Caching

Cache the growing conversation history:

type Message = { role: "user" | "assistant"; content: string };

async function chatWithCache(history: Message[], newMessage: string) {
  // Mark the last assistant message in history as the cache point
  const cachedHistory = history.map((msg, i) => ({
    role: msg.role,
    content: i === history.length - 1 && msg.role === "assistant"
      ? [{ type: "text" as const, text: msg.content, cache_control: { type: "ephemeral" as const } }]
      : msg.content,
  }));

  return client.messages.create({
    model: "claude-sonnet-4-6",
    messages: [
      ...cachedHistory,
      { role: "user", content: newMessage },
    ],
    max_tokens: 1024,
  });
}

Cache TTL and Invalidation

The cache TTL for ephemeral is 5 minutes. Content must be identical and ordered the same to hit the cache. Common cache misses:

| Cause | Fix | |-------|-----| | Timestamp in system prompt | Move timestamp to user message | | Dynamic user context before cache marker | Reorder — static content first | | Different model version | Cache is per model, pin model string | | Token count changed | Even adding a space breaks the cache |

Monitoring cache performance:

function logCacheMetrics(usage: Anthropic.Usage) {
  const cacheHitTokens = usage.cache_read_input_tokens ?? 0;
  const cacheMissTokens = usage.cache_write_input_tokens ?? 0;
  const regularTokens = usage.input_tokens;

  const hitRate = cacheHitTokens / (cacheHitTokens + cacheMissTokens + regularTokens);

  console.log({
    cache_hit_tokens: cacheHitTokens,
    cache_miss_tokens: cacheMissTokens,
    hit_rate: `${(hitRate * 100).toFixed(1)}%`,
    // Savings vs not caching: cache_hit_tokens * 0.9 * price_per_token
  });
}

Real Savings Calculation

For a coding assistant with a 4,000-token system prompt + 500 token average user query, handling 10,000 requests/day on Claude Sonnet ($3/M input):

| Scenario | Daily input cost | |----------|----------------| | No caching | 10,000 × 4,500 tokens × $3/M = $135/day | | With caching (90% hit rate) | 10,000 × (500 + 400 cached) × $3/M = $27/day | | Savings | $108/day = $3,240/month |

The 400 cached tokens = 4,000 × 10% (cache read price). Cache write cost (25% premium, 10% misses) is negligible at scale.

Stacking With Model Routing

Prompt caching compounds with model routing. Cache the system prompt on Haiku, route 70% of queries there, and the savings multiply:

Routing alone: 34% reduction
Caching alone: 60% reduction
Combined: 75–85% reduction

This is the stack we use at Shahriar Labs for all high-volume inference workloads.

FAQ

Does caching work with streaming? Yes. Add cache_control markers as normal; streaming and caching are independent.

Can I cache tool definitions? Yes. Tool definitions placed before the last cache_control marker are cached. This helps when you have many tools.

What's the minimum content worth caching? Anthropic requires a minimum of 1,024 tokens to create a cache entry. Shorter content isn't cacheable.

Does caching affect response quality? No. The model's output is identical whether context is cached or recomputed — caching only skips the forward-pass computation, not the attention to cached content.

How do I know if caching is working? Check usage.cache_read_input_tokens in the response. A non-zero value means you got a cache hit.