Semantic Caching Deep Dive: 73% Cost Reduction Patterns

Standard caching only hits on exact matches. Semantic caching goes further: if a user asks "What's the capital of France?" and the cache has "What is France's capital city?", they're semantically similar enough to return the cached response. On high-repetition workloads — customer support, FAQ bots, documentation Q&A — this cuts costs by 40–73%.

How Semantic Caching Works

Incoming query
      │
      ▼
Embed query → vector
      │
      ▼
Search cache by cosine similarity
      │
    ≥ 0.95?
    /       \
  YES         NO
   │           │
Return       Call LLM
cached        │
response   Cache result
               │
           Return response

The similarity threshold (0.95) is the critical tuning parameter — too low and you return wrong cached responses; too high and you miss valid cache hits.

Implementation with Qdrant + Redis

import { QdrantClient } from "@qdrant/js-client-rest";
import { createClient } from "redis";
import OpenAI from "openai";

interface CacheEntry {
  queryEmbedding: number[];
  query: string;
  response: string;
  model: string;
  cachedAt: number;
  hitCount: number;
}

class SemanticCache {
  private qdrant: QdrantClient;
  private redis: ReturnType<typeof createClient>;
  private openai: OpenAI;
  private readonly COLLECTION = "semantic-cache";
  private readonly SIMILARITY_THRESHOLD = 0.95;
  private readonly TTL_SECONDS = 3600; // 1 hour

  constructor() {
    this.qdrant = new QdrantClient({ url: "http://localhost:6333" });
    this.redis = createClient({ url: process.env.REDIS_URL });
    this.openai = new OpenAI();
  }

  async init() {
    await this.qdrant.createCollection(this.COLLECTION, {
      vectors: { size: 1536, distance: "Cosine" },
    });
    await this.redis.connect();
  }

  async get(query: string): Promise<string | null> {
    const embedding = await this.embed(query);

    const results = await this.qdrant.search(this.COLLECTION, {
      vector: embedding,
      limit: 1,
      score_threshold: this.SIMILARITY_THRESHOLD,
      with_payload: true,
    });

    if (results.length === 0) return null;

    const cacheKey = results[0].id as string;

    // Update hit count
    const entry = JSON.parse(await this.redis.get(cacheKey) ?? "null") as CacheEntry | null;
    if (!entry) {
      await this.qdrant.delete(this.COLLECTION, { points: [cacheKey] });
      return null;
    }

    entry.hitCount++;
    await this.redis.setEx(cacheKey, this.TTL_SECONDS, JSON.stringify(entry));

    return entry.response;
  }

  async set(query: string, response: string, model: string): Promise<void> {
    const embedding = await this.embed(query);
    const id = crypto.randomUUID();

    const entry: CacheEntry = {
      queryEmbedding: embedding,
      query,
      response,
      model,
      cachedAt: Date.now(),
      hitCount: 0,
    };

    // Store vector in Qdrant for similarity search
    await this.qdrant.upsert(this.COLLECTION, {
      points: [{ id, vector: embedding, payload: { query, cachedAt: entry.cachedAt } }],
    });

    // Store full response in Redis (TTL-managed)
    await this.redis.setEx(id, this.TTL_SECONDS, JSON.stringify(entry));
  }

  private async embed(text: string): Promise<number[]> {
    const response = await this.openai.embeddings.create({
      model: "text-embedding-3-small",
      input: text,
    });
    return response.data[0].embedding;
  }
}

Wrapping LLM Calls with Semantic Cache

class CachedLLMClient {
  private cache: SemanticCache;

  async complete(
    query: string,
    systemPrompt: string,
    model = "claude-sonnet-4-6"
  ): Promise<{ response: string; cacheHit: boolean }> {
    // Create cache key including system prompt hash
    const cacheKey = `${hashString(systemPrompt)}:${query}`;

    // Check cache
    const cached = await this.cache.get(cacheKey);
    if (cached) {
      return { response: cached, cacheHit: true };
    }

    // Call LLM
    const llmResponse = await anthropic.messages.create({
      model,
      system: systemPrompt,
      messages: [{ role: "user", content: query }],
      max_tokens: 1024,
    });
    const response = llmResponse.content[0].text;

    // Cache the result
    await this.cache.set(cacheKey, response, model);

    return { response, cacheHit: false };
  }
}

Tuning the Similarity Threshold

This is the most important parameter. Wrong values cause two failure modes:

| Threshold | Failure mode | Example | |-----------|-------------|---------| | Too low (≤0.90) | Wrong cache hits | "How do I reset my password?" returns response for "How do I change my email?" | | Too high (≥0.99) | Few cache hits | "What's the capital of France?" and "What is France's capital?" don't match |

// Test your threshold with labeled pairs
const TEST_PAIRS: Array<{ query1: string; query2: string; shouldMatch: boolean }> = [
  { query1: "What's the capital of France?", query2: "What is France's capital city?", shouldMatch: true },
  { query1: "How do I reset my password?", query2: "How do I change my email?", shouldMatch: false },
  { query1: "List Python list methods", query2: "What are Python array functions?", shouldMatch: true },
  { query1: "Error: connection timeout", query2: "Error: authentication failed", shouldMatch: false },
];

async function calibrateThreshold(): Promise<number> {
  const embeddings = await Promise.all(
    TEST_PAIRS.flatMap((p) => [p.query1, p.query2]).map(embed)
  );

  let bestThreshold = 0.95;
  let bestF1 = 0;

  for (const threshold of [0.88, 0.90, 0.92, 0.94, 0.95, 0.96, 0.97, 0.98]) {
    let tp = 0, fp = 0, fn = 0;

    for (let i = 0; i < TEST_PAIRS.length; i++) {
      const sim = cosineSimilarity(embeddings[i * 2], embeddings[i * 2 + 1]);
      const predicted = sim >= threshold;
      const actual = TEST_PAIRS[i].shouldMatch;

      if (predicted && actual) tp++;
      else if (predicted && !actual) fp++;
      else if (!predicted && actual) fn++;
    }

    const precision = tp / (tp + fp);
    const recall = tp / (tp + fn);
    const f1 = 2 * (precision * recall) / (precision + recall);

    if (f1 > bestF1) { bestF1 = f1; bestThreshold = threshold; }
  }

  return bestThreshold;
}

Cache Invalidation

// Invalidate entries when underlying data changes
async function invalidateCache(pattern: string) {
  // Find entries whose query matches the pattern
  const patternEmbedding = await embed(pattern);
  const similarEntries = await qdrant.search("semantic-cache", {
    vector: patternEmbedding,
    limit: 50,
    score_threshold: 0.85, // lower threshold for invalidation
  });

  // Delete from both stores
  await qdrant.delete("semantic-cache", { points: similarEntries.map((e) => e.id as string) });
  await Promise.all(similarEntries.map((e) => redis.del(e.id as string)));

  console.log(`Invalidated ${similarEntries.length} cache entries matching "${pattern}"`);
}

// Use when knowledge base is updated:
await invalidateCache("pricing"); // invalidate all price-related queries
await invalidateCache("API documentation"); // invalidate all API-related queries

Cost Savings Measurement

interface CacheMetrics {
  totalRequests: number;
  cacheHits: number;
  hitRate: number;
  tokensSaved: number;
  costSavedUSD: number;
}

async function getCacheMetrics(window: "1h" | "24h" | "7d"): Promise<CacheMetrics> {
  const { total, hits, tokens_saved } = await db.queryOne<any>(`
    SELECT
      COUNT(*) as total,
      COUNT(*) FILTER (WHERE cache_hit = true) as hits,
      SUM(estimated_tokens) FILTER (WHERE cache_hit = true) as tokens_saved
    FROM llm_requests
    WHERE created_at > NOW() - INTERVAL '${window}'
  `);

  const hitRate = hits / total;
  const costSaved = tokens_saved * 0.003 / 1000; // Sonnet input price

  return { totalRequests: total, cacheHits: hits, hitRate, tokensSaved: tokens_saved, costSavedUSD: costSaved };
}

When Semantic Caching Helps (and Hurts)

| Workload | Cache benefit | |----------|--------------| | FAQ bot, docs Q&A | ★★★★★ (>50% hit rate) | | Customer support | ★★★★☆ (30–50% hit rate) | | Code generation | ★★☆☆☆ (<20% — queries too unique) | | Creative writing | ★☆☆☆☆ (wrong to cache) | | Conversational chat | ★★★☆☆ (varies) |

Never cache outputs that must be personalized (financial advice, medical) or unique per-user (personal recommendations).

FAQ

What's the right TTL for cached responses? For static content (product descriptions): 24 hours. For dynamic content (support questions): 1–4 hours. For real-time data queries: don't cache. When in doubt, shorter TTL is safer.

Does semantic caching work across different models? Cache by model — a Haiku response and a Sonnet response to the same question are different quality. Include the model name in the cache key.

What about multilingual queries? Multilingual embedding models (e.g., text-embedding-3-small) handle this — "Quelle est la capitale de la France?" and "What's the capital of France?" may share a cached response if similarity is high enough. Test with your target languages.

Can I combine semantic caching with prompt caching? Yes — they work at different levels. Prompt caching (Anthropic) is server-side and caches the prefix computation. Semantic caching is application-level and skips the LLM call entirely. Combine both: Prompt Caching Mastery.

How much memory does the embedding index need? Each 1536-dimension float32 vector = 6KB. 100K cached queries ≈ 600MB for vectors (in Qdrant). Redis stores the full responses — plan for 1–5KB per entry. 100K entries ≈ 100–500MB Redis.