[SL_CORE]
< CD ..||5 min read

Model Routing in Production: Cut LLM Costs by 30%

# Model routing sends easy queries to cheap models and hard ones to powerful models. This single pattern cuts LLM API costs by 20–40% with no quality loss. Here's how to implement it.

[llm][cost-optimization][architecture][production]

Model routing is the practice of automatically selecting the cheapest model that can handle a given request. A classification layer decides whether to use a $0.25/M token model or a $15/M token model — and routes accordingly. Teams that implement this see 20–40% cost reduction with no user-visible quality degradation.

Why Routing Works

Not all queries need Claude Opus or GPT-5. A question like "what's 15% of 80?" doesn't need a 200B-parameter model. A question like "analyze these 10 conflicting legal clauses and identify the risk exposure" absolutely does.

The problem is that most applications send everything to the most capable (and expensive) model by default. Routing fixes this.

Request
   │
   ▼
┌─────────────────┐
│  Difficulty      │──── Simple  ──▶  Haiku / GPT-4o-mini  ($0.25/M)
│  Classifier      │──── Medium  ──▶  Sonnet / GPT-4o      ($3/M)
└─────────────────┘──── Complex ──▶  Opus / GPT-5          ($15/M)

Building a Difficulty Classifier

The classifier itself should be fast and cheap — ideally a local model or a simple heuristic.

Approach 1: Heuristic routing (zero latency, zero cost)

type Tier = "fast" | "standard" | "powerful";

function classifyQuery(prompt: string, context: string[]): Tier {
  const tokens = estimateTokens(prompt + context.join(""));
  const hasCode = /```|function |class |import /.test(prompt);
  const hasAnalysis = /analyz|compar|evaluat|explain why/.test(prompt.toLowerCase());
  const hasLongContext = tokens > 8000;

  if (hasLongContext || (hasCode && hasAnalysis)) return "powerful";
  if (hasCode || hasAnalysis || tokens > 2000) return "standard";
  return "fast";
}

Approach 2: LLM-based classifier (5–20ms, ~$0.001 per classification)

const CLASSIFIER_SYSTEM = `Classify user queries into one of three tiers:
- "fast": simple factual Q&A, basic formatting, short responses, math
- "standard": code generation, summaries, multi-step reasoning
- "powerful": complex analysis, long context, legal/medical/technical depth

Reply with JSON: {"tier": "fast"|"standard"|"powerful", "reason": "brief"}`;

async function classifyWithLLM(query: string): Promise<Tier> {
  const res = await anthropic.messages.create({
    model: "claude-haiku-4-5", // use cheapest model for classification
    max_tokens: 50,
    system: CLASSIFIER_SYSTEM,
    messages: [{ role: "user", content: query }],
  });
  const parsed = JSON.parse(res.content[0].text);
  return parsed.tier;
}

The Router Implementation

const MODEL_MAP: Record<Tier, string> = {
  fast: "claude-haiku-4-5-20251001",
  standard: "claude-sonnet-4-6",
  powerful: "claude-opus-4-8",
};

const PRICE_PER_MILLION: Record<Tier, { input: number; output: number }> = {
  fast:     { input: 0.25,  output: 1.25  },
  standard: { input: 3.00,  output: 15.00 },
  powerful: { input: 15.00, output: 75.00 },
};

async function routedCompletion(
  messages: MessageParam[],
  options: { forcetier?: Tier } = {}
) {
  const lastUser = messages.findLast((m) => m.role === "user");
  const tier = options.forcerier ?? await classifyQuery(String(lastUser?.content));

  const model = MODEL_MAP[tier];
  const response = await anthropic.messages.create({ model, messages, max_tokens: 4096 });

  // Log for cost tracking
  console.log(JSON.stringify({
    tier,
    model,
    input_tokens: response.usage.input_tokens,
    output_tokens: response.usage.output_tokens,
    cost_usd: (
      response.usage.input_tokens * PRICE_PER_MILLION[tier].input / 1_000_000 +
      response.usage.output_tokens * PRICE_PER_MILLION[tier].output / 1_000_000
    ).toFixed(6),
  }));

  return response;
}

Fallback Chain

Routing alone isn't enough — models fail, rate-limit, or return wrong-format responses. Add a fallback chain:

const FALLBACK_CHAIN: string[] = [
  "claude-sonnet-4-6",
  "claude-haiku-4-5-20251001",
  "gpt-4o-mini",  // cross-provider fallback
];

async function withFallback(messages: MessageParam[], primaryModel: string) {
  const chain = [primaryModel, ...FALLBACK_CHAIN.filter((m) => m !== primaryModel)];

  for (const model of chain) {
    try {
      return await anthropic.messages.create({ model, messages, max_tokens: 4096 });
    } catch (err: any) {
      if (err.status === 429 || err.status === 529) {
        console.warn(`Model ${model} rate limited, trying next`);
        continue;
      }
      throw err; // non-retriable error
    }
  }
  throw new Error("All models in fallback chain exhausted");
}

Real Cost Comparison

Assume a chat app handling 100,000 requests/day, average 500 input tokens + 200 output tokens:

| Strategy | Cost/day | Cost/month | |----------|----------|------------| | Always Opus | $900 | $27,000 | | Always Sonnet | $180 | $5,400 | | Routing (70% fast, 20% standard, 10% powerful) | $52 | $1,560 |

The routing distribution above is typical for a customer support or coding assistant — most queries are simple, a few need heavy lifting.

Measuring Routing Quality

Track two metrics:

  1. Upgrade rate — how often users explicitly request a better response. High upgrade rate means the classifier is too aggressive.
  2. Quality score — sample 1% of routed responses, score with a stronger model as judge.
async function scoreResponse(query: string, response: string): Promise<number> {
  const judge = await anthropic.messages.create({
    model: "claude-opus-4-8",
    max_tokens: 10,
    messages: [{
      role: "user",
      content: `Rate this response quality 1-10. Query: "${query}" Response: "${response}". Reply with only a number.`
    }],
  });
  return parseInt(judge.content[0].text.trim());
}

If scores drop below your threshold on any tier, widen that tier's routing criteria.

What We Use at Shahriar Labs

We run model routing across all Shahriar Labs products. The classifier is heuristic-first (free) with an LLM fallback for ambiguous queries. We've found 68% of queries route to Haiku, 25% to Sonnet, and 7% to Opus.

For the LetX editor, code formatting and syntax highlighting routes to Haiku; document understanding and cross-reference analysis routes to Opus. Cost dropped 34% month-over-month after rollout.

See also: Prompt Caching Mastery — combine routing with caching to compound the savings.

FAQ

Does routing add latency? Heuristic routing adds 0ms. LLM-based classification adds 50–100ms. Use heuristic routing for latency-sensitive paths.

What if the classifier routes wrong? Add a "retry with upgrade" button in your UI. Log misroutes and use them to tune thresholds. A 5% misroute rate is acceptable — the cost savings more than compensate.

Should I route within one provider or across providers? Start within one provider (Anthropic or OpenAI) to simplify billing and auth. Add cross-provider routing only after you hit rate limits or need provider-specific features.

Can I route based on user tier? Yes — free users always get fast tier; paying users get standard by default with option to request powerful. This is a common SaaS monetization pattern.

How do I handle tools/function calling across tiers? Not all models support the same tool schemas. Maintain a capability map per model and strip unsupported features before routing down.