[SL_CORE]
< CD ..||6 min read

Token Budget Optimization: Allocating Compute Wisely

# Token budgets control how much reasoning an LLM does before answering. Learn to allocate thinking tokens strategically: when extended reasoning pays off, when it wastes money, and how to measure the ROI.

[llm][cost-optimization][reasoning][production]

Token budgets became a first-class concept with Claude's extended thinking and OpenAI's o-series models. You explicitly allocate thinking tokens — separate from input/output tokens — that the model uses for internal reasoning. Allocate too few and quality suffers. Allocate too many and you waste money. Here's how to find the sweet spot.

What Token Budgets Control

Extended thinking models use a two-phase process:

  1. Thinking tokens — internal reasoning, not shown to users, typically 3× cheaper than output tokens
  2. Output tokens — the final response the user sees
User prompt (input tokens)
    │
    ▼
[Thinking phase — 1,000–100,000 thinking tokens]
    │ Internal, not shown to user
    ▼
Final response (output tokens)

More thinking tokens = more thorough reasoning = better answers for complex problems. Zero thinking tokens = faster, cheaper, worse on hard problems.

When Extended Thinking Pays Off

Run an ROI comparison on your specific tasks:

const THINKING_BENCHMARKS = [
  { task: "math-proof", budgets: [0, 1000, 5000, 10000], complexity: "high" },
  { task: "code-review", budgets: [0, 500, 2000, 5000], complexity: "medium" },
  { task: "text-summary", budgets: [0, 200, 500], complexity: "low" },
  { task: "legal-analysis", budgets: [0, 5000, 20000, 50000], complexity: "high" },
];

async function benchmarkThinkingBudget(
  task: string,
  budgets: number[],
  evalExamples: EvalExample[]
): Promise<ThinkingROI[]> {
  return Promise.all(budgets.map(async (budget) => {
    const results = await runEvaluations(task, evalExamples, budget);
    const cost = calculateThinkingCost(results);
    
    return {
      budget,
      qualityScore: results.avgScore,
      costPerTask: cost,
      costPerQualityPoint: cost / results.avgScore,
      worthIt: results.avgScore > results.baselineScore * 1.05, // 5% improvement threshold
    };
  }));
}

async function runEvaluations(task: string, examples: EvalExample[], thinkingBudget: number) {
  const scores: number[] = [];

  for (const example of examples) {
    const response = await anthropic.messages.create({
      model: "claude-opus-4-8",
      max_tokens: 8000,
      thinking: thinkingBudget > 0
        ? { type: "enabled", budget_tokens: thinkingBudget }
        : undefined,
      messages: [{ role: "user", content: example.input }],
    });

    const output = response.content.filter((b) => b.type === "text").map((b) => b.text).join("");
    const { score } = await llmJudge(example.input, output, example.criteria);
    scores.push(score);
  }

  return {
    avgScore: scores.reduce((a, b) => a + b, 0) / scores.length,
    baselineScore: 0, // set from budget=0 run
  };
}

Task Complexity Matrix

Based on empirical benchmarks across task types:

| Task type | Recommended thinking budget | Quality improvement | |-----------|---------------------------|---------------------| | Simple Q&A, summarization | 0 | <2% improvement, not worth it | | Code generation (<50 lines) | 0–500 | Marginal (3–5%) | | Code generation (complex) | 2,000–5,000 | Significant (10–20%) | | Mathematical proofs | 5,000–20,000 | Major (20–40%) | | Legal/contract analysis | 10,000–50,000 | Major (15–30%) | | Security vulnerability analysis | 5,000–20,000 | Major (25–35%) | | Creative writing | 500–2,000 | Modest (5–10%) |

Dynamic Budget Allocation

Route different task difficulties to different thinking budgets:

type TaskDifficulty = "trivial" | "simple" | "moderate" | "complex" | "expert";

async function classifyDifficulty(task: string): Promise<TaskDifficulty> {
  const response = await anthropic.messages.create({
    model: "claude-haiku-4-5-20251001",
    max_tokens: 20,
    messages: [{
      role: "user",
      content: `Classify this task difficulty: "${task.slice(0, 200)}"
trivial=basic Q&A/formatting, simple=single-step reasoning, moderate=multi-step, complex=expert-level analysis, expert=frontier research
Reply with just one word.`,
    }],
  });
  return response.content[0].text.trim().toLowerCase() as TaskDifficulty;
}

const BUDGET_BY_DIFFICULTY: Record<TaskDifficulty, number> = {
  trivial: 0,
  simple: 0,
  moderate: 1_000,
  complex: 8_000,
  expert: 32_000,
};

async function adaptiveThinkingCall(
  task: string,
  forceDifficulty?: TaskDifficulty
): Promise<{ response: string; thinkingTokens: number; cost: number }> {
  const difficulty = forceDifficulty ?? await classifyDifficulty(task);
  const budget = BUDGET_BY_DIFFICULTY[difficulty];

  const response = await anthropic.messages.create({
    model: "claude-opus-4-8",
    max_tokens: 4096,
    ...(budget > 0 ? { thinking: { type: "enabled", budget_tokens: budget } } : {}),
    messages: [{ role: "user", content: task }],
  });

  const thinkingTokens = response.usage.cache_read_input_tokens ?? 0; // thinking tokens in usage
  const textOutput = response.content.filter((b) => b.type === "text").map((b) => b.text).join("");

  return {
    response: textOutput,
    thinkingTokens,
    cost: calculateCostWithThinking("claude-opus-4-8", response.usage, thinkingTokens),
  };
}

Cost Calculation with Thinking

Thinking tokens are billed differently:

function calculateCostWithThinking(
  model: string,
  usage: Anthropic.Usage,
  thinkingTokens: number
): number {
  // Claude Opus pricing (per million tokens)
  const INPUT_PRICE = 15.00;
  const OUTPUT_PRICE = 75.00;
  const THINKING_PRICE = 3.00; // ~80% cheaper than output

  const inputCost = usage.input_tokens * INPUT_PRICE / 1_000_000;
  const outputCost = usage.output_tokens * OUTPUT_PRICE / 1_000_000;
  const thinkingCost = thinkingTokens * THINKING_PRICE / 1_000_000;

  return inputCost + outputCost + thinkingCost;
}

Thinking Token Budget vs. Context Window

Both consume the same context window. Manage them together:

function planTokenAllocation(
  systemPromptTokens: number,
  userInputTokens: number,
  maxContextWindow = 200_000
): {
  thinkingBudget: number;
  outputBudget: number;
  available: boolean;
} {
  const overhead = 500; // buffer
  const used = systemPromptTokens + userInputTokens + overhead;
  const remaining = maxContextWindow - used;

  if (remaining < 4096) {
    return { thinkingBudget: 0, outputBudget: 2048, available: false };
  }

  // Allocate: 70% thinking, 30% output (for complex tasks)
  const thinkingBudget = Math.min(Math.floor(remaining * 0.7), 50_000);
  const outputBudget = Math.min(Math.floor(remaining * 0.3), 8_192);

  return { thinkingBudget, outputBudget, available: true };
}

Monitoring Thinking Efficiency

Track whether thinking tokens actually improve outcomes:

interface ThinkingMetrics {
  taskId: string;
  thinkingTokensUsed: number;
  outputTokens: number;
  qualityScore: number;
  costUSD: number;
  costPerQualityPoint: number;
}

// Weekly analysis: is thinking budget allocation optimal?
async function analyzeThinkingROI(lookbackDays = 7): Promise<ThinkingROIReport> {
  const metrics = await db.query<ThinkingMetrics>(`
    SELECT thinking_tokens, output_tokens, quality_score, cost_usd
    FROM task_runs
    WHERE created_at > NOW() - $1 * INTERVAL '1 day'
      AND thinking_tokens > 0
  `, [lookbackDays]);

  // Group by thinking budget tier
  const tiers = [
    { label: "low", min: 0, max: 2000 },
    { label: "medium", min: 2000, max: 10000 },
    { label: "high", min: 10000, max: Infinity },
  ];

  return tiers.map((tier) => {
    const tierMetrics = metrics.filter(
      (m) => m.thinkingTokensUsed >= tier.min && m.thinkingTokensUsed < tier.max
    );
    return {
      tier: tier.label,
      avgQuality: avg(tierMetrics.map((m) => m.qualityScore)),
      avgCost: avg(tierMetrics.map((m) => m.costUSD)),
      count: tierMetrics.length,
    };
  });
}

FAQ

Does extended thinking work with all Claude models? Extended thinking (thinking: { type: "enabled", budget_tokens: N }) is available on Claude Opus models (claude-opus-4-8 and newer). Sonnet and Haiku use standard generation without separate thinking.

What's the minimum useful thinking budget? Below 1,000 tokens, thinking rarely helps. The model needs enough budget to actually reason through a problem. Start at 1,000 for moderate complexity, 5,000 for complex tasks.

Is 50,000 thinking tokens ever worth it? For frontier-level tasks (complex proofs, multi-step legal analysis, novel security vulnerabilities): yes. At $0.003/K thinking tokens and 50K budget = $0.15 additional cost. If the task is worth $10+, it's worth it.

Can I see what the model is thinking? Yes — thinking blocks are returned in the response (type "thinking"). Log them for debugging but don't show them to users (they're often exploratory and incomplete-looking).

How do thinking tokens interact with prompt caching? Thinking tokens aren't cacheable — they're generated fresh each call. The system prompt and user input can still be cached. See Prompt Caching Mastery for combining both.