Cost Per Task, Not Token: Measuring Real AI Economics

Every LLM pricing page shows dollars per million tokens. This metric is useful for comparing models but terrible for business decisions. A task that takes 1,000 tokens with a 90% success rate costs more per outcome than a task that takes 3,000 tokens with a 99% success rate. Cost per successfully completed task is the metric that matters.

Why Cost Per Token Misleads

Consider two agent configurations for a code review task:

| Config | Tokens used | Token cost | Success rate | Cost per successful review | |--------|------------|-----------|-------------|---------------------------| | Haiku, single-shot | 800 | $0.001 | 60% | $0.0017 | | Sonnet, with verification | 3,200 | $0.010 | 95% | $0.0105 | | Haiku, 3-shot retry | 2,400 | $0.003 | 90% | $0.0033 |

Haiku single-shot looks cheapest per token. But if you need 100 successful reviews, it costs $0.17. The Haiku with retries costs $0.33. Sonnet with verification costs $1.05. For this task, Haiku with retries is the optimal choice — not because of token cost but because of the cost-to-outcome ratio.

Defining "Task Completion"

First, define what success means for each agent type:

type TaskOutcome = "success" | "partial" | "failure" | "timeout";

interface TaskDefinition {
  type: string;
  successCriteria: (output: string) => Promise<TaskOutcome>;
  maxCost: number;   // abort if this is exceeded
  maxRetries: number;
}

const CODE_REVIEW_TASK: TaskDefinition = {
  type: "code_review",
  successCriteria: async (output) => {
    // Check if output contains required sections
    const hasIssues = /##\s*Issues|No issues found/i.test(output);
    const hasSuggestions = /##\s*Suggestions|Looks good/i.test(output);
    const hasSeverity = /critical|high|medium|low/i.test(output);

    if (hasIssues && hasSuggestions && hasSeverity) return "success";
    if (hasIssues || hasSuggestions) return "partial";
    return "failure";
  },
  maxCost: 0.05,
  maxRetries: 2,
};

The Cost Per Task Calculator

interface TaskRun {
  taskType: string;
  outcome: TaskOutcome;
  totalTokensIn: number;
  totalTokensOut: number;
  totalCostUSD: number;
  model: string;
  retryCount: number;
  durationMs: number;
}

class TaskCostTracker {
  private runs: TaskRun[] = [];

  record(run: TaskRun) {
    this.runs.push(run);
  }

  getMetrics(taskType: string, timeWindowMs = 86400000 /* 24h */): TaskMetrics {
    const recent = this.runs.filter(
      (r) => r.taskType === taskType && Date.now() - r.durationMs < timeWindowMs
    );

    const successful = recent.filter((r) => r.outcome === "success");
    const totalCost = recent.reduce((sum, r) => sum + r.totalCostUSD, 0);
    const successRate = successful.length / recent.length;
    const costPerTask = totalCost / recent.length;
    const costPerSuccess = successful.length > 0 ? totalCost / successful.length : Infinity;

    return {
      taskType,
      totalRuns: recent.length,
      successRate,
      costPerTask,
      costPerSuccess,
      avgTokensIn: avg(recent.map((r) => r.totalTokensIn)),
      avgTokensOut: avg(recent.map((r) => r.totalTokensOut)),
      avgDurationMs: avg(recent.map((r) => r.durationMs)),
      avgRetries: avg(recent.map((r) => r.retryCount)),
    };
  }

  compareStrategies(taskType: string): StrategyComparison[] {
    const byModel = groupBy(
      this.runs.filter((r) => r.taskType === taskType),
      (r) => r.model
    );

    return Object.entries(byModel).map(([model, runs]) => {
      const successRate = runs.filter((r) => r.outcome === "success").length / runs.length;
      const totalCost = runs.reduce((sum, r) => sum + r.totalCostUSD, 0);
      return {
        model,
        successRate,
        costPerSuccess: totalCost / (runs.length * successRate),
        sampleSize: runs.length,
      };
    });
  }
}

Measuring in the Agent Loop

Wrap your agent run with cost tracking:

async function trackedAgentRun(
  task: TaskDefinition,
  input: string,
  tracker: TaskCostTracker
): Promise<{ outcome: TaskOutcome; output: string }> {
  let totalCostUSD = 0;
  let totalTokensIn = 0;
  let totalTokensOut = 0;
  let retryCount = 0;
  const startTime = Date.now();

  for (let attempt = 0; attempt <= task.maxRetries; attempt++) {
    if (attempt > 0) retryCount++;

    const model = attempt === 0 ? "claude-haiku-4-5-20251001" : "claude-sonnet-4-6";
    const response = await anthropic.messages.create({
      model,
      max_tokens: 2048,
      messages: [{ role: "user", content: input }],
    });

    const cost = calculateCost(model, response.usage);
    totalCostUSD += cost;
    totalTokensIn += response.usage.input_tokens;
    totalTokensOut += response.usage.output_tokens;

    // Abort if cost exceeds budget
    if (totalCostUSD > task.maxCost) {
      tracker.record({
        taskType: task.type,
        outcome: "failure",
        totalTokensIn,
        totalTokensOut,
        totalCostUSD,
        model,
        retryCount,
        durationMs: Date.now() - startTime,
      });
      return { outcome: "failure", output: "Cost limit exceeded" };
    }

    const output = response.content[0].text;
    const outcome = await task.successCriteria(output);

    if (outcome === "success" || attempt === task.maxRetries) {
      tracker.record({
        taskType: task.type,
        outcome,
        totalTokensIn,
        totalTokensOut,
        totalCostUSD,
        model,
        retryCount,
        durationMs: Date.now() - startTime,
      });
      return { outcome, output };
    }
    // partial/failure → retry with escalated model
  }

  return { outcome: "failure", output: "" };
}

Building a Cost Dashboard

Key metrics to track per task type, per model, per time period:

interface CostDashboard {
  // Business metrics (what matters)
  costPerSuccessfulTask: number;
  dailyCostForSLA: number;   // cost to maintain target success rate
  
  // Operational metrics
  successRate: number;
  p50LatencyMs: number;
  p99LatencyMs: number;
  avgRetriesPerTask: number;
  
  // Financial metrics
  totalSpendToday: number;
  projectedMonthlySpend: number;
  costTrend: "increasing" | "stable" | "decreasing";
}

Log this to your existing observability stack (Datadog, Grafana, or a simple Postgres table):

CREATE TABLE task_runs (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  task_type VARCHAR(100) NOT NULL,
  outcome VARCHAR(20) NOT NULL,
  model VARCHAR(100) NOT NULL,
  tokens_in INTEGER NOT NULL,
  tokens_out INTEGER NOT NULL,
  cost_usd NUMERIC(10, 8) NOT NULL,
  retry_count INTEGER DEFAULT 0,
  duration_ms INTEGER NOT NULL,
  created_at TIMESTAMPTZ DEFAULT NOW()
);

-- Query: cost per successful task by type, last 7 days
SELECT
  task_type,
  COUNT(*) FILTER (WHERE outcome = 'success') AS successes,
  SUM(cost_usd) / NULLIF(COUNT(*) FILTER (WHERE outcome = 'success'), 0) AS cost_per_success,
  AVG(cost_usd) AS avg_cost_per_run,
  AVG(CASE WHEN outcome = 'success' THEN 1 ELSE 0 END) AS success_rate
FROM task_runs
WHERE created_at > NOW() - INTERVAL '7 days'
GROUP BY task_type
ORDER BY cost_per_success DESC;

Optimizing Based on Task Metrics

Once you have cost-per-task data, optimization becomes evidence-based:

High cost_per_success + low success_rate → improve prompt / use stronger model
Low success_rate + many retries → fix root cause, not retry count
High token count + high success → try compression or caching
Low success_rate on cheap model → route to more powerful model

See Model Routing in Production for implementing model selection based on task difficulty metrics.

FAQ

How do I define success for open-ended tasks like "answer the question"? Use LLM-as-judge: a second LLM evaluates whether the answer is complete, accurate, and relevant. Score 1–10, set a threshold (e.g., ≥7 = success). This adds ~$0.001 per evaluation.

Should I include failed runs in cost calculations? Yes — failed runs are part of the real cost. A system with 50% success rate effectively costs 2× per outcome. Including failures gives the honest picture.

What's a reasonable cost per task for common use cases? Customer support response: $0.005–0.02. Code review: $0.01–0.10. Research summary: $0.05–0.50. Document extraction: $0.002–0.01. These ranges vary hugely by complexity and quality bar.

How do I set cost budgets per task? Start with 3× the median successful run cost. Adjust based on business value: a $500 contract negotiation can afford a $0.50 AI cost; a $2 customer support ticket can afford $0.02.

What about latency — how does it factor in? Track cost_per_second_of_latency for user-facing tasks. Users tolerate up to ~10s for complex queries; beyond that, perceived value drops. Include latency in your optimization function: minimize cost + latency_penalty.