[SL_CORE]
< CD ..||6 min read

AI Observability: Trace-Driven Debugging for Agents

# Standard APM tools can't debug AI agents — a correct and incorrect agent run look identical in logs. Learn trace-driven observability: structured tracing, cost tracking, and quality monitoring for production AI.

[observability][ai-agents][debugging][production]

AI agents fail silently. A standard log shows "request completed in 4.2s" — it doesn't tell you which tool call returned garbage, which LLM response went off-track, or why costs spiked 5× today. You need trace-driven observability: capturing every step of the agent's reasoning, not just the start and end.

Why Standard APM Fails for Agents

Standard APM tracks: request duration, error rate, throughput. For agents, these metrics are insufficient:

  • No reasoning visibility: which LLM call caused the wrong answer?
  • No cost attribution: which part of the workflow is expensive?
  • No quality tracking: did the agent actually solve the task?
  • No tool failure context: what did the tool return when the agent went wrong?

A trace for an AI agent needs to capture the full reasoning tree.

The Agent Trace Structure

Every agent run should produce a trace like this:

interface AgentTrace {
  traceId: string;
  runId: string;
  startedAt: number;
  completedAt?: number;
  status: "running" | "success" | "failed" | "timeout";
  goal: string;
  spans: AgentSpan[];
  totalCost: number;
  totalTokens: { input: number; output: number };
  outcome?: string;
}

interface AgentSpan {
  spanId: string;
  parentSpanId?: string;
  type: "llm_call" | "tool_call" | "retrieval" | "subagent";
  name: string;
  startedAt: number;
  duration: number;
  input: unknown;
  output: unknown;
  error?: string;
  tokens?: { input: number; output: number };
  cost?: number;
  model?: string;
}

Building a Lightweight Tracer

class AgentTracer {
  private trace: AgentTrace;
  private activeSpans: Map<string, AgentSpan> = new Map();

  constructor(goal: string) {
    this.trace = {
      traceId: crypto.randomUUID(),
      runId: crypto.randomUUID(),
      startedAt: Date.now(),
      status: "running",
      goal,
      spans: [],
      totalCost: 0,
      totalTokens: { input: 0, output: 0 },
    };
  }

  startSpan(type: AgentSpan["type"], name: string, input: unknown, parentSpanId?: string): string {
    const spanId = crypto.randomUUID();
    const span: AgentSpan = {
      spanId,
      parentSpanId,
      type,
      name,
      startedAt: Date.now(),
      duration: 0,
      input,
      output: null,
    };
    this.activeSpans.set(spanId, span);
    return spanId;
  }

  endSpan(spanId: string, output: unknown, meta?: Partial<AgentSpan>) {
    const span = this.activeSpans.get(spanId);
    if (!span) return;

    span.duration = Date.now() - span.startedAt;
    span.output = output;
    Object.assign(span, meta);

    if (meta?.tokens) {
      this.trace.totalTokens.input += meta.tokens.input;
      this.trace.totalTokens.output += meta.tokens.output;
    }
    if (meta?.cost) {
      this.trace.totalCost += meta.cost;
    }

    this.activeSpans.delete(spanId);
    this.trace.spans.push(span);
  }

  failSpan(spanId: string, error: string) {
    this.endSpan(spanId, null, { error });
  }

  complete(status: "success" | "failed", outcome?: string) {
    this.trace.status = status;
    this.trace.completedAt = Date.now();
    this.trace.outcome = outcome;
    return this.trace;
  }
}

Wrapping LLM Calls with Tracing

async function tracedLLMCall(
  tracer: AgentTracer,
  parentSpanId: string | undefined,
  model: string,
  messages: Anthropic.MessageParam[],
  options: Partial<Anthropic.MessageCreateParamsNonStreaming> = {}
) {
  const spanId = tracer.startSpan("llm_call", model, { messages }, parentSpanId);

  try {
    const response = await anthropic.messages.create({
      model,
      messages,
      max_tokens: 2048,
      ...options,
    });

    const cost = calculateCost(model, response.usage);

    tracer.endSpan(spanId, response.content, {
      tokens: { input: response.usage.input_tokens, output: response.usage.output_tokens },
      cost,
      model,
    });

    return response;
  } catch (err: any) {
    tracer.failSpan(spanId, err.message);
    throw err;
  }
}

function calculateCost(model: string, usage: Anthropic.Usage): number {
  const rates: Record<string, [number, number]> = {
    "claude-haiku-4-5-20251001":   [0.00025, 0.00125],
    "claude-sonnet-4-6":           [0.003,   0.015],
    "claude-opus-4-8":             [0.015,   0.075],
  };
  const [iRate, oRate] = rates[model] ?? [0.003, 0.015];
  return usage.input_tokens * iRate / 1000 + usage.output_tokens * oRate / 1000;
}

Tracing Tool Calls

async function tracedToolCall(
  tracer: AgentTracer,
  parentSpanId: string,
  toolName: string,
  args: unknown,
  executor: (args: unknown) => Promise<unknown>
) {
  const spanId = tracer.startSpan("tool_call", toolName, args, parentSpanId);

  try {
    const result = await Promise.race([
      executor(args),
      new Promise((_, reject) =>
        setTimeout(() => reject(new Error("Tool timeout after 30s")), 30_000)
      ),
    ]);
    tracer.endSpan(spanId, result);
    return result;
  } catch (err: any) {
    tracer.failSpan(spanId, err.message);
    throw err;
  }
}

Shipping Traces to Observability Platforms

Export traces to OpenTelemetry (compatible with Langfuse, Datadog, Jaeger):

import { trace, SpanStatusCode } from "@opentelemetry/api";

async function exportToOTel(agentTrace: AgentTrace) {
  const tracer = trace.getTracer("ai-agent");

  for (const span of agentTrace.spans) {
    const otelSpan = tracer.startSpan(span.name, {
      startTime: span.startedAt,
      attributes: {
        "ai.type": span.type,
        "ai.model": span.model ?? "",
        "ai.tokens.input": span.tokens?.input ?? 0,
        "ai.tokens.output": span.tokens?.output ?? 0,
        "ai.cost.usd": span.cost ?? 0,
        "ai.trace.id": agentTrace.traceId,
      },
    });

    if (span.error) {
      otelSpan.setStatus({ code: SpanStatusCode.ERROR, message: span.error });
    }

    otelSpan.end(span.startedAt + span.duration);
  }
}

For simpler setups, log structured JSON and ship to your existing log aggregator:

function logTrace(trace: AgentTrace) {
  console.log(JSON.stringify({
    type: "agent_trace",
    traceId: trace.traceId,
    status: trace.status,
    duration: (trace.completedAt ?? Date.now()) - trace.startedAt,
    cost_usd: trace.totalCost.toFixed(6),
    tokens_in: trace.totalTokens.input,
    tokens_out: trace.totalTokens.output,
    spans: trace.spans.length,
    failed_spans: trace.spans.filter((s) => s.error).length,
    goal_summary: trace.goal.slice(0, 100),
  }));
}

Cost Anomaly Detection

Alert when a single trace costs more than expected:

const COST_THRESHOLDS: Record<string, number> = {
  chat_agent: 0.05,       // $0.05 per conversation
  research_agent: 0.50,   // $0.50 per deep research run
  code_review: 0.10,      // $0.10 per review
};

function checkCostAnomaly(trace: AgentTrace, agentType: string) {
  const threshold = COST_THRESHOLDS[agentType];
  if (threshold && trace.totalCost > threshold * 3) {
    console.error(JSON.stringify({
      alert: "cost_anomaly",
      traceId: trace.traceId,
      cost: trace.totalCost,
      threshold,
      multiplier: (trace.totalCost / threshold).toFixed(1),
      topSpans: trace.spans
        .sort((a, b) => (b.cost ?? 0) - (a.cost ?? 0))
        .slice(0, 3)
        .map((s) => ({ name: s.name, cost: s.cost })),
    }));
  }
}

Quality Metrics

Track outcome quality alongside cost and latency:

interface QualityMetric {
  traceId: string;
  taskCompleted: boolean;
  toolErrors: number;
  retries: number;
  contextUtilization: number; // tokens used / max tokens
  hallucinated: boolean;      // from a separate verification step
}

function measureQuality(trace: AgentTrace): QualityMetric {
  const toolErrors = trace.spans
    .filter((s) => s.type === "tool_call" && s.error)
    .length;

  const retries = trace.spans
    .filter((s) => s.type === "llm_call")
    .length - 1; // first call isn't a retry

  return {
    traceId: trace.traceId,
    taskCompleted: trace.status === "success",
    toolErrors,
    retries: Math.max(0, retries),
    contextUtilization: trace.totalTokens.input / 200_000,
    hallucinated: false, // set by verification step
  };
}

See also Monitoring Agentic AI for dashboards and alerting on these metrics.

FAQ

What's the difference between logging and tracing? Logs are flat events. Traces are trees: parent-child spans that show the causal chain. Tracing is essential for understanding why an agent took 45 tool calls to complete a simple task.

How much does tracing add to latency? Synchronous span recording: <1ms. Async export to a backend: 0ms (non-blocking). The overhead is negligible.

Should I trace 100% of requests? In development: yes. In production: trace 100% but export full detail for only 1–5% (sampled). Always export on errors.

What's the best tracing backend for AI agents? Langfuse (open-source, AI-native), OpenTelemetry + Jaeger (standard), or Datadog APM (if you're already using it). Avoid building your own — the query and alerting layers are the hard parts.

How long should I retain traces? Keep 7 days of full traces. Aggregate metrics forever. Retain traces for failed runs for 30 days for debugging.