LLM Evaluation Frameworks: Beyond Benchmarks to Prod
# Public benchmarks don't predict production performance. Build domain-specific evaluation pipelines with DeepEval, Langfuse, and custom metrics that actually reflect what your users experience.
GPT-5 scores 94% on MMLU. Claude Opus achieves state-of-the-art on HumanEval. These benchmarks tell you nothing about whether either model will work for your specific customer support bot or code generation tool. Build your own evaluation framework — one that measures what your users actually care about.
Why Public Benchmarks Fail
Public benchmarks measure general capability. Production evaluation measures task-specific performance.
| Benchmark | Tests | Doesn't test | |-----------|-------|-------------| | MMLU | General knowledge | Your domain terminology | | HumanEval | Python algorithms | Your codebase's patterns | | MT-Bench | Multi-turn chat | Your product's specific use cases | | TruthfulQA | Truthfulness in general | Hallucinations about your product |
A model that scores 5% better on MMLU may score 15% worse on your domain. Always evaluate on your use cases.
DeepEval: Production-Grade Evaluation
DeepEval is an open-source evaluation framework with LLM-based metrics:
pip install deepeval
# or for TypeScript:
npm install deepeval
import { evaluate, LLMTestCase, AnswerRelevancyMetric, FaithfulnessMetric, ContextualPrecisionMetric } from "deepeval";
// Define test cases
const testCases: LLMTestCase[] = [
{
input: "How do I reset my password?",
actualOutput: await myRAGSystem("How do I reset my password?"),
expectedOutput: "To reset your password, click 'Forgot Password' on the login page.",
retrievalContext: ["Password reset instructions..."],
},
// ... more test cases
];
// Run evaluation with multiple metrics
const results = await evaluate(testCases, [
new AnswerRelevancyMetric({ threshold: 0.7 }), // Is answer relevant to question?
new FaithfulnessMetric({ threshold: 0.8 }), // Is answer grounded in context?
new ContextualPrecisionMetric({ threshold: 0.7 }), // Is retrieved context relevant?
]);
console.log(results.testResults);
Custom Domain Metrics
For domain-specific quality, build custom metrics:
import { BaseMetric, LLMTestCase } from "deepeval";
import Anthropic from "@anthropic-ai/sdk";
class TechnicalAccuracyMetric extends BaseMetric {
name = "technical_accuracy";
private domain: "code" | "security" | "math";
constructor(domain: "code" | "security" | "math", threshold = 0.8) {
super(threshold);
this.domain = domain;
}
async measure(testCase: LLMTestCase): Promise<number> {
const domainCriteria = {
code: "code is syntactically valid, follows best practices, handles edge cases",
security: "identifies security vulnerabilities accurately, no false positives on safe code",
math: "calculations are correct, units are right, reasoning is valid",
};
const response = await anthropic.messages.create({
model: "claude-opus-4-8",
max_tokens: 200,
messages: [{
role: "user",
content: `Evaluate this ${this.domain} response for technical accuracy.
Criteria: ${domainCriteria[this.domain]}
Input: ${testCase.input}
Response: ${testCase.actualOutput}
Return JSON: {"score": 0.0-1.0, "issues": ["..."], "reasoning": "brief"}`,
}],
});
const { score } = JSON.parse(response.content[0].text);
this.score = score;
return score;
}
}
Regression Suite Structure
Organize your eval suite to catch specific failure modes:
interface EvalSuite {
name: string;
description: string;
cases: Array<{
id: string;
category: string;
weight: number; // some categories matter more
input: string;
idealOutput?: string;
criteria: string[];
mustNotContain?: string[]; // zero-tolerance items
}>;
thresholds: {
overall: number;
byCategory: Record<string, number>;
mustPass: string[]; // case IDs that must pass regardless of overall score
};
}
const SUPPORT_BOT_SUITE: EvalSuite = {
name: "customer-support-v2",
description: "Customer support agent evaluation",
cases: [
{
id: "password-reset-basic",
category: "account-management",
weight: 1.0,
input: "I forgot my password",
criteria: [
"Provides password reset steps",
"Does not ask for current password",
],
mustNotContain: ["your password is", "I cannot help"],
},
{
id: "refund-escalation",
category: "billing",
weight: 2.0, // billing is high priority
input: "I want a refund for last month",
criteria: [
"Acknowledges refund request",
"Escalates to human agent",
],
},
// ... more cases
],
thresholds: {
overall: 0.88,
byCategory: { billing: 0.95, security: 0.99, account: 0.85 },
mustPass: ["security-jailbreak-1", "security-pii-leak"],
},
};
Running Evals Against Multiple Models
Compare models on your specific use case before migrating:
async function compareModels(
suite: EvalSuite,
models: string[]
): Promise<ModelComparison[]> {
const results = await Promise.all(
models.map(async (model) => {
const scores = await runSuite(suite, model);
return { model, ...scores };
})
);
return results.sort((a, b) => b.overallScore - a.overallScore);
}
const comparison = await compareModels(SUPPORT_BOT_SUITE, [
"claude-haiku-4-5-20251001",
"claude-sonnet-4-6",
"gpt-4o-mini",
"gpt-4o",
]);
// Output:
// claude-sonnet-4-6: 94.2% ($0.018/task)
// gpt-4o: 92.8% ($0.022/task)
// gpt-4o-mini: 87.1% ($0.003/task)
// claude-haiku-4-5: 85.3% ($0.004/task)
Langfuse: Production Eval Tracking
Langfuse provides evaluation tracking integrated with LLM tracing:
import Langfuse from "langfuse";
const langfuse = new Langfuse({
secretKey: process.env.LANGFUSE_SECRET_KEY,
publicKey: process.env.LANGFUSE_PUBLIC_KEY,
});
// Trace LLM calls
async function tracedCompletion(input: string, userId: string) {
const trace = langfuse.trace({ name: "support-response", userId });
const generation = trace.generation({
name: "main-response",
model: "claude-sonnet-4-6",
input,
});
const response = await anthropic.messages.create({/* ... */});
const output = response.content[0].text;
generation.end({ output, usage: response.usage });
// Add quality score (async evaluation)
const { score } = await evaluateQuality(input, output);
trace.score({ name: "quality", value: score });
await langfuse.flushAsync();
return output;
}
Langfuse shows you evaluation trends over time, broken down by model, user segment, and prompt version.
The Evaluation Pyramid
┌──────────────┐
│ Human review │ ← 1% of prod traffic
│ (gold labels)│
/└──────────────┘\
/ ┌────────────┐ \
/ │ LLM judge │ \
/ │ (sampled) │ \
/ └────────────┘ \
/ ┌──────────────────┐ \
/ │ Deterministic │ \
/ │ (regex, schema) │ \
/─────┴──────────────────┴────────\
│ Unit evals (golden dataset, CI) │ ← 100% of changes
└───────────────────────────────────┘
Cheap deterministic checks run on every change. LLM-as-judge runs on sampled traffic. Human review is the ground truth for recalibrating the automated metrics.
FAQ
How do I handle open-ended tasks with no right answer? Use LLM-as-judge with specific rubrics: "Does the response address the question completely?", "Is the tone appropriate for customer support?", "Would this response satisfy a typical user?". Rate each rubric 1–5, average them.
What's a good size for a golden dataset? 50 minimum to catch obvious regressions. 200 for reliable statistical significance. 1000+ for production-confidence metrics. Build it over time — add examples whenever you find a real failure.
Should eval results block deployment? Block on: security tests, critical category drops >5%, new failure modes. Warn on: general quality drop >2%, new edge case failures. Never block on: non-significant score fluctuations.
How do I calibrate LLM-as-judge scores? Have humans rate 50 outputs, then see how closely the LLM judge matches. A well-calibrated judge agrees with humans >80% of the time. If agreement is lower, refine the judging prompt.
Can I use the same model as both the task model and the judge? Avoid using Sonnet to evaluate Sonnet outputs — it tends to agree with itself. Use Opus to evaluate Sonnet/Haiku outputs, or use a different provider entirely for the judge.