Domain-Specific LLM Agents: Fine-Tuning vs Specialized Tools
# Domain-specific agents outperform general ones on specialized tasks. The right approach isn't always fine-tuning — specialized tools, curated RAG, and constrained prompts often beat expensive training runs.
A medical diagnosis agent built on a general LLM with no domain adaptation will miss nuances that a properly domain-adapted agent catches. But fine-tuning is expensive and slow. In most cases, specialized tools + curated RAG + constrained prompts produce equivalent results at a fraction of the cost.
The Domain Adaptation Stack
Level 1: Prompt + constraints (free, fast)
Level 2: Domain-specific RAG (moderate cost)
Level 3: Specialized tools (build once, reuse forever)
Level 4: Fine-tuning (expensive, only when above fails)
Always start at Level 1 and move up only when the previous level provably doesn't meet quality requirements.
Level 1: Domain-Constrained Prompts
The first step is telling the model what domain it's in and what it must/must not do:
function buildDomainSystemPrompt(domain: Domain): string {
const domainPrompts: Record<string, string> = {
medical: `You are a clinical documentation assistant.
Vocabulary: use standard medical terminology (ICD-10, CPT codes, SNOMED CT)
Never: provide diagnoses, recommend treatments, interpret test results without physician review
Always: note when clinical judgment is required, flag abnormal values for physician attention
Format: use standard clinical note structure (SOAP, BIRP, etc.)`,
legal: `You are a legal research assistant specializing in contract law.
Jurisdiction: default to US federal law, note state variations where significant
Never: provide legal advice, predict case outcomes, or make definitive statements about law applicability
Always: cite sources, note jurisdictional variations, flag ambiguous language for attorney review`,
security: `You are a security code reviewer focused on OWASP Top 10 vulnerabilities.
Framework: CVSS scoring, NIST security framework
Never: provide exploit code, assist in offensive security without explicit authorization context
Always: provide specific remediation steps, reference CVEs where applicable`,
};
return domainPrompts[domain] ?? "You are a helpful assistant.";
}
Level 2: Domain-Specific RAG
Curate your knowledge base for the domain. Quality matters more than quantity:
interface DomainKnowledgeBase {
name: string;
documents: DomainDocument[];
chunkingStrategy: "sentence" | "section" | "paragraph" | "fixed";
embeddingModel: string;
metadata: {
authorityLevel: "primary" | "secondary" | "tertiary";
dateRange: { from: string; to: string };
jurisdiction?: string; // for legal
specialty?: string; // for medical
};
}
// Medical RAG: curate high-quality sources
const MEDICAL_KNOWLEDGE_BASE: DomainKnowledgeBase = {
name: "medical-clinical",
documents: [
// Clinical guidelines (highest authority)
...await loadDocuments("sources/clinical-guidelines/**/*.pdf", "primary"),
// Drug references
...await loadDocuments("sources/drug-references/**/*.txt", "primary"),
// Internal protocols
...await loadDocuments("sources/internal-protocols/**/*.md", "primary"),
],
chunkingStrategy: "section", // medical docs are structured by section
embeddingModel: "text-embedding-3-large", // higher quality for critical domain
metadata: { authorityLevel: "primary", dateRange: { from: "2023-01-01", to: "2026-01-01" } },
};
// When querying, filter by authority level and recency
async function domainRAGQuery(
query: string,
kb: DomainKnowledgeBase,
filter?: { authorityLevel?: string; minDate?: string }
) {
const embedding = await embed(query);
return vectorDB.search(kb.name, embedding, {
filter: {
authorityLevel: filter?.authorityLevel ?? "primary",
...(filter?.minDate ? { date: { gte: filter.minDate } } : {}),
},
limit: 5,
});
}
Level 3: Specialized Tools
Domain-specific tools give the agent capabilities general tools lack:
// Medical agent tools
const MEDICAL_TOOLS: Anthropic.Tool[] = [
{
name: "lookup_drug_interaction",
description: "Check for drug-drug interactions. Returns interaction severity and clinical notes.",
input_schema: {
type: "object",
properties: {
drug_a: { type: "string", description: "Generic drug name or brand name" },
drug_b: { type: "string", description: "Generic drug name or brand name" },
},
required: ["drug_a", "drug_b"],
},
},
{
name: "icd10_lookup",
description: "Look up ICD-10 diagnosis codes. Returns code, description, and parent category.",
input_schema: {
type: "object",
properties: {
query: { type: "string", description: "Diagnosis description or partial code" },
},
required: ["query"],
},
},
{
name: "calculate_dosage",
description: "Calculate medication dosage based on patient weight and renal function.",
input_schema: {
type: "object",
properties: {
drug: { type: "string" },
weight_kg: { type: "number" },
gfr: { type: "number", description: "Glomerular filtration rate" },
indication: { type: "string" },
},
required: ["drug", "weight_kg"],
},
},
];
// Legal agent tools
const LEGAL_TOOLS: Anthropic.Tool[] = [
{
name: "search_case_law",
description: "Search for relevant case law citations. Returns case name, citation, holding, and relevance.",
input_schema: {
type: "object",
properties: {
query: { type: "string" },
jurisdiction: { type: "string", enum: ["federal", "ca", "ny", "tx"] },
dateRange: { type: "object", properties: { from: { type: "string" }, to: { type: "string" } } },
},
required: ["query"],
},
},
];
When to Fine-Tune
Fine-tuning domain agents makes sense when:
- Format is critical and complex — specific document structures that prompt engineering can't reliably produce
- Domain vocabulary is extensive — terminology the base model misuses consistently
- Latency is critical — fine-tuned model with shorter system prompt
- High volume justifies cost — >10,000 tasks/day where quality gains compound
// Evaluate whether fine-tuning is justified
async function evaluateFinetuningNeed(
baselineQuality: number,
targetQuality: number,
dailyRequests: number,
costPerRequest: number,
finetuningCost: number,
qualityImprovementFromFT: number
): Promise<{ justified: boolean; paybackDays: number }> {
const qualityGap = targetQuality - baselineQuality;
const promptEngineeringMax = 0.15; // max 15% improvement from prompting
if (qualityGap <= promptEngineeringMax) {
return { justified: false, paybackDays: Infinity };
}
// Cost savings from reduced retries with fine-tuned model
const currentRetryRate = 1 - baselineQuality;
const ftRetryRate = 1 - (baselineQuality + qualityImprovementFromFT);
const dailyCostSavings = (currentRetryRate - ftRetryRate) * dailyRequests * costPerRequest;
const paybackDays = dailyCostSavings > 0 ? finetuningCost / dailyCostSavings : Infinity;
return { justified: paybackDays < 90, paybackDays };
}
Domain Validation Layer
Add domain-specific validation to catch agent errors:
// Medical output validator
async function validateMedicalOutput(output: string): Promise<ValidationResult> {
const checks = [
// Never recommend specific treatments
{ test: () => !/prescribe|recommend .{0,30}mg|take .{0,20}mg daily/i.test(output), issue: "Contains treatment recommendation" },
// Always flag abnormal values
{ test: () => !/elevated|high|low|abnormal/i.test(output) || /physician|provider|review/i.test(output), issue: "Abnormal value without physician referral" },
// ICD-10 codes should be valid format
{ test: () => !/\b[A-Z]\d{2}(?:\.\d{1,4})?\b/.test(output) || validateICD10Codes(output), issue: "Invalid ICD-10 code format" },
];
const failures = checks.filter((c) => !c.test());
return { valid: failures.length === 0, issues: failures.map((f) => f.issue) };
}
FAQ
Should domain agents be separate deployments or same service? Separate deployments. Different domains have different: scaling needs (medical gets less traffic than customer support), update schedules (legal needs more frequent knowledge updates), and access controls.
How do I keep domain knowledge current? Set up ingestion pipelines: when domain sources update (new guidelines, new case law), auto-ingest to the RAG knowledge base. Track source freshness dates. Flag outdated context to the agent.
Can one agent serve multiple domains? Yes, but with explicit routing: "based on the user's question, this is a medical query" → load medical tools + knowledge base. Don't mix domains in a single agent session — it degrades reliability.
How specific should domain specialization be? Match specialization to the task distribution. "Medical" is too broad if 80% of queries are about cardiology — build a cardiology specialist. Over-specialization adds maintenance burden without proportional quality gain.
What's the evaluation dataset for a domain agent? 100+ examples created with domain experts. For medical: reviewed by licensed clinicians. For legal: reviewed by attorneys. For security: reviewed by security engineers. Domain expertise in the eval set is non-negotiable for high-stakes applications.