[SL_CORE]
< CD ..||6 min read

Fine-Tuning vs RAG vs Prompting: Decision Framework

# Fine-tuning, RAG, and prompt engineering solve different problems. Use the wrong one and you'll waste months. This decision framework tells you exactly which to choose — and when to combine them.

[fine-tuning][rag][prompt-engineering][llm]

Fine-tuning, RAG, and prompt engineering are not competing approaches — they solve different problems. Teams that choose wrong spend months building the wrong solution. This framework tells you which one to use based on what you're actually trying to solve.

The Core Distinction

| Approach | Changes the model? | Changes the context? | Best for | |----------|-------------------|---------------------|---------| | Prompt engineering | No | No | Format, tone, reasoning style | | RAG | No | Yes (adds retrieved docs) | Knowledge, facts, dynamic data | | Fine-tuning | Yes | No | Style, behavior, specialized format |

The most common mistake: using fine-tuning when the problem is actually a knowledge problem (RAG's job), or using RAG when the problem is a behavior problem (fine-tuning's job).

When to Use Prompt Engineering (Default)

Start here. Always. Prompt engineering is free, instant, and reversible.

Use prompt engineering when:

  • The model can do the task but doesn't do it in your preferred style
  • You need specific output formats (JSON, markdown tables, numbered lists)
  • You want specific reasoning patterns (chain-of-thought, step-by-step)
  • You need to constrain behavior ("only answer about X", "always ask clarifying questions")
// Before: model returns verbose paragraphs
const badPrompt = "Analyze this error and tell me what's wrong";

// After: precise format control via prompt
const goodPrompt = `Analyze this error. Return JSON only:
{
  "root_cause": "one sentence",
  "fix": "specific code change",
  "severity": "critical|high|medium|low"
}

Error: ${errorMessage}`;

If prompt engineering doesn't solve it after 2–3 serious attempts, move to RAG or fine-tuning.

When to Use RAG

Use RAG when the model doesn't know the answer — not when it behaves wrong.

RAG's job: give the model access to knowledge it wasn't trained on.

Use RAG when:

  • Answering questions about your private documents, codebase, or database
  • The answer changes frequently (product prices, docs, news)
  • You need citations/sources in the answer
  • The knowledge base is >100 pages (too large for prompt)
// RAG is the answer when:
"What's the return policy for order #12345?"  // → private order data
"How do I configure the auth middleware?"     // → your internal docs
"What did the user say in last week's call?"  // → CRM data
"What are the latest API changes?"            // → changelog, updated frequently

Don't use RAG when:

  • The model already knows the facts (general knowledge)
  • You want the model to write in a specific style (fine-tuning's job)
  • Latency is critical and retrieval adds too much (use fine-tuning or larger cache)

See RAG from Scratch for implementation.

When to Fine-Tune

Fine-tune when you need to change how the model behaves, not what it knows.

Fine-tuning is expensive ($50–$500+ per run) and slow (hours). Reserve it for:

Use fine-tuning when:

  • You need a very specific output format the model consistently gets wrong despite good prompting
  • You have a specialized domain vocabulary the model misuses
  • You need the model to adopt a persona or writing style across all outputs
  • Latency is critical and you need to shorten the system prompt (fine-tuned behaviors need no instruction)
  • You have 100+ labeled examples of input → desired output

Real example where fine-tuning wins:

# Problem: model generates SQL but uses wrong dialect (MySQL vs. PostgreSQL)
# Prompt engineering: works but requires large, fragile system prompts
# Fine-tuning: 200 examples of (schema + question) → correct PostgreSQL query
#   Result: short system prompt, consistent format, 40ms faster (no system prompt tokens)

Don't fine-tune when:

  • You have <100 examples (few-shot prompting works better)
  • The data changes frequently (fine-tuned knowledge goes stale)
  • You need citations (fine-tuning bakes in knowledge, can't point to source)
  • You're still iterating on the product (fine-tuning locks in behavior)

The Hybrid: RAG + Fine-Tuning

For the highest-quality specialized applications, combine both:

| Layer | Role | |-------|------| | Fine-tuning | Teaches the model how to answer (format, style, reasoning pattern) | | RAG | Gives the model what to answer (current facts, private data) |

Example: a legal contract analyzer

  • Fine-tuned on 500 contract analysis examples → model knows the right analytical framework and output format
  • RAG over the actual contracts → model uses the framework on current, real contract text

This combination is harder to build but outperforms either approach alone.

The Decision Tree

Can prompt engineering solve it?
├── YES → Use prompt engineering. Done.
└── NO
    │
    Is it a knowledge/data problem?
    ├── YES → Does the data change frequently or is it private?
    │         ├── YES → RAG
    │         └── NO, static general knowledge → consider fine-tuning or larger prompt
    │
    └── NO, it's a behavior problem
        │
        Do you have 100+ examples?
        ├── YES → Fine-tuning
        └── NO → Improve examples and prompting first

Cost/Complexity Comparison

| Approach | One-time cost | Ongoing cost | Iteration speed | |----------|-------------|-------------|-----------------| | Prompt engineering | Low | Token cost only | Minutes | | RAG | Medium (infra, embedding) | Vector DB + embeddings | Hours | | Fine-tuning | High ($50–$500/run) | Inference (same as base) | Days | | RAG + Fine-tuning | High | Both above | Week+ |

At Shahriar Labs, we default to prompt engineering, add RAG when there's a knowledge problem (like in LetX), and reserve fine-tuning for format-critical workflows in QuantumSketch.

FAQ

Can fine-tuning add knowledge that wasn't in the training data? Yes, but it's unreliable. Fine-tuned "knowledge" can't be cited, goes stale, and the model may hallucinate about it. Use RAG for knowledge — fine-tuning for behavior.

How many examples do I need to fine-tune? Minimum viable: 50–100. Good results: 500–1000. Beyond 5000, returns diminish. Quality matters more than quantity — 100 perfect examples beat 1000 mediocre ones.

Is RAG's retrieval always accurate? No — this is the main limitation. Poor chunking, wrong embedding model, or bad queries return irrelevant chunks. See Agentic RAG for iterative retrieval that catches these failures.

Does fine-tuning work with Claude? Anthropic offers fine-tuning for Claude on selected plans. For most use cases, Claude's instruction-following is strong enough that fine-tuning isn't needed. OpenAI's fine-tuning API is more accessible for experimentation.

What about RLHF / constitutional AI? These are training techniques, not product-level choices. You'd need Anthropic to do this with Claude. At the application level, your choices remain: prompt engineering, RAG, or fine-tuning (instruction tuning).