[SL_CORE]
< CD ..||6 min read

Agentic RAG: From Naive Retrieval to Agent-Driven Search

# Agentic RAG replaces passive retrieval with an agent that decides what to search, validates results, and retries when needed. Learn the patterns that make RAG actually reliable in production.

[rag][ai-agents][retrieval][architecture]

Naive RAG is fragile: it embeds the query once, fetches the top-K chunks, and hopes they contain the answer. Agentic RAG replaces this passive step with an agent that reasons about what to retrieve, validates what it gets, and iterates until it has enough evidence to answer correctly.

What's Wrong with Naive RAG

In naive RAG, retrieval is a black box — the LLM never knows if the retrieved chunks actually answered the question. The result: confidently wrong answers backed by irrelevant context.

Three failure modes:

  1. Query-document mismatch — the user asks about "authentication failure" but docs use "login error"
  2. Multi-hop gaps — answer requires combining two facts from different documents
  3. Missing context — the retrieved chunk references another section not retrieved

Agentic RAG fixes all three.

The Agentic RAG Architecture

User Query
    │
    ▼
┌──────────────────────────────────────┐
│  Retrieval Agent                      │
│  ┌────────────────────────────────┐   │
│  │ 1. Decompose query into subqueries │
│  │ 2. Search for each subquery        │
│  │ 3. Validate retrieved content      │
│  │ 4. Identify gaps → retry search    │
│  │ 5. Synthesize final answer         │
│  └────────────────────────────────┘   │
└──────────────────────────────────────┘
    │
    ▼
Answer with evidence

Implementing the Retrieval Agent

The agent gets search as a tool and decides how to use it:

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

const RETRIEVAL_AGENT_SYSTEM = `You are a research assistant with access to a knowledge base.
To answer questions:
1. Break complex questions into specific sub-queries
2. Search for each sub-query
3. Check if results answer the question
4. Search again with different terms if results are insufficient
5. Synthesize a complete answer with sources

Always cite which source chunks support your answer.`;

const tools: Anthropic.Tool[] = [
  {
    name: "search_knowledge_base",
    description: "Search the knowledge base for relevant information. Returns top-5 relevant chunks.",
    input_schema: {
      type: "object" as const,
      properties: {
        query: { type: "string", description: "Search query — be specific" },
        filter: { type: "string", description: "Optional: filter by document type (api-docs, tutorials, changelogs)" },
      },
      required: ["query"],
    },
  },
];

async function agenticRAG(userQuestion: string): Promise<string> {
  const messages: Anthropic.MessageParam[] = [
    { role: "user", content: userQuestion },
  ];

  // Agentic loop
  while (true) {
    const response = await client.messages.create({
      model: "claude-sonnet-4-6",
      max_tokens: 2048,
      system: RETRIEVAL_AGENT_SYSTEM,
      tools,
      messages,
    });

    if (response.stop_reason === "end_turn") {
      // Agent finished — extract text answer
      return response.content
        .filter((b) => b.type === "text")
        .map((b) => b.text)
        .join("\n");
    }

    if (response.stop_reason === "tool_use") {
      // Execute tool calls
      messages.push({ role: "assistant", content: response.content });

      const toolResults: Anthropic.ToolResultBlockParam[] = [];
      for (const block of response.content) {
        if (block.type !== "tool_use") continue;

        if (block.name === "search_knowledge_base") {
          const { query, filter } = block.input as { query: string; filter?: string };
          const chunks = await searchKnowledgeBase(query, filter);
          toolResults.push({
            type: "tool_result",
            tool_use_id: block.id,
            content: chunks.length > 0
              ? chunks.map((c, i) => `[Chunk ${i + 1}] ${c}`).join("\n\n")
              : "No relevant results found for this query.",
          });
        }
      }

      messages.push({ role: "user", content: toolResults });
    }
  }
}

Query Decomposition

The most impactful improvement: break multi-part questions into specific sub-queries before searching.

async function decomposeQuery(question: string): Promise<string[]> {
  const response = await client.messages.create({
    model: "claude-haiku-4-5-20251001",
    max_tokens: 200,
    messages: [{
      role: "user",
      content: `Break this question into 2-4 specific search queries for a knowledge base.
Question: "${question}"
Return a JSON array of strings. Be specific — search queries should use terminology from technical docs.`,
    }],
  });

  try {
    return JSON.parse(response.content[0].text);
  } catch {
    return [question]; // fallback to original
  }
}

For "How do I configure authentication and set up rate limiting?" → two separate queries:

  • "configure authentication API keys"
  • "rate limiting setup configuration"

This dramatically improves recall on multi-topic questions.

Relevance Validation

After retrieval, verify the chunks actually answer the question before generating:

async function validateRelevance(
  question: string,
  chunks: string[]
): Promise<{ relevant: boolean; gaps: string[] }> {
  const response = await client.messages.create({
    model: "claude-haiku-4-5-20251001",
    max_tokens: 200,
    messages: [{
      role: "user",
      content: `Does this context sufficiently answer the question?
Question: "${question}"
Context: ${chunks.slice(0, 3).join("\n---\n")}

Reply with JSON: {"relevant": true/false, "gaps": ["what's missing if any"]}`,
    }],
  });

  return JSON.parse(response.content[0].text);
}

If relevant: false, the agent searches again with the identified gaps as new queries.

Iterative Retrieval Pattern

The full iterative loop:

async function iterativeRetrieval(question: string, maxRounds = 3): Promise<string[]> {
  const allChunks: string[] = [];
  const searchedQueries = new Set<string>();

  const queries = await decomposeQuery(question);

  for (let round = 0; round < maxRounds; round++) {
    const newQueries = queries.filter((q) => !searchedQueries.has(q));
    if (newQueries.length === 0) break;

    const results = await Promise.all(
      newQueries.map((q) => { searchedQueries.add(q); return searchKnowledgeBase(q); })
    );
    allChunks.push(...results.flat());

    const { relevant, gaps } = await validateRelevance(question, allChunks);
    if (relevant) break;

    // Add gap-filling queries for next round
    queries.push(...gaps);
  }

  return deduplicateChunks(allChunks);
}

Self-Correcting RAG

When the agent produces an answer, have a second agent check it:

async function ragWithVerification(question: string): Promise<string> {
  const chunks = await iterativeRetrieval(question);
  const answer = await generateAnswer(question, chunks);

  // Verify answer against source chunks
  const verification = await client.messages.create({
    model: "claude-haiku-4-5-20251001",
    max_tokens: 100,
    messages: [{
      role: "user",
      content: `Is this answer fully supported by the context? 
Answer: "${answer}"
Context: ${chunks.join("\n")}
Reply: {"supported": true/false, "unsupported_claims": []}`,
    }],
  });

  const { supported, unsupported_claims } = JSON.parse(verification.content[0].text);
  if (!supported) {
    // Regenerate with explicit instruction to stay grounded
    return generateAnswer(question, chunks, {
      constraint: `Do not claim: ${unsupported_claims.join(", ")}`,
    });
  }
  return answer;
}

When to Use Agentic vs. Naive RAG

| Scenario | Use | |----------|-----| | Simple factual Q&A, <500ms latency required | Naive RAG | | Multi-hop reasoning, complex questions | Agentic RAG | | High-stakes answers (legal, medical) | Agentic RAG + verification | | Batch document processing | Naive RAG (cost) | | Chatbots with conversation history | Agentic RAG |

See also RAG from Scratch for the foundation, and GraphRAG and Knowledge Graphs for structured knowledge retrieval.

FAQ

How much slower is agentic RAG? Naive RAG: 500ms–1s. Agentic RAG with 2 search rounds: 3–8s. Worth it for quality-critical applications; too slow for real-time autocomplete.

How many search rounds should I allow? Cap at 3 rounds. Beyond that, the returns diminish and costs rise. If 3 rounds aren't enough, the knowledge base has gaps — fix the content, not the retrieval.

Can agentic RAG hallucinate? Yes — the agent controls the search, but the generator can still hallucinate. Always include source-grounding instructions and use the verification step for high-stakes use cases.

What's the cost difference vs. naive RAG? Agentic RAG uses 3–5× more tokens (multiple search rounds + validation calls). Offset this by using fast/cheap models (Haiku) for the orchestration layer and only using Sonnet/Opus for final answer generation.

Does agentic RAG work with private documents? Yes — the search tool calls your internal vector DB. The agent never sees documents it doesn't retrieve.