RAG from Scratch: Chunking, Embeddings, and Retrieval

RAG (Retrieval-Augmented Generation) is the go-to pattern for grounding LLM answers in your own documents. Instead of hoping the model memorized your data, you retrieve the relevant chunks at query time and inject them into the prompt. The model answers based on what you give it, not what it guesses.

The Core RAG Pipeline

Document ──▶ Chunk ──▶ Embed ──▶ Vector DB
                                      │
Query ──▶ Embed ──▶ Similarity search ─┘
                                      │
                                   Top-K chunks ──▶ LLM ──▶ Answer

Four steps: index your documents, embed the query, retrieve similar chunks, generate an answer with those chunks as context.

Step 1: Document Chunking

Chunking strategy matters more than the model. Bad chunks = bad retrieval = wrong answers.

Fixed-size chunking (simplest, often good enough):

function chunkByTokens(text: string, chunkSize = 512, overlap = 64): string[] {
  const words = text.split(/\s+/);
  const chunks: string[] = [];

  for (let i = 0; i < words.length; i += chunkSize - overlap) {
    chunks.push(words.slice(i, i + chunkSize).join(" "));
    if (i + chunkSize >= words.length) break;
  }
  return chunks;
}

The overlap (64 tokens here) ensures context isn't cut off at boundaries.

Semantic chunking (better for structured docs):

function chunkBySection(markdown: string): string[] {
  // Split on H2/H3 headers
  return markdown
    .split(/(?=^#{1,3} )/m)
    .map((s) => s.trim())
    .filter((s) => s.length > 50);
}

For code documentation, chunk by function/class boundaries. For legal documents, chunk by paragraph. Match chunk boundaries to the semantic units in your domain.

| Document type | Best chunk strategy | |--------------|---------------------| | Prose articles | Sentence-aware, 256–512 tokens | | Code docs | Per function/class | | PDFs | Per paragraph + page metadata | | Q&A pairs | Keep Q+A together as one chunk | | Legal contracts | Per clause/article |

Step 2: Embedding Models

The embedding model converts text to vectors. Choose based on your domain:

| Model | Dimensions | Speed | Best for | |-------|-----------|-------|---------| | text-embedding-3-small (OpenAI) | 1536 | Fast | General purpose | | text-embedding-3-large (OpenAI) | 3072 | Slow | High accuracy | | nomic-embed-text | 768 | Local | Self-hosted | | mxbai-embed-large | 1024 | Local | Code + technical |

import OpenAI from "openai";

const openai = new OpenAI();

async function embedTexts(texts: string[]): Promise<number[][]> {
  const response = await openai.embeddings.create({
    model: "text-embedding-3-small",
    input: texts,
    encoding_format: "float",
  });
  return response.data.map((d) => d.embedding);
}

// Batch for efficiency — don't call one-at-a-time
async function indexDocuments(chunks: string[]) {
  const BATCH_SIZE = 100;
  const vectors: number[][] = [];

  for (let i = 0; i < chunks.length; i += BATCH_SIZE) {
    const batch = chunks.slice(i, i + BATCH_SIZE);
    const embeddings = await embedTexts(batch);
    vectors.push(...embeddings);
  }
  return vectors;
}

Always batch embedding calls. Single-item calls are 10–20× more expensive per token than batches.

Step 3: Vector Storage

Use a vector database for similarity search at scale. For ≤100K chunks, even SQLite with sqlite-vec works. For production:

import { QdrantClient } from "@qdrant/js-client-rest";

const qdrant = new QdrantClient({ url: "http://localhost:6333" });

// Create collection
await qdrant.createCollection("knowledge-base", {
  vectors: { size: 1536, distance: "Cosine" },
});

// Index chunks
async function indexToQdrant(chunks: string[], vectors: number[][]) {
  const points = chunks.map((text, i) => ({
    id: i,
    vector: vectors[i],
    payload: { text, source: "docs", timestamp: Date.now() },
  }));

  await qdrant.upsert("knowledge-base", { points });
}

// Search
async function search(queryVector: number[], topK = 5) {
  const results = await qdrant.search("knowledge-base", {
    vector: queryVector,
    limit: topK,
    with_payload: true,
  });
  return results.map((r) => r.payload!.text as string);
}

Step 4: Retrieval and Generation

Putting it together:

import Anthropic from "@anthropic-ai/sdk";

const anthropic = new Anthropic();

async function rag(question: string): Promise<string> {
  // 1. Embed the query
  const [queryVector] = await embedTexts([question]);

  // 2. Retrieve relevant chunks
  const chunks = await search(queryVector, 5);

  // 3. Build context
  const context = chunks
    .map((chunk, i) => `[Source ${i + 1}]\n${chunk}`)
    .join("\n\n");

  // 4. Generate answer
  const response = await anthropic.messages.create({
    model: "claude-sonnet-4-6",
    max_tokens: 1024,
    system: `You are a helpful assistant. Answer questions using ONLY the provided context. 
If the answer isn't in the context, say "I don't have that information."`,
    messages: [
      {
        role: "user",
        content: `Context:\n${context}\n\nQuestion: ${question}`,
      },
    ],
  });

  return response.content[0].text;
}

Improving Retrieval Quality

Problem: retrieved chunks are relevant but miss the answer

Use query rewriting — generate multiple versions of the query:

async function rewriteQuery(query: string): Promise<string[]> {
  const response = await anthropic.messages.create({
    model: "claude-haiku-4-5-20251001",
    max_tokens: 200,
    messages: [{
      role: "user",
      content: `Generate 3 different search queries for: "${query}". Return JSON array of strings.`,
    }],
  });
  return JSON.parse(response.content[0].text);
}

async function ragWithQueryExpansion(question: string) {
  const queries = [question, ...await rewriteQuery(question)];
  const allVectors = await Promise.all(queries.map((q) => embedTexts([q]).then((e) => e[0])));

  // Deduplicate chunks across all queries
  const seen = new Set<string>();
  const chunks: string[] = [];
  for (const vec of allVectors) {
    for (const chunk of await search(vec, 3)) {
      if (!seen.has(chunk)) { seen.add(chunk); chunks.push(chunk); }
    }
  }
  return generateAnswer(question, chunks);
}

Problem: chunks are too long and dilute relevance

Reduce chunk size to 128–256 tokens. Smaller chunks = more precise retrieval. If you need surrounding context, fetch adjacent chunks after finding the matching one ("contextual retrieval").

Hybrid Search: Vector + Keyword

Vector search misses exact matches. Add BM25 keyword search for names, IDs, and code symbols:

async function hybridSearch(query: string, topK = 5) {
  const [vectorResults, keywordResults] = await Promise.all([
    vectorSearch(query, topK * 2),
    bm25Search(query, topK * 2),
  ]);

  // Reciprocal Rank Fusion
  const scores: Record<string, number> = {};
  const rank = (results: string[], k = 60) =>
    results.forEach((r, i) => { scores[r] = (scores[r] ?? 0) + 1 / (k + i + 1); });

  rank(vectorResults);
  rank(keywordResults);

  return Object.entries(scores)
    .sort((a, b) => b[1] - a[1])
    .slice(0, topK)
    .map(([text]) => text);
}

See Hybrid Search and Reranking for a deeper dive.

FAQ

How many chunks should I retrieve (top-K)? Start with K=5. Too few = missing context; too many = diluted signal and higher cost. Tune based on answer quality in your domain.

Should I store metadata with chunks? Yes — always store source, page, timestamp, and any domain-specific fields. Use metadata filters to scope retrieval (e.g., "only search docs from 2026").

How do I handle documents that change? Re-embed on update. Use a content hash to detect changes. Delete old vectors by document ID before inserting new ones.

What about PDFs with tables and images? Use a specialized parser (PyMuPDF, Marker, or Claude's vision API). Tables need special handling — convert to text or markdown before chunking.

Is RAG better than fine-tuning? RAG is better when: data changes frequently, you need citations, or you want to audit what the model used. Fine-tuning wins when: data is stable, you need a specific style/format, or latency is critical. See Fine-Tuning vs RAG vs Prompting.