Agent Memory Architecture: Beyond Session Context

Context windows are temporary. When an agent conversation ends, everything in that window disappears. For agents that need to remember user preferences, learn from past mistakes, or accumulate knowledge over weeks, you need a persistent memory architecture. Here's how to build one.

The Four Memory Types

Cognitive science gives us a useful framework. Agents need the same four types:

| Memory type | What it stores | Persistence | Access pattern | |-------------|---------------|-------------|----------------| | Working | Current task context | Session only | In-context | | Episodic | Past conversations and events | Long-term | Retrieval by time/topic | | Semantic | Facts, knowledge, preferences | Long-term | Retrieval by query | | Procedural | Skills, workflows, learned behaviors | Long-term | Applied automatically |

Most agents only have working memory (the context window). Adding the others turns a stateless tool into a learning, personalizing agent.

Working Memory: Managing the Context Window

Working memory = what's in the prompt right now. The constraint is the context window size.

Prioritize content in this order when the window fills:

interface WorkingMemoryManager {
  systemPrompt: string;       // always present
  taskContext: string;        // current task details
  recentHistory: Message[];   // last N turns
  retrievedMemories: string[]; // relevant episodic/semantic memories
  tools: Tool[];              // available tools
}

function buildContextWindow(wm: WorkingMemoryManager, maxTokens: number): string {
  const parts = [
    wm.systemPrompt,
    wm.retrievedMemories.join("\n"),
    wm.taskContext,
    formatHistory(wm.recentHistory.slice(-10)), // keep last 10 turns max
  ];

  // Trim to fit — drop oldest history first
  let context = parts.join("\n\n");
  while (estimateTokens(context) > maxTokens && wm.recentHistory.length > 2) {
    wm.recentHistory.shift();
    context = parts.join("\n\n");
  }
  return context;
}

Episodic Memory: Storing Past Interactions

Episodic memory stores what happened: past conversations, completed tasks, errors encountered.

interface EpisodicMemory {
  id: string;
  timestamp: number;
  summary: string;        // LLM-generated summary of the episode
  keyFacts: string[];     // extracted facts
  outcome: "success" | "failure" | "partial";
  embedding: number[];    // for semantic retrieval
}

// After each conversation, compress it into a memory
async function createEpisodicMemory(
  conversation: Message[],
  outcome: EpisodicMemory["outcome"]
): Promise<EpisodicMemory> {
  const response = await anthropic.messages.create({
    model: "claude-haiku-4-5-20251001",
    max_tokens: 300,
    messages: [{
      role: "user",
      content: `Summarize this conversation in 2-3 sentences and extract 3-5 key facts.
Return JSON: {"summary": "...", "keyFacts": ["fact1", ...]}.
Conversation: ${conversation.map((m) => `${m.role}: ${m.content}`).join("\n")}`,
    }],
  });

  const { summary, keyFacts } = JSON.parse(response.content[0].text);
  const [embedding] = await embedTexts([summary]);

  return {
    id: crypto.randomUUID(),
    timestamp: Date.now(),
    summary,
    keyFacts,
    outcome,
    embedding,
  };
}

Store these in a vector database with timestamp metadata. Retrieve by semantic similarity to the current context.

Semantic Memory: Long-Term Knowledge and Preferences

Semantic memory stores facts about users, the world, and learned knowledge — without the conversational wrapper.

interface SemanticMemory {
  id: string;
  content: string;         // "User prefers Python over TypeScript"
  category: string;        // "preference" | "fact" | "skill" | "constraint"
  confidence: number;      // 0–1
  sources: string[];       // episodic memory IDs that support this
  updatedAt: number;
  embedding: number[];
}

// Extract semantic memories from episodic ones
async function extractSemanticMemories(
  episode: EpisodicMemory,
  userId: string
): Promise<SemanticMemory[]> {
  const response = await anthropic.messages.create({
    model: "claude-haiku-4-5-20251001",
    max_tokens: 400,
    messages: [{
      role: "user",
      content: `Extract generalizable facts or preferences from this episode summary.
Only extract things that would be useful to know in future conversations.
Return JSON array: [{"content": "...", "category": "preference|fact|skill|constraint", "confidence": 0.0-1.0}]
Episode: ${episode.summary}\nKey facts: ${episode.keyFacts.join(", ")}`,
    }],
  });

  const extracted = JSON.parse(response.content[0].text);
  return Promise.all(
    extracted.map(async (m: any) => ({
      id: crypto.randomUUID(),
      content: m.content,
      category: m.category,
      confidence: m.confidence,
      sources: [episode.id],
      updatedAt: Date.now(),
      embedding: (await embedTexts([m.content]))[0],
    }))
  );
}

When the same fact appears multiple times, update confidence rather than duplicating:

async function upsertSemanticMemory(
  newMemory: SemanticMemory,
  existingMemories: SemanticMemory[]
): Promise<SemanticMemory> {
  // Find similar existing memory
  const similar = existingMemories.find(
    (m) => cosineSimilarity(m.embedding, newMemory.embedding) > 0.92
  );

  if (similar) {
    // Update confidence, add source
    return {
      ...similar,
      confidence: Math.min(1, similar.confidence + 0.1),
      sources: [...new Set([...similar.sources, ...newMemory.sources])],
      updatedAt: Date.now(),
    };
  }
  return newMemory;
}

Procedural Memory: Learned Workflows

Procedural memory stores how to do things — workflows that have worked before.

interface ProceduralMemory {
  id: string;
  trigger: string;         // "when asked to debug TypeScript errors"
  steps: string[];         // ordered steps the agent should follow
  successRate: number;     // measured from outcomes
  lastUsed: number;
}

// Inject relevant procedural memories into system prompt
async function getRelevantProcedures(
  currentTask: string,
  allProcedures: ProceduralMemory[]
): Promise<ProceduralMemory[]> {
  if (allProcedures.length === 0) return [];

  const taskEmbedding = (await embedTexts([currentTask]))[0];
  const triggerEmbeddings = await embedTexts(allProcedures.map((p) => p.trigger));

  return allProcedures
    .map((p, i) => ({ proc: p, score: cosineSimilarity(taskEmbedding, triggerEmbeddings[i]) }))
    .filter(({ score }) => score > 0.75)
    .sort((a, b) => b.score - a.score)
    .slice(0, 3)
    .map(({ proc }) => proc);
}

Memory Retrieval at Query Time

Before each agent response, retrieve relevant memories:

async function loadRelevantMemories(
  query: string,
  userId: string
): Promise<{ episodic: string[]; semantic: string[] }> {
  const queryEmbedding = (await embedTexts([query]))[0];

  const [episodic, semantic] = await Promise.all([
    // Recent + relevant episodic memories
    vectorDB.search("episodic", queryEmbedding, {
      filter: { userId },
      limit: 3,
      scoreThreshold: 0.7,
    }),
    // High-confidence semantic facts
    vectorDB.search("semantic", queryEmbedding, {
      filter: { userId, confidence: { gte: 0.7 } },
      limit: 5,
      scoreThreshold: 0.75,
    }),
  ]);

  return {
    episodic: episodic.map((m) => `[Past experience]: ${m.payload.summary}`),
    semantic: semantic.map((m) => `[Known fact]: ${m.payload.content}`),
  };
}

Then inject into the system prompt:

function buildSystemWithMemory(
  baseSystem: string,
  memories: { episodic: string[]; semantic: string[] }
): string {
  const memoryBlock = [
    memories.semantic.length ? `## What I know about you:\n${memories.semantic.join("\n")}` : "",
    memories.episodic.length ? `## Relevant past context:\n${memories.episodic.join("\n")}` : "",
  ].filter(Boolean).join("\n\n");

  return memoryBlock ? `${baseSystem}\n\n${memoryBlock}` : baseSystem;
}

Memory Lifecycle: Forgetting

Without pruning, memory grows unbounded. Implement decay:

async function pruneMemories(userId: string, maxEpisodic = 100) {
  const all = await getEpisodicMemories(userId);

  if (all.length <= maxEpisodic) return;

  // Keep: recent memories + high-access memories
  // Delete: old, never-accessed memories with low confidence
  const toDelete = all
    .sort((a, b) => {
      const recencyScore = (Date.now() - a.timestamp) / 86400000; // days old
      return recencyScore - (a.accessCount ?? 0) * 10; // penalize old, reward accessed
    })
    .slice(maxEpisodic)
    .map((m) => m.id);

  await vectorDB.deleteMany("episodic", toDelete);
}

Run this as a daily background job. Also decay confidence scores on semantic memories that haven't been reinforced:

// Run weekly
async function decaySemanticConfidence(userId: string) {
  const stale = await getSemanticMemories(userId, { 
    lastUpdatedBefore: Date.now() - 30 * 86400000 // 30 days 
  });
  
  for (const m of stale) {
    if (m.confidence > 0.3) {
      await updateMemory(m.id, { confidence: m.confidence * 0.9 });
    }
  }
}

FAQ

How much does memory retrieval add to latency? Vector search on <1M memories: 5–15ms. Embedding the query: 20–50ms. Total: 25–65ms. Run in parallel with other setup work to hide latency.

Should I store raw conversation transcripts or summaries? Summaries. Raw transcripts grow too large and contain noise. Use LLM-generated summaries (like shown above) — they're 10× smaller and search better.

How do I handle memory conflicts? When two memories contradict, keep both with their confidence scores. Let the agent reason about the conflict. Update confidence based on which version turns out to be correct.

Is this GDPR/privacy compliant? Memory is personal data. Store by userId, implement deletion on request, and document retention periods. Don't store sensitive data (passwords, financial details) in semantic memory.

Can I use a regular database instead of a vector DB? For <1,000 memories per user, a regular DB with keyword search works. For larger scales or semantic retrieval, you need vector search. Use Qdrant or pgvector on Postgres.