Prompt Versioning: Treat System Prompts Like Prod Code

System prompts are configuration that controls your product's behavior. When teams treat them as informal text they paste into a dashboard, they end up with: untested changes breaking user-facing features, no way to roll back, no ability to A/B test, and no history of what changed when. Treat system prompts like production code.

The Problem with Ad-Hoc Prompt Management

Week 1: Developer tweaks system prompt in prod dashboard
Week 2: Quality drops 15% — no one knows why
Week 3: Another change, quality comes back
Week 4: Marketing wants to test two versions
Result: chaos

This is avoidable with three things: version control, testing before deployment, and A/B infrastructure.

Prompt Version Schema

interface PromptVersion {
  id: string;             // auto-generated UUID
  name: string;           // human-readable "customer-support-v3"
  version: string;        // semver "1.2.3"
  content: string;        // the actual prompt
  model: string;          // "claude-sonnet-4-6"
  parameters: {
    maxTokens: number;
    temperature?: number;
  };
  author: string;
  createdAt: string;
  tags: string[];
  evalResults?: EvalSummary;
  status: "draft" | "testing" | "active" | "deprecated";
  successorId?: string;   // points to newer version if deprecated
}

interface EvalSummary {
  passRate: number;
  avgQualityScore: number;
  totalExamples: number;
  runAt: string;
  comparedTo?: string;    // version ID this was compared against
  improvement?: number;   // relative to compared version
}

Git-Based Prompt Repository

Store prompts as files in a git repo — same workflow as code:

prompts/
  customer-support/
    v1.0.0.md
    v1.1.0.md
    v2.0.0.md           ← current
    README.md           ← what this agent does
  code-review/
    v1.0.0.md
    v1.1.0.md           ← current
  CHANGELOG.md

Each prompt file:

---
version: "2.0.0"
model: "claude-sonnet-4-6"
maxTokens: 1024
tags: ["customer-support", "tier-1"]
---

You are a customer support agent for Shahriar Labs products.

## Your responsibilities
- Answer questions about LetX and QuantumSketch
- Help users troubleshoot common issues
- Escalate billing questions to human agents

## Rules
...

Commit messages explain why a prompt changed:

feat(customer-support): improve escalation trigger wording

Previous version was escalating too aggressively for basic billing questions.
Clarified the threshold: only escalate when user explicitly asks for refund.

Eval results: pass rate 91% → 94% on escalation test cases.

Prompt Registry Service

class PromptRegistry {
  private db: Database;

  async register(prompt: Omit<PromptVersion, "id" | "status">): Promise<string> {
    const id = crypto.randomUUID();
    await this.db.insert("prompts", { ...prompt, id, status: "draft" });
    return id;
  }

  async promote(id: string, newStatus: "testing" | "active"): Promise<void> {
    if (newStatus === "active") {
      // Deprecate previous active version
      await this.db.update("prompts",
        { status: "deprecated" },
        { name: await this.getPromptName(id), status: "active" }
      );
    }
    await this.db.update("prompts", { status: newStatus }, { id });
  }

  async getActive(name: string): Promise<PromptVersion> {
    return this.db.findOne("prompts", { name, status: "active" });
  }

  async rollback(name: string): Promise<void> {
    const active = await this.getActive(name);
    const previous = await this.db.findOne("prompts", {
      name,
      status: "deprecated",
      createdAt: { $lt: active.createdAt },
    }, { sort: { createdAt: -1 } });

    if (!previous) throw new Error("No previous version to roll back to");

    await this.promote(active.id, "deprecated" as any);
    await this.promote(previous.id, "active");

    console.log(`Rolled back ${name} from ${active.version} to ${previous.version}`);
  }
}

A/B Testing Infrastructure

interface PromptExperiment {
  id: string;
  name: string;
  promptA: string;       // version ID
  promptB: string;       // version ID
  trafficSplit: number;  // 0–1, fraction going to B
  status: "running" | "concluded";
  startedAt: number;
  concludedAt?: number;
  winner?: "a" | "b" | "inconclusive";
}

class PromptABTester {
  async assignVariant(
    userId: string,
    experiment: PromptExperiment
  ): Promise<"a" | "b"> {
    // Consistent assignment per user (not random per request)
    const hash = this.hash(`${userId}:${experiment.id}`);
    return hash < experiment.trafficSplit ? "b" : "a";
  }

  async getPromptForUser(
    userId: string,
    promptName: string
  ): Promise<PromptVersion> {
    const experiment = await this.getActiveExperiment(promptName);

    if (!experiment) {
      return promptRegistry.getActive(promptName);
    }

    const variant = await this.assignVariant(userId, experiment);
    const promptId = variant === "a" ? experiment.promptA : experiment.promptB;
    const prompt = await promptRegistry.getById(promptId);

    // Track assignment for analysis
    await this.recordAssignment(userId, experiment.id, variant);

    return prompt;
  }

  async analyzeExperiment(experimentId: string): Promise<ExperimentAnalysis> {
    const { aResults, bResults } = await this.getResults(experimentId);

    const aStats = computeStats(aResults.map((r) => r.qualityScore));
    const bStats = computeStats(bResults.map((r) => r.qualityScore));

    // Two-proportion z-test for statistical significance
    const significant = isStatisticallySignificant(aStats, bStats, aResults.length, bResults.length);

    return {
      variantA: { mean: aStats.mean, n: aResults.length },
      variantB: { mean: bStats.mean, n: bResults.length },
      uplift: (bStats.mean - aStats.mean) / aStats.mean,
      significant,
      winner: significant ? (bStats.mean > aStats.mean ? "b" : "a") : "inconclusive",
    };
  }

  private hash(input: string): number {
    let hash = 0;
    for (let i = 0; i < input.length; i++) {
      hash = ((hash << 5) - hash) + input.charCodeAt(i);
      hash = hash & hash;
    }
    return Math.abs(hash) / 2147483647;
  }
}

CI/CD Gate for Prompt Changes

# .github/workflows/prompt-eval.yml
name: Prompt Eval Gate

on:
  pull_request:
    paths:
      - "prompts/**"

jobs:
  eval:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm install
      - name: Run evals on changed prompts
        run: |
          npx ts-node scripts/eval-changed-prompts.ts
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
          REQUIRED_PASS_RATE: "0.90"
      - name: Comment eval results on PR
        uses: actions/github-script@v7
        with:
          script: |
            const results = JSON.parse(process.env.EVAL_RESULTS);
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              body: formatEvalResults(results)
            });

See Testing LLM Applications for the eval framework this CI step uses.

FAQ

Should prompts be in code or a database? Both: git for version history and review; database for serving active prompts at runtime. Sync database from git on deploy. Never edit the database directly — all changes go through git.

How do I handle secrets in prompts? Use template variables: {{COMPANY_NAME}} in the stored prompt, injected at runtime. Never hardcode internal URLs, keys, or confidential information in version-controlled prompts.

How long should I run an A/B test? Until you have statistical significance (p < 0.05) with at least 100 samples per variant — typically 3–7 days for moderate traffic apps.

What's a meaningful improvement to ship?

5% improvement in pass rate or quality score. Changes under 5% are in the noise; ship them only if they reduce prompt length (cost savings) or improve a specific failure mode.

When should a prompt change be a major vs. minor version bump? Major (2.0.0): changes the agent's scope or persona. Minor (1.1.0): improves quality without behavior change. Patch (1.0.1): fixes a specific failure case. Use this consistently for changelog clarity.