DevOps for AI Backends: CI/CD and Safe Model Rollouts
# Deploying AI backends requires different DevOps practices: model version pinning, gradual rollouts, cost budget gates, and rollback strategies that account for non-deterministic behavior.
Standard CI/CD treats deployment as binary: tests pass → deploy. AI backends have a third dimension: quality. A deployment can pass all unit tests and still degrade user experience if the underlying model behavior changed. DevOps for AI adds eval gates, model version pinning, and cost-aware deployment strategies.
Model Version Pinning
Never use floating model aliases in production code:
// Wrong: "claude-sonnet-latest" changes behavior without you knowing
const model = "claude-sonnet-latest";
// Right: pin the exact model version
const model = "claude-sonnet-4-6-20251120"; // or whatever the pinned version is
// Better: configure via environment variable for staged rollouts
const model = process.env.LLM_MODEL ?? "claude-sonnet-4-6-20251120";
Pin model versions the same way you pin npm packages. Update deliberately, test before rollout, have a rollback plan.
Environment-Specific Model Config
// config/ai.ts
const modelConfig: Record<string, { model: string; maxTokens: number }> = {
development: {
model: "claude-haiku-4-5-20251001", // cheaper for local dev
maxTokens: 1024,
},
staging: {
model: "claude-sonnet-4-6-20251120", // test with prod model
maxTokens: 2048,
},
production: {
model: process.env.LLM_MODEL_PRODUCTION ?? "claude-sonnet-4-6-20251120",
maxTokens: 4096,
},
};
export const getModelConfig = () => modelConfig[process.env.NODE_ENV ?? "development"];
Stage model upgrades: develop → staging → canary (5% of prod) → full prod.
CI/CD Pipeline with Eval Gate
# .github/workflows/deploy.yml
name: Deploy AI Backend
on:
push:
branches: [main]
jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with: { node-version: "22" }
- run: npm ci
- run: npm test
eval-gate:
needs: test
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- run: npm ci
- name: Run LLM Evals
id: eval
run: |
RESULT=$(npx ts-node scripts/run-evals.ts --format=json)
echo "eval_result=$RESULT" >> $GITHUB_OUTPUT
# Parse pass rate
PASS_RATE=$(echo $RESULT | jq '.passRate')
echo "pass_rate=$PASS_RATE" >> $GITHUB_OUTPUT
if (( $(echo "$PASS_RATE < 0.88" | bc -l) )); then
echo "Eval failed: pass rate $PASS_RATE < 0.88"
exit 1
fi
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
- name: Post eval results
uses: actions/github-script@v7
if: always()
with:
script: |
const result = JSON.parse('${{ steps.eval.outputs.eval_result }}');
github.rest.issues.createComment({
issue_number: context.issue?.number,
body: `## LLM Eval Results\n\n| Category | Pass Rate |\n|----------|----------|\n${Object.entries(result.byCategory).map(([k,v]) => `| ${k} | ${(v*100).toFixed(1)}% |`).join('\n')}\n\n**Overall: ${(result.passRate*100).toFixed(1)}%**`
});
deploy:
needs: [test, eval-gate]
runs-on: ubuntu-latest
environment: production
steps:
- uses: actions/checkout@v4
- name: Deploy to Cloudflare
run: npx wrangler deploy
env:
CLOUDFLARE_API_TOKEN: ${{ secrets.CF_API_TOKEN }}
Cost Budget Gate
Prevent expensive deployments from accidentally 10× your monthly bill:
// scripts/cost-estimate.ts — runs in CI before deploy
async function estimateDeploymentCost(): Promise<{ estimatedDailyCost: number; acceptable: boolean }> {
// Run 100 representative requests against the new version
const sample = await loadRepresentativeSample(100);
const costs: number[] = [];
for (const request of sample) {
const response = await callNewModel(request.input);
costs.push(calculateCost(response.model, response.usage));
}
const avgCostPerRequest = costs.reduce((a, b) => a + b, 0) / costs.length;
const dailyRequests = await getDailyRequestVolume();
const estimatedDailyCost = avgCostPerRequest * dailyRequests;
const DAILY_BUDGET = parseFloat(process.env.DAILY_COST_BUDGET ?? "200");
return {
estimatedDailyCost,
acceptable: estimatedDailyCost < DAILY_BUDGET,
};
}
Canary Deployments for Model Upgrades
When upgrading model versions, route a small percentage to the new version:
// Feature flag for model version
async function getModelForRequest(userId: string): Promise<string> {
const canaryPercentage = parseInt(process.env.CANARY_MODEL_PERCENTAGE ?? "0");
if (canaryPercentage > 0) {
const hash = hashUserId(userId) % 100;
if (hash < canaryPercentage) {
return process.env.CANARY_MODEL ?? "claude-sonnet-4-6-20260101";
}
}
return process.env.PRODUCTION_MODEL ?? "claude-sonnet-4-6-20251120";
}
// Ramp up gradually:
// Day 1: CANARY_MODEL_PERCENTAGE=5 → 5% of users
// Day 3: CANARY_MODEL_PERCENTAGE=25 → 25% of users (if metrics look good)
// Day 5: CANARY_MODEL_PERCENTAGE=100 → full rollout
Rollback Strategy
When evals fail or quality drops post-deploy:
#!/bin/bash
# scripts/rollback.sh — run when quality alerts fire
echo "Rolling back LLM model version..."
# 1. Immediately switch back to previous model
wrangler secret put LLM_MODEL <<< "$PREVIOUS_MODEL_VERSION"
# 2. Notify team
curl -X POST "$SLACK_WEBHOOK" \
-d '{"text":"⚠️ LLM model rolled back to '"$PREVIOUS_MODEL_VERSION"' — quality alert triggered"}'
# 3. Reset canary percentage
wrangler secret put CANARY_MODEL_PERCENTAGE <<< "0"
echo "Rollback complete. Monitor quality metrics for next 30 minutes."
Health Check Endpoint
// app/api/health/route.ts
export async function GET() {
const checks = await Promise.allSettled([
checkLLMConnectivity(),
checkVectorDBConnectivity(),
checkDatabaseConnectivity(),
checkCacheBudget(),
]);
const health = {
status: checks.every((c) => c.status === "fulfilled") ? "healthy" : "degraded",
timestamp: new Date().toISOString(),
checks: {
llm: checks[0].status === "fulfilled" ? "ok" : "error",
vectordb: checks[1].status === "fulfilled" ? "ok" : "error",
database: checks[2].status === "fulfilled" ? "ok" : "error",
costBudget: checks[3].status === "fulfilled"
? (checks[3] as PromiseFulfilledResult<any>).value
: "error",
},
version: process.env.DEPLOY_VERSION ?? "unknown",
};
return Response.json(health, {
status: health.status === "healthy" ? 200 : 503,
});
}
async function checkLLMConnectivity(): Promise<void> {
const start = Date.now();
await anthropic.messages.create({
model: "claude-haiku-4-5-20251001",
max_tokens: 5,
messages: [{ role: "user", content: "ping" }],
});
const latencyMs = Date.now() - start;
if (latencyMs > 5000) throw new Error(`LLM latency too high: ${latencyMs}ms`);
}
Monitoring Deployment Impact
-- Compare quality metrics before and after deployment
SELECT
CASE WHEN deployed_at > '2026-06-01T14:00:00Z' THEN 'after' ELSE 'before' END as deployment,
COUNT(*) as requests,
AVG(quality_score) as avg_quality,
AVG(cost_usd) as avg_cost,
AVG(latency_ms) as avg_latency,
SUM(CASE WHEN status = 'failed' THEN 1 ELSE 0 END)::float / COUNT(*) as error_rate
FROM agent_traces
WHERE deployed_at BETWEEN '2026-06-01T12:00:00Z' AND '2026-06-01T16:00:00Z'
GROUP BY deployment;
See Monitoring Agentic AI for the full observability stack this integrates with.
FAQ
Should model upgrades go through the same PR process as code changes? Yes — create a PR that changes the pinned model version, run evals as CI checks, get review, then merge. Treat it with the same rigor as any production config change.
How do I test against the new model in CI without paying for full eval runs? Keep a "smoke test" eval set of 20 critical cases that runs on every PR. Reserve the full 200-case suite for pre-deploy validation. The smoke test catches regressions cheaply.
What's the right canary percentage to start with? 5% for high-risk changes (major model version, behavior changes). 25% for low-risk changes (minor prompt tweaks, patch model versions). Full rollout in 3–7 days if no issues.
How do I handle model deprecations from the provider? Set calendar reminders when you pin a model version — check the provider's deprecation timeline. Schedule migration PRs 1 month before deprecation. Never wait until the last day.
Should I use feature flags or env vars for model routing? Env vars for server-side model selection (simpler, no SDK needed). Feature flags (LaunchDarkly, Unleash) if you need per-user control or gradual rollouts with automatic rollback based on metrics.