[SL_CORE]
< CD ..||6 min read

Microservices for AI: Agent Service Boundaries & Comms

# AI agents map naturally to microservices: each agent is a service with a specific capability. Learn how to define service boundaries, handle communication, and manage failures in distributed AI architectures.

[microservices][ai-agents][architecture][go]

Multi-agent systems and microservices share the same core challenge: distributed systems where each component has a clear responsibility, communicates over a network, and can fail independently. The patterns that make microservices reliable — service contracts, circuit breakers, health checks — apply directly to AI agent architectures.

Agent as Microservice

Each AI agent should be a standalone service with:

  • A single, well-defined responsibility
  • A documented API (request schema + response schema)
  • Independent deployability
  • Its own scaling and resource profile
┌────────────────────────────────────────────────┐
│ API Gateway / Orchestrator                      │
├──────────────┬───────────────┬─────────────────┤
│ Research     │ Code Review   │ Summarization   │
│ Agent (Go)   │ Agent (Go)    │ Agent (Go)      │
│ Port 8081    │ Port 8082     │ Port 8083       │
└──────────────┴───────────────┴─────────────────┘

Service Contract (gRPC + Protobuf)

Define agent APIs with protobuf for type safety across services:

// proto/agents/v1/agents.proto
syntax = "proto3";
package agents.v1;

service ResearchAgent {
  rpc Research(ResearchRequest) returns (ResearchResponse);
  rpc ResearchStream(ResearchRequest) returns (stream ResearchProgress);
}

message ResearchRequest {
  string query = 1;
  int32 max_sources = 2;
  double max_cost_usd = 3;
  int64 deadline_unix_ms = 4;
}

message ResearchResponse {
  string summary = 1;
  repeated Source sources = 2;
  float confidence = 3;
  float actual_cost_usd = 4;
}

message Source {
  string url = 1;
  string title = 2;
  string relevant_excerpt = 3;
}

message ResearchProgress {
  int32 step = 1;
  int32 total_steps = 2;
  string current_action = 3;
  float cost_so_far = 4;
}

Go Agent Service Implementation

package main

import (
    "context"
    "net"
    "google.golang.org/grpc"
    pb "github.com/shahriarlabs/agents/proto/agents/v1"
)

type ResearchAgentServer struct {
    pb.UnimplementedResearchAgentServer
    llmClient    *anthropic.Client
    searchClient *SearchClient
}

func (s *ResearchAgentServer) Research(ctx context.Context, req *pb.ResearchRequest) (*pb.ResearchResponse, error) {
    // Cost budget enforcement
    costTracker := NewCostTracker(float64(req.MaxCostUsd))
    
    // Execute research with tool-using agent
    result, err := s.runResearchAgent(ctx, req.Query, req.MaxSources, costTracker)
    if err != nil {
        return nil, status.Errorf(codes.Internal, "research failed: %v", err)
    }

    return &pb.ResearchResponse{
        Summary:       result.Summary,
        Sources:       result.Sources,
        Confidence:    float32(result.Confidence),
        ActualCostUsd: float32(costTracker.TotalCost()),
    }, nil
}

func main() {
    lis, err := net.Listen("tcp", ":8081")
    if err != nil {
        log.Fatalf("failed to listen: %v", err)
    }
    
    s := grpc.NewServer(
        grpc.UnaryInterceptor(metricsInterceptor),
        grpc.StreamInterceptor(streamMetricsInterceptor),
    )
    pb.RegisterResearchAgentServer(s, &ResearchAgentServer{})
    s.Serve(lis)
}

HTTP/REST Alternative

For simpler setups or when gRPC adds too much overhead:

// agent-service/src/server.ts
import express from "express";
import { z } from "zod";

const app = express();
app.use(express.json());

const ResearchRequestSchema = z.object({
  query: z.string().min(1).max(1000),
  maxSources: z.number().min(1).max(20).default(5),
  maxCostUSD: z.number().positive().default(0.10),
});

app.post("/research", async (req, res) => {
  const parseResult = ResearchRequestSchema.safeParse(req.body);
  if (!parseResult.success) {
    return res.status(400).json({ error: parseResult.error.flatten() });
  }

  try {
    const result = await runResearchAgent(parseResult.data);
    res.json(result);
  } catch (err: any) {
    if (err.type === "cost_exceeded") {
      return res.status(402).json({ error: "Cost budget exceeded", spentUSD: err.spent });
    }
    res.status(500).json({ error: "Research failed", message: err.message });
  }
});

// Health check for load balancer
app.get("/health", async (req, res) => {
  const llmOk = await checkLLMConnectivity();
  res.status(llmOk ? 200 : 503).json({ status: llmOk ? "healthy" : "degraded" });
});

app.listen(8081);

Circuit Breaker Pattern

Prevent cascading failures when an agent service is degraded:

type CircuitState = "closed" | "open" | "half-open";

class CircuitBreaker {
  private state: CircuitState = "closed";
  private failures = 0;
  private lastFailureTime = 0;

  constructor(
    private readonly maxFailures = 5,
    private readonly resetTimeMs = 30_000
  ) {}

  async execute<T>(fn: () => Promise<T>): Promise<T> {
    if (this.state === "open") {
      if (Date.now() - this.lastFailureTime > this.resetTimeMs) {
        this.state = "half-open";
      } else {
        throw new Error("Circuit breaker OPEN — agent unavailable");
      }
    }

    try {
      const result = await fn();
      if (this.state === "half-open") {
        this.state = "closed";
        this.failures = 0;
      }
      return result;
    } catch (err) {
      this.failures++;
      this.lastFailureTime = Date.now();
      if (this.failures >= this.maxFailures) {
        this.state = "open";
      }
      throw err;
    }
  }

  getState(): CircuitState { return this.state; }
}

// Usage in orchestrator
const researchBreaker = new CircuitBreaker(5, 30_000);

async function callResearchAgent(query: string) {
  return researchBreaker.execute(() => researchAgentClient.research(query));
}

Service Discovery and Load Balancing

For multiple instances of the same agent:

class AgentRegistry {
  private instances: Map<string, string[]> = new Map(); // agentType → URLs

  register(agentType: string, url: string) {
    const urls = this.instances.get(agentType) ?? [];
    if (!urls.includes(url)) {
      this.instances.set(agentType, [...urls, url]);
    }
  }

  getEndpoint(agentType: string, strategy: "round-robin" | "random" = "round-robin"): string {
    const urls = this.instances.get(agentType) ?? [];
    if (urls.length === 0) throw new Error(`No instances of ${agentType} registered`);

    if (strategy === "random") return urls[Math.floor(Math.random() * urls.length)];

    // Round-robin with counter
    const idx = this.getRoundRobinIndex(agentType, urls.length);
    return urls[idx];
  }
}

Inter-Service Authentication

Each agent service should authenticate callers:

// Shared auth middleware
function agentAuthMiddleware(req: express.Request, res: express.Response, next: express.NextFunction) {
  const serviceToken = req.headers["x-service-token"];
  const callerName = req.headers["x-caller-agent"];

  if (!serviceToken || serviceToken !== process.env.INTER_SERVICE_TOKEN) {
    return res.status(401).json({ error: "Unauthorized" });
  }

  // Log which agent is calling this service
  req.callerAgent = String(callerName ?? "unknown");
  next();
}

app.use(agentAuthMiddleware);

Scaling Considerations

| Agent type | Scaling strategy | |-----------|-----------------| | CPU-intensive analysis | Horizontal (more instances) | | LLM-heavy (rate limited) | Queue + single worker per API key | | Memory-heavy (RAG) | Vertical + read replicas | | Streaming responses | Stateful, sticky sessions |

The key insight: LLM rate limits make horizontal scaling different from traditional services. Instead of adding more instances, add more API keys and route across them.

FAQ

Should each agent have its own database? Ideally yes (microservices data isolation). Practically, sharing a Postgres instance with schema-level isolation is fine for smaller deployments. Never share connection pools — each service gets its own.

gRPC vs REST for agent services? gRPC for: internal services, streaming, generated client code. REST for: external APIs, simpler debugging, non-Go/non-Java services. Most AI agent services are internal — gRPC's type safety and streaming support are worth the complexity.

How do I handle agent service versioning? Version APIs semantically: /v1/research, /v2/research. Maintain backward compatibility within major versions. Use feature flags to roll out breaking changes gradually. See DevOps for AI Backends.

What's the recommended timeout for agent calls? Base it on the task: quick tasks (< 5s), long tasks (30–120s). Always set timeouts — unbounded waits cause cascading failures. For very long tasks (>120s), use the async pattern instead. See Async AI.

How do I share context between agent services? Pass context in request headers or body — don't rely on shared state. If agents need shared memory, use a shared Redis cache with explicit TTLs. Stateless services are easier to scale and debug.