Building AI Service Backends in Go: Performance Guide

Go is an underappreciated choice for AI backends. Its goroutine model handles parallel LLM calls elegantly, its HTTP performance is excellent for streaming responses, and its small memory footprint means you can run more inference workers on the same hardware. Here's how to build production AI services in Go.

Why Go for AI Backends

| Requirement | Go advantage | |-------------|-------------| | Parallel LLM calls | Goroutines + channels — spawn 50 concurrent calls trivially | | Streaming responses | io.Reader/io.Writer are first-class | | Connection pooling | http.Transport built-in, highly tunable | | Memory efficiency | 10–20MB base footprint vs. 100–300MB for Node.js/Python | | Deployment | Single static binary, no runtime dependencies |

Python is better for ML research; Go is better for production serving.

The Anthropic SDK in Go

There's no official Go SDK from Anthropic — use the HTTP API directly with a typed client:

package anthropic

import (
    "bytes"
    "context"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "time"
)

type Client struct {
    apiKey  string
    baseURL string
    http    *http.Client
}

func NewClient(apiKey string) *Client {
    return &Client{
        apiKey:  apiKey,
        baseURL: "https://api.anthropic.com/v1",
        http: &http.Client{
            Timeout: 120 * time.Second,
            Transport: &http.Transport{
                MaxIdleConns:        100,
                MaxIdleConnsPerHost: 20,
                IdleConnTimeout:     90 * time.Second,
            },
        },
    }
}

type MessageRequest struct {
    Model     string      `json:"model"`
    MaxTokens int         `json:"max_tokens"`
    System    string      `json:"system,omitempty"`
    Messages  []Message   `json:"messages"`
    Stream    bool        `json:"stream,omitempty"`
}

type Message struct {
    Role    string `json:"role"`
    Content string `json:"content"`
}

type MessageResponse struct {
    ID      string    `json:"id"`
    Content []Content `json:"content"`
    Usage   Usage     `json:"usage"`
    Model   string    `json:"model"`
}

type Content struct {
    Type string `json:"type"`
    Text string `json:"text"`
}

type Usage struct {
    InputTokens  int `json:"input_tokens"`
    OutputTokens int `json:"output_tokens"`
}

func (c *Client) CreateMessage(ctx context.Context, req MessageRequest) (*MessageResponse, error) {
    body, err := json.Marshal(req)
    if err != nil {
        return nil, fmt.Errorf("marshal: %w", err)
    }

    httpReq, err := http.NewRequestWithContext(ctx, "POST", c.baseURL+"/messages", bytes.NewReader(body))
    if err != nil {
        return nil, fmt.Errorf("new request: %w", err)
    }
    httpReq.Header.Set("Content-Type", "application/json")
    httpReq.Header.Set("x-api-key", c.apiKey)
    httpReq.Header.Set("anthropic-version", "2023-06-01")

    resp, err := c.http.Do(httpReq)
    if err != nil {
        return nil, fmt.Errorf("do request: %w", err)
    }
    defer resp.Body.Close()

    if resp.StatusCode != http.StatusOK {
        body, _ := io.ReadAll(resp.Body)
        return nil, fmt.Errorf("api error %d: %s", resp.StatusCode, body)
    }

    var result MessageResponse
    if err := json.NewDecoder(resp.Body).Decode(&result); err != nil {
        return nil, fmt.Errorf("decode: %w", err)
    }
    return &result, nil
}

Parallel LLM Calls with Goroutines

The killer feature: process multiple documents or queries concurrently.

func processDocumentsBatch(ctx context.Context, client *anthropic.Client, docs []string) ([]string, error) {
    type result struct {
        index   int
        summary string
        err     error
    }

    results := make([]result, len(docs))
    ch := make(chan result, len(docs))

    // Rate limit: max 10 concurrent calls
    sem := make(chan struct{}, 10)

    for i, doc := range docs {
        go func(idx int, document string) {
            sem <- struct{}{}        // acquire
            defer func() { <-sem }() // release

            resp, err := client.CreateMessage(ctx, anthropic.MessageRequest{
                Model:     "claude-haiku-4-5-20251001",
                MaxTokens: 200,
                Messages: []anthropic.Message{
                    {Role: "user", Content: "Summarize in 2 sentences: " + document},
                },
            })
            if err != nil {
                ch <- result{index: idx, err: err}
                return
            }
            ch <- result{index: idx, summary: resp.Content[0].Text}
        }(i, doc)
    }

    for range docs {
        r := <-ch
        if r.err != nil {
            return nil, fmt.Errorf("doc %d: %w", r.index, r.err)
        }
        results[r.index] = r
    }

    summaries := make([]string, len(docs))
    for i, r := range results {
        summaries[i] = r.summary
    }
    return summaries, nil
}

This processes all documents concurrently, bounded by the semaphore (10 concurrent API calls max, respecting Anthropic's rate limits).

Streaming Responses

SSE (Server-Sent Events) streaming for real-time output:

func (c *Client) StreamMessage(ctx context.Context, req anthropic.MessageRequest, w http.ResponseWriter) error {
    req.Stream = true
    body, _ := json.Marshal(req)

    httpReq, _ := http.NewRequestWithContext(ctx, "POST", c.baseURL+"/messages", bytes.NewReader(body))
    httpReq.Header.Set("Content-Type", "application/json")
    httpReq.Header.Set("x-api-key", c.apiKey)
    httpReq.Header.Set("anthropic-version", "2023-06-01")

    resp, err := c.http.Do(httpReq)
    if err != nil {
        return err
    }
    defer resp.Body.Close()

    // Set SSE headers
    w.Header().Set("Content-Type", "text/event-stream")
    w.Header().Set("Cache-Control", "no-cache")
    w.Header().Set("Connection", "keep-alive")
    flusher := w.(http.Flusher)

    scanner := bufio.NewScanner(resp.Body)
    for scanner.Scan() {
        line := scanner.Text()
        if !strings.HasPrefix(line, "data: ") {
            continue
        }
        data := strings.TrimPrefix(line, "data: ")
        if data == "[DONE]" {
            break
        }

        // Forward to client
        fmt.Fprintf(w, "data: %s\n\n", data)
        flusher.Flush()
    }
    return scanner.Err()
}

HTTP Handler with Rate Limiting

import (
    "golang.org/x/time/rate"
    "net/http"
)

type AIHandler struct {
    client  *anthropic.Client
    limiter *rate.Limiter
}

func NewAIHandler(client *anthropic.Client) *AIHandler {
    return &AIHandler{
        client:  client,
        limiter: rate.NewLimiter(rate.Limit(50), 100), // 50 req/s, burst 100
    }
}

func (h *AIHandler) Complete(w http.ResponseWriter, r *http.Request) {
    if !h.limiter.Allow() {
        http.Error(w, `{"error":"rate limit exceeded"}`, http.StatusTooManyRequests)
        return
    }

    var req struct {
        Messages []anthropic.Message `json:"messages"`
        Stream   bool                `json:"stream"`
    }
    if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
        http.Error(w, `{"error":"invalid request"}`, http.StatusBadRequest)
        return
    }

    if req.Stream {
        h.client.StreamMessage(r.Context(), anthropic.MessageRequest{
            Model:    "claude-sonnet-4-6",
            MaxTokens: 2048,
            Messages:  req.Messages,
        }, w)
        return
    }

    resp, err := h.client.CreateMessage(r.Context(), anthropic.MessageRequest{
        Model:    "claude-sonnet-4-6",
        MaxTokens: 2048,
        Messages:  req.Messages,
    })
    if err != nil {
        http.Error(w, `{"error":"llm error"}`, http.StatusInternalServerError)
        return
    }

    w.Header().Set("Content-Type", "application/json")
    json.NewEncoder(w).Encode(resp)
}

Caching with Redis

import "github.com/redis/go-redis/v9"

type CachedClient struct {
    inner  *anthropic.Client
    redis  *redis.Client
    ttl    time.Duration
}

func (c *CachedClient) CreateMessage(ctx context.Context, req anthropic.MessageRequest) (*anthropic.MessageResponse, error) {
    // Cache key from request hash
    key := "llm:" + hashRequest(req)

    cached, err := c.redis.Get(ctx, key).Result()
    if err == nil {
        var resp anthropic.MessageResponse
        json.Unmarshal([]byte(cached), &resp)
        return &resp, nil
    }

    resp, err := c.inner.CreateMessage(ctx, req)
    if err != nil {
        return nil, err
    }

    data, _ := json.Marshal(resp)
    c.redis.Set(ctx, key, data, c.ttl)
    return resp, nil
}

Structured Logging for AI Calls

Use slog (Go 1.21+) for structured JSON logs:

import "log/slog"

func logLLMCall(model string, usage anthropic.Usage, latency time.Duration, err error) {
    cost := calculateCost(model, usage)
    
    attrs := []slog.Attr{
        slog.String("model", model),
        slog.Int("input_tokens", usage.InputTokens),
        slog.Int("output_tokens", usage.OutputTokens),
        slog.Float64("cost_usd", cost),
        slog.Duration("latency", latency),
    }
    
    if err != nil {
        attrs = append(attrs, slog.String("error", err.Error()))
        slog.LogAttrs(context.Background(), slog.LevelError, "llm_call_failed", attrs...)
        return
    }
    slog.LogAttrs(context.Background(), slog.LevelInfo, "llm_call", attrs...)
}

See Scaling AI Inference with Go for connection pooling details and load balancing across multiple model endpoints.

FAQ

Is there an official Anthropic Go SDK? Not as of mid-2026. The community liushuangls/go-anthropic package is mature and widely used. Or build a thin wrapper like shown above for full control.

How do I handle context cancellation? Pass ctx to every LLM call. When the HTTP request is cancelled (user closes browser), ctx.Done() signals all goroutines to stop. This prevents runaway LLM calls from orphaned requests.

Should I use net/http or a framework like Echo/Fiber? For simple AI APIs, net/http with gorilla/mux or chi is fine. For complex routing, middleware stacks, or request validation, Echo or Fiber save boilerplate. Fiber's zero-allocation design gives a small edge for very high-throughput streaming.

How do I test LLM calls in Go? Interface out the LLM client and use mock implementations in tests. Test the non-LLM logic (routing, caching, formatting) exhaustively. For LLM output testing, record/replay fixtures.

What's the memory cost per goroutine? Initial goroutine stack: 2–8KB. 1,000 concurrent LLM calls ≈ 2–8MB goroutine overhead. Negligible compared to HTTP response buffers.