Scaling AI Inference with Go: Pools, Caching, Balancing
# High-throughput AI services in Go require connection pooling, request batching, in-process caching, and load balancing across model endpoints. Here are the patterns for 1000+ requests per second.
Go's concurrency primitives — goroutines, channels, sync.Pool — map perfectly to the parallelism patterns in AI inference. This guide covers the techniques that let a Go AI service handle 1,000+ concurrent LLM requests without falling over.
Connection Pool Architecture
Every LLM request is an HTTP/2 connection. Managing these connections efficiently is the foundation of high-throughput Go AI services.
package inference
import (
"crypto/tls"
"net"
"net/http"
"time"
)
func NewOptimizedHTTPClient() *http.Client {
return &http.Client{
Timeout: 120 * time.Second,
Transport: &http.Transport{
// Connection pool settings
MaxIdleConns: 200,
MaxIdleConnsPerHost: 50, // Per inference endpoint
MaxConnsPerHost: 100, // Hard limit per host
IdleConnTimeout: 90 * time.Second,
// Tune for LLM workloads (large responses)
WriteBufferSize: 32 * 1024, // 32KB write buffer
ReadBufferSize: 256 * 1024, // 256KB read buffer (for long completions)
// TCP tuning
DialContext: (&net.Dialer{
Timeout: 10 * time.Second,
KeepAlive: 30 * time.Second,
}).DialContext,
// Enable HTTP/2 for multiplexing
TLSClientConfig: &tls.Config{MinVersion: tls.VersionTLS12},
ForceAttemptHTTP2: true,
},
}
}
Request Batching with Dynamic Windows
Instead of sending requests one by one, batch them into windows to maximize throughput:
type BatchRequest struct {
Prompt string
MaxTokens int
Response chan BatchResult
}
type BatchResult struct {
Text string
Err error
}
type DynamicBatcher struct {
queue chan *BatchRequest
maxBatch int
maxWaitMs int
client *http.Client
}
func NewDynamicBatcher(maxBatch, maxWaitMs int) *DynamicBatcher {
b := &DynamicBatcher{
queue: make(chan *BatchRequest, 10000),
maxBatch: maxBatch,
maxWaitMs: maxWaitMs,
client: NewOptimizedHTTPClient(),
}
go b.processLoop()
return b
}
func (b *DynamicBatcher) processLoop() {
for {
// Collect a batch
batch := make([]*BatchRequest, 0, b.maxBatch)
timer := time.NewTimer(time.Duration(b.maxWaitMs) * time.Millisecond)
// Wait for first item
select {
case req := <-b.queue:
batch = append(batch, req)
case <-timer.C:
continue
}
timer.Stop()
// Fill batch within window
deadline := time.Now().Add(time.Duration(b.maxWaitMs) * time.Millisecond)
for len(batch) < b.maxBatch {
remaining := time.Until(deadline)
if remaining <= 0 {
break
}
select {
case req := <-b.queue:
batch = append(batch, req)
case <-time.After(remaining):
goto process
}
}
process:
go b.processBatch(batch)
}
}
func (b *DynamicBatcher) processBatch(batch []*BatchRequest) {
// Send all requests concurrently
var wg sync.WaitGroup
for _, req := range batch {
wg.Add(1)
go func(r *BatchRequest) {
defer wg.Done()
result, err := b.callLLM(r.Prompt, r.MaxTokens)
r.Response <- BatchResult{Text: result, Err: err}
}(req)
}
wg.Wait()
}
// Submit a request to the batcher
func (b *DynamicBatcher) Submit(prompt string, maxTokens int) (string, error) {
req := &BatchRequest{
Prompt: prompt,
MaxTokens: maxTokens,
Response: make(chan BatchResult, 1),
}
b.queue <- req
result := <-req.Response
return result.Text, result.Err
}
In-Process Response Cache
LRU cache for identical or semantically similar requests:
import "github.com/hashicorp/golang-lru/v2"
type ResponseCache struct {
exact *lru.Cache[string, string] // exact match cache
mu sync.RWMutex
}
func NewResponseCache(capacity int) *ResponseCache {
cache, _ := lru.New[string, string](capacity)
return &ResponseCache{exact: cache}
}
func (c *ResponseCache) Get(prompt, systemPrompt string) (string, bool) {
key := hashKey(systemPrompt + "|" + prompt)
return c.exact.Get(key)
}
func (c *ResponseCache) Set(prompt, systemPrompt, response string) {
key := hashKey(systemPrompt + "|" + prompt)
c.exact.Add(key, response)
}
func hashKey(s string) string {
h := sha256.Sum256([]byte(s))
return hex.EncodeToString(h[:])
}
// Integration in the inference service
type CachedInferenceService struct {
cache *ResponseCache
batcher *DynamicBatcher
metrics *InferenceMetrics
}
func (s *CachedInferenceService) Complete(ctx context.Context, prompt, system string) (string, error) {
// Check cache
if cached, ok := s.cache.Get(prompt, system); ok {
s.metrics.cacheHits.Add(1)
return cached, nil
}
// Cache miss → call model
start := time.Now()
response, err := s.batcher.Submit(prompt, 2048)
if err != nil {
return "", err
}
s.metrics.recordLatency(time.Since(start))
s.cache.Set(prompt, system, response)
return response, nil
}
Load Balancing Across Multiple Endpoints
Spread load across multiple API keys or model endpoints:
type LoadBalancer struct {
endpoints []Endpoint
mu sync.Mutex
current int // round-robin index
health []bool // endpoint health
}
type Endpoint struct {
URL string
APIKey string
Client *http.Client
// Track inflight requests for least-connections routing
inflight atomic.Int64
}
func NewLoadBalancer(endpoints []Endpoint) *LoadBalancer {
lb := &LoadBalancer{
endpoints: endpoints,
health: make([]bool, len(endpoints)),
}
// Initialize all as healthy
for i := range lb.health {
lb.health[i] = true
}
go lb.healthCheckLoop()
return lb
}
// Least-connections routing (better for variable LLM response times)
func (lb *LoadBalancer) GetEndpoint() *Endpoint {
lb.mu.Lock()
defer lb.mu.Unlock()
var best *Endpoint
var bestCount int64 = math.MaxInt64
for i, ep := range lb.endpoints {
if !lb.health[i] {
continue
}
count := ep.inflight.Load()
if count < bestCount {
bestCount = count
best = &lb.endpoints[i]
}
}
return best
}
func (lb *LoadBalancer) healthCheckLoop() {
ticker := time.NewTicker(30 * time.Second)
for range ticker.C {
for i, ep := range lb.endpoints {
healthy := lb.checkHealth(ep)
lb.mu.Lock()
lb.health[i] = healthy
lb.mu.Unlock()
}
}
}
// Use in inference: track inflight
func (s *InferenceService) callWithLoadBalancing(prompt string) (string, error) {
ep := s.lb.GetEndpoint()
if ep == nil {
return "", errors.New("no healthy endpoints")
}
ep.inflight.Add(1)
defer ep.inflight.Add(-1)
return s.callEndpoint(ep, prompt)
}
Concurrency Limiter
Prevent overwhelming the API with too many concurrent requests:
type SemaphorePool struct {
sem chan struct{}
}
func NewSemaphorePool(maxConcurrent int) *SemaphorePool {
sem := make(chan struct{}, maxConcurrent)
for i := 0; i < maxConcurrent; i++ {
sem <- struct{}{}
}
return &SemaphorePool{sem: sem}
}
func (p *SemaphorePool) Acquire(ctx context.Context) error {
select {
case <-p.sem:
return nil
case <-ctx.Done():
return ctx.Err()
}
}
func (p *SemaphorePool) Release() {
p.sem <- struct{}{}
}
// Usage
pool := NewSemaphorePool(50) // max 50 concurrent LLM calls
func (s *Service) Complete(ctx context.Context, prompt string) (string, error) {
if err := pool.Acquire(ctx); err != nil {
return "", fmt.Errorf("queue full: %w", err)
}
defer pool.Release()
return s.callLLM(ctx, prompt)
}
Metrics Collection
type InferenceMetrics struct {
requestsTotal atomic.Int64
cacheHits atomic.Int64
errors atomic.Int64
latencyBuckets [10]atomic.Int64 // 100ms buckets
}
func (m *InferenceMetrics) recordLatency(d time.Duration) {
bucket := int(d.Milliseconds() / 100)
if bucket >= len(m.latencyBuckets) {
bucket = len(m.latencyBuckets) - 1
}
m.latencyBuckets[bucket].Add(1)
}
func (m *InferenceMetrics) Report() map[string]interface{} {
total := m.requestsTotal.Load()
hits := m.cacheHits.Load()
return map[string]interface{}{
"requests_total": total,
"cache_hit_rate": float64(hits) / float64(total),
"error_rate": float64(m.errors.Load()) / float64(total),
"p50_latency_ms": m.percentileLatency(0.5),
"p99_latency_ms": m.percentileLatency(0.99),
}
}
Performance Benchmarks
On a 4-core machine with a single API key (Anthropic rate limits ~500 RPM):
| Configuration | Throughput | p99 Latency | |--------------|------------|-------------| | Naive sequential | 60 req/min | 2.1s | | Goroutine per request | 450 req/min | 4.2s | | Batched + connection pool | 490 req/min | 1.8s | | Multi-key load balanced | 1,500+ req/min | 1.9s |
See Building AI Backends in Go for the foundational patterns this builds on.
FAQ
How many goroutines should I use for concurrent LLM calls? Don't bound by goroutine count — bound by API rate limit. 10-50 concurrent requests per API key is typical. More than that triggers 429s. Use the semaphore pool pattern to enforce this limit.
Is sync.Pool useful for LLM request objects? Yes — if you're creating many request structs per second, sync.Pool reduces GC pressure. Profile first — premature optimization applies here.
How do I handle streaming with load balancing? Sticky connections for streaming: once a streaming request starts on an endpoint, keep it there until completion. Track streaming requests separately from regular requests in your inflight counters.
What's the right batch window? 10-50ms. Shorter = lower latency, smaller batches. Longer = higher throughput, higher latency. For user-facing apps: 10ms. For background processing: 50-100ms.
Does connection pooling help with rate limiting? No — rate limiting is API-side, not connection-side. Connection pools reduce connection establishment latency (TCP handshake, TLS negotiation). Rate limiting requires separate token bucket logic.