Rate Limiting LLM APIs: Quota Management in Production

Anthropic, OpenAI, and Google all throttle API usage. Hit a rate limit in production and users see spinning loaders or errors. The fix isn't just exponential backoff — it's proactive quota management: tracking usage, staying under limits, and routing around them when they're hit.

Understanding LLM Rate Limits

Most providers enforce two limits simultaneously:

| Limit type | What it measures | Typical values (paid tier) | |-----------|-----------------|--------------------------| | RPM | Requests per minute | 500–5,000 | | TPM | Tokens per minute (input + output) | 100K–2M | | TPDAY | Tokens per day | varies by plan |

You can hit TPM limits while well under RPM limits (long prompts). Always track both.

Rate Limit Headers

Providers return remaining quota in response headers. Read them:

interface RateLimitInfo {
  requestsLimit: number;
  requestsRemaining: number;
  requestsReset: Date;
  tokensLimit: number;
  tokensRemaining: number;
  tokensReset: Date;
}

function parseAnthropicRateLimits(headers: Headers): RateLimitInfo {
  return {
    requestsLimit:     parseInt(headers.get("anthropic-ratelimit-requests-limit") ?? "0"),
    requestsRemaining: parseInt(headers.get("anthropic-ratelimit-requests-remaining") ?? "0"),
    requestsReset:     new Date(headers.get("anthropic-ratelimit-requests-reset") ?? ""),
    tokensLimit:       parseInt(headers.get("anthropic-ratelimit-tokens-limit") ?? "0"),
    tokensRemaining:   parseInt(headers.get("anthropic-ratelimit-tokens-remaining") ?? "0"),
    tokensReset:       new Date(headers.get("anthropic-ratelimit-tokens-reset") ?? ""),
  };
}

Track remaining quota proactively — don't wait for a 429 to know you're close to the limit.

Token Budget Tracker

Implement a client-side token budget:

import { createClient } from "redis";

class TokenBudgetTracker {
  private redis: ReturnType<typeof createClient>;
  private readonly RPM_LIMIT: number;
  private readonly TPM_LIMIT: number;

  constructor(rpmLimit: number, tpmLimit: number) {
    this.RPM_LIMIT = rpmLimit;
    this.TPM_LIMIT = tpmLimit;
    this.redis = createClient({ url: process.env.REDIS_URL });
  }

  async canSend(estimatedTokens: number): Promise<{
    allowed: boolean;
    waitMs: number;
  }> {
    const now = Date.now();
    const windowKey = Math.floor(now / 60000); // 1-minute window

    const [requestCount, tokenCount] = await Promise.all([
      this.redis.incr(`rpm:${windowKey}`),
      this.redis.incrBy(`tpm:${windowKey}`, estimatedTokens),
    ]);

    // Set expiry on first access
    if (requestCount === 1) await this.redis.expire(`rpm:${windowKey}`, 120);
    if (tokenCount === estimatedTokens) await this.redis.expire(`tpm:${windowKey}`, 120);

    const rpmExceeded = requestCount > this.RPM_LIMIT;
    const tpmExceeded = tokenCount > this.TPM_LIMIT;

    if (!rpmExceeded && !tpmExceeded) {
      return { allowed: true, waitMs: 0 };
    }

    // Calculate wait time until window resets
    const windowEndMs = (windowKey + 1) * 60000;
    const waitMs = windowEndMs - now + 1000; // +1s buffer

    return { allowed: false, waitMs };
  }

  async recordActualUsage(inputTokens: number, outputTokens: number) {
    const windowKey = Math.floor(Date.now() / 60000);
    // Correct the estimate with actual usage
    await this.redis.incrBy(`tpm:${windowKey}`, outputTokens);
  }
}

Request Queue with Priority

Queue requests when approaching limits, prioritize by user tier:

import PQueue from "p-queue";

interface QueuedRequest {
  id: string;
  priority: number;  // higher = sooner
  estimatedTokens: number;
  execute: () => Promise<unknown>;
  resolve: (value: unknown) => void;
  reject: (error: unknown) => void;
}

class RateLimitedLLMQueue {
  private queue: PQueue;
  private budgetTracker: TokenBudgetTracker;

  constructor(concurrency = 5) {
    this.queue = new PQueue({ concurrency });
    this.budgetTracker = new TokenBudgetTracker(500, 100_000);
  }

  async enqueue<T>(
    execute: () => Promise<T>,
    options: { priority?: number; estimatedTokens?: number } = {}
  ): Promise<T> {
    const estimatedTokens = options.estimatedTokens ?? 2000;
    const priority = options.priority ?? 5;

    return this.queue.add(async () => {
      // Check budget before executing
      const { allowed, waitMs } = await this.budgetTracker.canSend(estimatedTokens);
      if (!allowed) {
        await sleep(waitMs);
      }
      return execute();
    }, { priority }) as Promise<T>;
  }
}

// Usage: enterprise requests get priority 10, free users get priority 1
const llmQueue = new RateLimitedLLMQueue();

// Enterprise request
const result = await llmQueue.enqueue(
  () => callLLM(prompt),
  { priority: 10, estimatedTokens: 3000 }
);

Exponential Backoff with Jitter

When you do hit a 429, back off intelligently:

async function withRetry<T>(
  fn: () => Promise<T>,
  maxRetries = 4,
  baseDelayMs = 1000
): Promise<T> {
  let lastError: Error;

  for (let attempt = 0; attempt <= maxRetries; attempt++) {
    try {
      return await fn();
    } catch (err: any) {
      lastError = err;

      if (err.status !== 429 && err.status !== 529) {
        throw err; // non-retriable
      }

      if (attempt === maxRetries) break;

      // Check Retry-After header
      const retryAfterMs = err.headers?.get("retry-after")
        ? parseInt(err.headers.get("retry-after")) * 1000
        : undefined;

      // Exponential backoff with full jitter
      const exponentialMs = baseDelayMs * Math.pow(2, attempt);
      const jitterMs = Math.random() * exponentialMs;
      const delayMs = retryAfterMs ?? (exponentialMs + jitterMs);

      console.warn(`Rate limited (attempt ${attempt + 1}), waiting ${Math.round(delayMs)}ms`);
      await sleep(delayMs);
    }
  }

  throw lastError!;
}

Full jitter (not just Math.random() * baseDelay) prevents thundering herd — all clients backing off simultaneously and then hammering the API at the same time.

Multi-Provider Fallback for High Availability

When Anthropic is rate-limited, fall back to OpenAI:

type Provider = "anthropic" | "openai" | "google";

class MultiProviderClient {
  private providerStatus: Map<Provider, { available: boolean; resetAt?: Date }> = new Map([
    ["anthropic", { available: true }],
    ["openai", { available: true }],
    ["google", { available: true }],
  ]);

  private readonly PROVIDER_ORDER: Provider[] = ["anthropic", "openai", "google"];

  async complete(prompt: string, options: { preferProvider?: Provider } = {}): Promise<string> {
    const order = options.preferProvider
      ? [options.preferProvider, ...this.PROVIDER_ORDER.filter((p) => p !== options.preferProvider)]
      : this.PROVIDER_ORDER;

    for (const provider of order) {
      const status = this.providerStatus.get(provider)!;
      if (!status.available && status.resetAt && status.resetAt > new Date()) {
        continue; // skip unavailable providers
      }

      try {
        const result = await this.callProvider(provider, prompt);
        this.providerStatus.set(provider, { available: true });
        return result;
      } catch (err: any) {
        if (err.status === 429) {
          const resetAt = new Date(Date.now() + 60_000); // try again in 60s
          this.providerStatus.set(provider, { available: false, resetAt });
          console.warn(`Provider ${provider} rate limited, trying next`);
          continue;
        }
        throw err;
      }
    }

    throw new Error("All providers rate limited");
  }

  private async callProvider(provider: Provider, prompt: string): Promise<string> {
    switch (provider) {
      case "anthropic": return callAnthropic(prompt);
      case "openai": return callOpenAI(prompt);
      case "google": return callGemini(prompt);
    }
  }
}

See Model Routing in Production for routing by quality, not just availability.

FAQ

Should I pre-estimate tokens or measure them? Pre-estimate with a fast tokenizer (tiktoken) for budget tracking; measure actual usage from response headers for billing and accuracy. The estimate is for proactive throttling; actual counts are for accounting.

What's the difference between RPM and TPM limits? RPM limits how many API calls you make. TPM limits the total tokens processed. You'll typically hit TPM first when sending large prompts, and RPM first when sending many small requests.

How do I handle rate limits in streaming responses? The 429 comes before the stream starts. Your retry logic applies normally — you retry the entire request, not a mid-stream fragment.

Should free-tier users ever wait, or should I just reject? Show free users a "you're going fast — wait 30 seconds" message rather than a hard error. This is better UX and keeps them in the funnel. Paying users should rarely wait.

What's the right number of retries? 3–4 retries with exponential backoff. More than 4 means the API is probably down (not just rate-limited), and retrying indefinitely wastes resources. Alert your team after 4 consecutive failures.