[SL_CORE]
< CD ..||6 min read

Streaming LLM Responses: Minimizing TTFT and ITL

# Streaming LLM responses reduces perceived latency dramatically. Learn to optimize Time to First Token (TTFT), Inter-Token Latency (ITL), and build streaming UIs that feel instantaneous.

[streaming][llm][performance][nextjs]

Non-streaming LLM responses make users wait 2–8 seconds staring at a spinner. Streaming starts showing text within 300–800ms — the difference between "fast" and "slow" in user perception. But streaming has nuances: server-sent events, backpressure, error handling mid-stream. Here's the complete guide.

The Two Key Metrics

Time to First Token (TTFT) — how long until the first character appears. Dominates perceived latency. Inter-Token Latency (ITL) — time between subsequent tokens. Affects reading experience.

User perception thresholds:

  • TTFT < 500ms: feels instant
  • TTFT 500ms–1s: acceptable
  • TTFT > 2s: feels slow (add a skeleton/loading state)
  • ITL < 30ms: smooth text flow
  • ITL > 100ms: visible "typing" effect (can be a feature)

Streaming with Anthropic SDK

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

// Full streaming implementation
async function* streamCompletion(prompt: string): AsyncGenerator<string> {
  const stream = await client.messages.stream({
    model: "claude-sonnet-4-6",
    max_tokens: 2048,
    messages: [{ role: "user", content: prompt }],
  });

  for await (const event of stream) {
    if (
      event.type === "content_block_delta" &&
      event.delta.type === "text_delta"
    ) {
      yield event.delta.text;
    }
  }

  // Get final message for usage stats
  const finalMessage = await stream.finalMessage();
  console.log("Tokens used:", finalMessage.usage);
}

// Usage
for await (const chunk of streamCompletion("Explain RAG")) {
  process.stdout.write(chunk);
}

Next.js Streaming API Route

// app/api/stream/route.ts
import { streamText } from "ai";
import { anthropic } from "@ai-sdk/anthropic";

export async function POST(req: Request) {
  const { messages, systemPrompt } = await req.json();

  const result = streamText({
    model: anthropic("claude-sonnet-4-6"),
    system: systemPrompt,
    messages,
    onError: (error) => console.error("Stream error:", error),
  });

  return result.toDataStreamResponse({
    headers: {
      // Disable buffering at all proxy layers
      "X-Accel-Buffering": "no",
      "Cache-Control": "no-cache, no-transform",
    },
  });
}

Client-Side Streaming UI

// Progressive rendering with React
"use client";
import { useChat } from "ai/react";

export function StreamingChat() {
  const { messages, input, handleInputChange, handleSubmit, isLoading, error } = useChat({
    api: "/api/stream",
    streamProtocol: "data",
    onError: (err) => console.error("Chat error:", err),
  });

  return (
    <div className="flex flex-col h-screen">
      <div className="flex-1 overflow-y-auto p-4 space-y-4">
        {messages.map((message) => (
          <div key={message.id} className={`flex ${message.role === "user" ? "justify-end" : "justify-start"}`}>
            <div className={`max-w-3xl rounded-xl px-4 py-3 ${
              message.role === "user" ? "bg-blue-600 text-white" : "bg-gray-100"
            }`}>
              {/* Streaming text renders token by token */}
              <p className="whitespace-pre-wrap">{message.content}</p>
            </div>
          </div>
        ))}

        {/* Show typing indicator while waiting for first token */}
        {isLoading && messages[messages.length - 1]?.role === "user" && (
          <div className="flex justify-start">
            <div className="bg-gray-100 rounded-xl px-4 py-3">
              <span className="flex gap-1">
                <span className="w-2 h-2 bg-gray-400 rounded-full animate-bounce" style={{ animationDelay: "0ms" }} />
                <span className="w-2 h-2 bg-gray-400 rounded-full animate-bounce" style={{ animationDelay: "150ms" }} />
                <span className="w-2 h-2 bg-gray-400 rounded-full animate-bounce" style={{ animationDelay: "300ms" }} />
              </span>
            </div>
          </div>
        )}
      </div>

      <form onSubmit={handleSubmit} className="border-t p-4 flex gap-2">
        <input
          value={input}
          onChange={handleInputChange}
          placeholder="Message..."
          className="flex-1 border rounded-lg px-4 py-2 focus:outline-none focus:ring-2"
          disabled={isLoading}
        />
        <button
          type="submit"
          disabled={isLoading || !input.trim()}
          className="px-6 py-2 bg-blue-600 text-white rounded-lg disabled:opacity-50"
        >
          {isLoading ? "..." : "Send"}
        </button>
      </form>

      {error && <div className="p-2 text-red-500 text-sm">Error: {error.message}</div>}
    </div>
  );
}

Optimizing TTFT

TTFT is dominated by:

  1. Network round-trip to API (~50–200ms)
  2. Model loading the context (scales with input tokens)
  3. Time-to-first-token generation

Reduce input tokens — the single biggest lever for TTFT:

// Before: sending full conversation history every time
const messages = fullConversationHistory; // 10K tokens

// After: compress old context
const messages = [
  { role: "user", content: `[Summary of prior conversation]: ${summary}` },
  ...recentMessages.slice(-5), // only last 5 turns verbatim
  { role: "user", content: currentMessage },
];
// 10K tokens → 2K tokens = ~4× faster TTFT

Warm up connections — pre-establish connections before users need them:

// In your application startup, make a tiny API call to warm connection pools
async function warmAPIConnections() {
  await anthropic.messages.create({
    model: "claude-haiku-4-5-20251001",
    max_tokens: 1,
    messages: [{ role: "user", content: "ping" }],
  });
}

Use the right model — Haiku has 2–3× lower TTFT than Sonnet. If quality is acceptable, use the faster model:

| Model | Avg TTFT | Use for | |-------|---------|---------| | claude-haiku-4-5 | 200–400ms | High-frequency, speed-critical | | claude-sonnet-4-6 | 400–800ms | Balanced quality/speed | | claude-opus-4-8 | 800–1500ms | Quality-critical only |

Handling Mid-Stream Errors

Streams can fail after sending some tokens. Handle gracefully:

async function robustStream(
  prompt: string,
  onChunk: (chunk: string) => void,
  onError: (error: Error, partialResponse: string) => void
): Promise<string> {
  let accumulated = "";

  try {
    const stream = await client.messages.stream({
      model: "claude-sonnet-4-6",
      max_tokens: 2048,
      messages: [{ role: "user", content: prompt }],
    });

    for await (const event of stream) {
      if (event.type === "content_block_delta" && event.delta.type === "text_delta") {
        const chunk = event.delta.text;
        accumulated += chunk;
        onChunk(chunk);
      }
    }

    return accumulated;
  } catch (err: any) {
    // Stream failed mid-way
    onError(err, accumulated);

    // If we got some content, return what we have
    if (accumulated.length > 100) {
      return accumulated + "\n\n[Response interrupted]";
    }

    // Otherwise retry
    throw err;
  }
}

SSE-Based Streaming Without AI SDK

For custom streaming implementations:

// Server: Server-Sent Events
export async function GET(req: Request) {
  const { searchParams } = new URL(req.url);
  const prompt = searchParams.get("prompt") ?? "";

  const encoder = new TextEncoder();
  const stream = new ReadableStream({
    async start(controller) {
      const send = (data: string) =>
        controller.enqueue(encoder.encode(`data: ${JSON.stringify({ text: data })}\n\n`));

      try {
        const anthropicStream = await client.messages.stream({
          model: "claude-sonnet-4-6",
          max_tokens: 2048,
          messages: [{ role: "user", content: prompt }],
        });

        for await (const event of anthropicStream) {
          if (event.type === "content_block_delta" && event.delta.type === "text_delta") {
            send(event.delta.text);
          }
        }

        controller.enqueue(encoder.encode("data: [DONE]\n\n"));
      } catch (err: any) {
        controller.enqueue(encoder.encode(`data: ${JSON.stringify({ error: err.message })}\n\n`));
      } finally {
        controller.close();
      }
    },
  });

  return new Response(stream, {
    headers: {
      "Content-Type": "text/event-stream",
      "Cache-Control": "no-cache",
      "Connection": "keep-alive",
      "X-Accel-Buffering": "no",
    },
  });
}

// Client: EventSource
const source = new EventSource(`/api/stream?prompt=${encodeURIComponent(query)}`);
source.onmessage = (e) => {
  const { text, error } = JSON.parse(e.data);
  if (text === "[DONE]") { source.close(); return; }
  if (error) { showError(error); source.close(); return; }
  appendText(text);
};

FAQ

How do I stream through a Next.js middleware? Add X-Accel-Buffering: no to response headers to disable buffering at nginx/Cloudflare. Without this, middleware or proxies buffer the stream and users see delayed batches of tokens.

Does streaming work with Cloudflare Pages? Yes — Cloudflare Workers support streaming responses natively. Use ReadableStream and set X-Accel-Buffering: no.

How do I handle streaming in mobile apps? Use HTTP chunked transfer encoding or WebSocket. EventSource (SSE) works on iOS/Android but requires eventsource polyfill on older React Native versions.

Should I cache streaming responses? Cache the complete response (after stream ends), not the stream itself. On repeat queries, serve the cached full text immediately — no streaming needed.

How do I stop a stream when the user cancels? Pass an AbortController signal to the fetch call. When the user cancels, controller.abort() stops the stream server-side (Next.js and the AI SDK honor this). The Anthropic API stops billing for tokens not yet generated.