Multimodal LLM Integration: Vision and Docs in Production

Multimodal LLMs can see. Claude, GPT-4o, and Gemini all accept images as input — you send an image URL or base64 content and the model reasons about it the same way it reasons about text. In production, this unlocks: document processing without OCR, visual bug detection, UI screenshot analysis, and chart/graph understanding.

Sending Images to Claude

Two methods: URL reference or base64:

import Anthropic from "@anthropic-ai/sdk";

const client = new Anthropic();

// Method 1: URL (model fetches at inference time)
async function analyzeImageFromURL(imageUrl: string, question: string): Promise<string> {
  const response = await client.messages.create({
    model: "claude-sonnet-4-6",
    max_tokens: 1024,
    messages: [{
      role: "user",
      content: [
        { type: "image", source: { type: "url", url: imageUrl } },
        { type: "text", text: question },
      ],
    }],
  });
  return response.content[0].text;
}

// Method 2: Base64 (send image data directly)
async function analyzeImageFromFile(imagePath: string, question: string): Promise<string> {
  const imageBuffer = await fs.readFile(imagePath);
  const base64 = imageBuffer.toString("base64");
  const mediaType = getMediaType(imagePath); // "image/jpeg" | "image/png" | "image/webp" | "image/gif"

  const response = await client.messages.create({
    model: "claude-sonnet-4-6",
    max_tokens: 1024,
    messages: [{
      role: "user",
      content: [
        { type: "image", source: { type: "base64", media_type: mediaType, data: base64 } },
        { type: "text", text: question },
      ],
    }],
  });
  return response.content[0].text;
}

function getMediaType(path: string): "image/jpeg" | "image/png" | "image/webp" | "image/gif" {
  const ext = path.split(".").pop()?.toLowerCase();
  const types: Record<string, "image/jpeg" | "image/png" | "image/webp" | "image/gif"> = {
    jpg: "image/jpeg", jpeg: "image/jpeg", png: "image/png", webp: "image/webp", gif: "image/gif"
  };
  return types[ext ?? ""] ?? "image/jpeg";
}

Document Processing (PDFs, Screenshots, Forms)

Extract structured data from documents without traditional OCR:

const INVOICE_SCHEMA = z.object({
  invoiceNumber: z.string(),
  date: z.string(),
  vendor: z.object({ name: z.string(), address: z.string().optional() }),
  lineItems: z.array(z.object({
    description: z.string(),
    quantity: z.number(),
    unitPrice: z.number(),
    total: z.number(),
  })),
  subtotal: z.number(),
  tax: z.number().optional(),
  total: z.number(),
  dueDate: z.string().optional(),
});

type Invoice = z.infer<typeof INVOICE_SCHEMA>;

async function extractInvoice(imageBase64: string): Promise<Invoice> {
  const response = await client.messages.create({
    model: "claude-sonnet-4-6",
    max_tokens: 2000,
    messages: [{
      role: "user",
      content: [
        { type: "image", source: { type: "base64", media_type: "image/png", data: imageBase64 } },
        {
          type: "text",
          text: `Extract all information from this invoice as JSON matching this schema:
${JSON.stringify(INVOICE_SCHEMA.shape, null, 2)}

Return only valid JSON, no other text.`,
        },
      ],
    }],
  });

  const jsonStr = response.content[0].text.trim();
  return INVOICE_SCHEMA.parse(JSON.parse(jsonStr));
}

Multi-Image Analysis

Send multiple images in one request:

async function compareScreenshots(
  before: string,  // base64
  after: string,   // base64
): Promise<{ changes: string[]; regressions: string[]; improvements: string[] }> {
  const response = await client.messages.create({
    model: "claude-sonnet-4-6",
    max_tokens: 1500,
    messages: [{
      role: "user",
      content: [
        { type: "text", text: "Before deployment:" },
        { type: "image", source: { type: "base64", media_type: "image/png", data: before } },
        { type: "text", text: "After deployment:" },
        { type: "image", source: { type: "base64", media_type: "image/png", data: after } },
        {
          type: "text",
          text: "Compare these UI screenshots. What changed? Identify regressions (worse) and improvements (better). Return JSON: {changes: [], regressions: [], improvements: []}",
        },
      ],
    }],
  });

  return JSON.parse(response.content[0].text);
}

Image Preprocessing for Better Results

Raw images often need preprocessing before sending to the model:

import sharp from "sharp";

async function preprocessForLLM(
  inputPath: string,
  options: {
    maxWidth?: number;
    maxHeight?: number;
    quality?: number;
    grayscale?: boolean;
  } = {}
): Promise<string> {
  const { maxWidth = 2048, maxHeight = 2048, quality = 85 } = options;

  let pipeline = sharp(inputPath)
    .resize(maxWidth, maxHeight, { fit: "inside", withoutEnlargement: true });

  if (options.grayscale) pipeline = pipeline.grayscale();

  const outputBuffer = await pipeline
    .jpeg({ quality })
    .toBuffer();

  return outputBuffer.toString("base64");
}

// Optimize token usage: resize large images
// Claude's image pricing is based on image size (token equivalent)
// 1568×1568 image = ~1600 tokens
// 800×800 image = ~400 tokens (4× cheaper)
async function optimizeImageForCost(inputBase64: string): Promise<string> {
  const buffer = Buffer.from(inputBase64, "base64");
  const metadata = await sharp(buffer).metadata();

  // Only resize if larger than threshold
  if ((metadata.width ?? 0) > 1000 || (metadata.height ?? 0) > 1000) {
    const resized = await sharp(buffer)
      .resize(1000, 1000, { fit: "inside" })
      .jpeg({ quality: 80 })
      .toBuffer();
    return resized.toString("base64");
  }

  return inputBase64;
}

Handling PDF Pages

PDFs need page-by-page processing:

import pdf from "pdf-parse";
import { createCanvas } from "canvas";

async function processPDFDocument(
  pdfBuffer: Buffer,
  extractionTask: string
): Promise<string[]> {
  // Convert each page to image
  const pages = await convertPDFToImages(pdfBuffer);
  const results: string[] = [];

  // Process pages in parallel (with concurrency limit)
  const sem = new Semaphore(3);
  await Promise.all(
    pages.map(async (pageImage, i) => {
      await sem.acquire();
      try {
        const result = await client.messages.create({
          model: "claude-sonnet-4-6",
          max_tokens: 500,
          messages: [{
            role: "user",
            content: [
              { type: "image", source: { type: "base64", media_type: "image/png", data: pageImage } },
              { type: "text", text: `Page ${i + 1}/${pages.length}: ${extractionTask}` },
            ],
          }],
        });
        results[i] = result.content[0].text;
      } finally {
        sem.release();
      }
    })
  );

  return results;
}

Cost Management for Vision

Vision calls are more expensive than text:

| Image size | Estimated tokens | Sonnet cost | |-----------|-----------------|------------| | 200×200 | ~150 | $0.0005 | | 800×800 | ~600 | $0.002 | | 1568×1568 | ~1600 | $0.005 | | Max (4K) | ~6400 | $0.019 |

function estimateImageCost(widthPx: number, heightPx: number): number {
  // Claude's tile-based pricing: 512×512 tiles
  const tilesW = Math.ceil(widthPx / 512);
  const tilesH = Math.ceil(heightPx / 512);
  const tiles = tilesW * tilesH;
  const tokenEquivalent = tiles * 340 + 85; // approximate

  const SONNET_INPUT_PRICE = 0.003 / 1000;
  return tokenEquivalent * SONNET_INPUT_PRICE;
}

// Budget check before vision call
async function guardrailedVisionCall(
  imageBuffer: Buffer,
  question: string,
  maxCostUSD = 0.05
): Promise<string> {
  const metadata = await sharp(imageBuffer).metadata();
  const estimatedCost = estimateImageCost(metadata.width ?? 800, metadata.height ?? 600);

  if (estimatedCost > maxCostUSD) {
    // Downscale to fit budget
    const scaleFactor = Math.sqrt(maxCostUSD / estimatedCost);
    const newWidth = Math.floor((metadata.width ?? 800) * scaleFactor);
    const newHeight = Math.floor((metadata.height ?? 600) * scaleFactor);

    const resized = await sharp(imageBuffer)
      .resize(newWidth, newHeight)
      .toBuffer();

    return analyzeImageBuffer(resized, question);
  }

  return analyzeImageBuffer(imageBuffer, question);
}

Common Production Edge Cases

1. Image too large: Base64 encoding increases size 33%. Images over 5MB may time out. Always preprocess.

2. Low contrast or small text: Preprocess with contrast enhancement before sending for text extraction tasks.

3. Multiple documents in one image: Ask the model to identify boundaries first, then process each section.

4. Handwritten text: Works but accuracy is lower. Set user expectations and add a confidence field to outputs.

5. Charts and graphs: Claude can extract values from charts but may misread exact numbers. Add validation: extracted values should sum to total, percentages should add to 100%.

FAQ

How does vision quality compare to traditional OCR? Claude outperforms traditional OCR for: structured forms, mixed text/images, handwriting, and context-dependent extraction. Traditional OCR wins on: pure text documents at scale, offline processing, and cost.

Can I analyze video? Not directly — extract frames first. For a 60-second video at 1 fps = 60 images. Select keyframes intelligently to reduce cost.

What image formats are supported? JPEG, PNG, GIF (first frame only), WebP. Convert PDFs and TIFFs to PNG/JPEG before sending.

Does image analysis support streaming? Yes — the token generation streams the same way as text. The image is processed server-side before generation begins; there's no partial image streaming.

How do I maintain image privacy? Avoid sending images with PII (faces, documents with personal info) to external APIs without consent and legal basis. For sensitive images, use self-hosted models (LLaVA, Qwen2-VL) or process locally.