Open-Source vs Proprietary LLMs: When to Use Each

Llama 4, Qwen 3, and DeepSeek R2 have dramatically closed the capability gap with Claude and GPT-5 in 2026. For many tasks, open-source models running on your own infrastructure now match proprietary API quality. But "can do the task" and "should use for production" are different questions.

The 2026 Model Landscape

Open-Source Models Worth Using in Production

| Model | Parameters | Best for | Hosting options | |-------|-----------|---------|----------------| | Llama 4 Scout | 17B | Fast inference, edge deployment | Ollama, vLLM, Groq | | Qwen 3 32B | 32B | Multilingual, coding, instruction following | vLLM, Together AI | | DeepSeek R2 | 70B | Complex reasoning, math | vLLM (high VRAM) | | Mistral Nemo | 12B | Lightweight, function calling | Ollama, vLLM | | Phi-4 | 14B | Small device deployment | Ollama, LM Studio |

Proprietary APIs

| Model | Best for | When worth the premium | |-------|---------|----------------------| | Claude Opus 4.8 | Complex reasoning, writing | Safety-critical, highest quality tasks | | Claude Sonnet 4.6 | Balanced production work | Most commercial applications | | GPT-5 | Code generation, tools | When OpenAI ecosystem compatibility matters | | Gemini 2.5 Pro | Long context (1M), multimodal | Document analysis, video |

The Decision Framework

Can this task be solved with an open-source model?
│
├── Test: run your eval suite on Qwen 3 or Llama 4
│
├── Quality matches proprietary (>95% relative)?
│   └── YES → Use open-source (cost savings: 70–90%)
│
├── Quality gap <10%?
│   ├── Data residency requirement? → Open-source
│   ├── Volume >5M tokens/day? → Open-source
│   ├── Latency <200ms required? → Depends on hardware
│   └── Otherwise → Proprietary (simpler, maintained)
│
└── Quality gap >10%?
    └── Use proprietary for this task

When Open-Source Wins

Data residency / privacy — some industries (healthcare, government, finance) cannot send data to third-party APIs. Self-hosted models are the only option.

High volume — at >5M tokens/day, self-hosting a 7–13B model on dedicated GPUs costs less than API fees.

API cost (Claude Haiku):    5M tokens × $0.80/M = $4/day = $1,460/year
Self-hosted (H100 1× GPU):  ~$3/hour × 24 = $72/day = $26,280/year
Break-even: ~18M tokens/day

At the high end (100M+ tokens/day), self-hosting pays back in 3 months.

Customization — proprietary APIs don't let you modify the model. Self-hosting enables: fine-tuning on your data, custom tokenization, model merging, RLHF on your feedback.

Latency — on a co-located GPU, p99 latency for a 7B model is 50–100ms. API round-trip adds 100–300ms of network overhead.

When Proprietary Wins

Quality for complex tasks — Claude Opus and GPT-5 still lead on: multi-step reasoning, nuanced writing, complex code generation, and tasks requiring broad world knowledge. The gap is real for 15–20% of complex use cases.

Operational simplicity — no GPU clusters to manage, no inference infrastructure to scale, no model updates to monitor. For teams without dedicated ML infrastructure, API is far simpler.

Safety and alignment — proprietary models have more extensive safety fine-tuning and content filtering. For consumer-facing products where trust and safety are critical, this matters.

Support and SLA — API providers offer uptime guarantees, support contracts, and compliance certifications. Self-hosted models have no SLA beyond your own infrastructure.

Running Open-Source Models

Local development (Ollama):

# Run Qwen 3 locally for development
ollama pull qwen3:32b
ollama serve

# OpenAI-compatible API at localhost:11434
curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3:32b", "messages": [{"role": "user", "content": "Hello"}]}'

Production (vLLM on GPU):

# Deploy Llama 4 Scout with vLLM
docker run --gpus all \
  -p 8000:8000 \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-4-Scout-17B-Instruct \
  --tensor-parallel-size 1 \
  --max-model-len 8192

Code integration (OpenAI-compatible client):

import OpenAI from "openai";

// Use OpenAI SDK with any OpenAI-compatible endpoint
const client = new OpenAI({
  baseURL: process.env.LLM_BASE_URL ?? "https://api.anthropic.com/v1",
  apiKey: process.env.LLM_API_KEY ?? process.env.ANTHROPIC_API_KEY,
});

// Same code works for Ollama, vLLM, Together AI, or Anthropic
const response = await client.chat.completions.create({
  model: process.env.LLM_MODEL ?? "claude-sonnet-4-6",
  messages: [{ role: "user", content: prompt }],
});

Writing model-agnostic code with the OpenAI-compatible interface means you can switch between providers without code changes.

Hybrid Architecture: Best of Both

class HybridLLMRouter {
  async complete(task: string, options: { requiresHighQuality?: boolean; sensitive?: boolean }) {
    // Route to open-source for: non-sensitive, moderate complexity, high volume
    if (!options.requiresHighQuality && !options.sensitive) {
      return this.callOpenSource(task);
    }

    // Route to proprietary for: high quality required, or sensitive data
    return this.callProprietary(task);
  }

  private async callOpenSource(task: string) {
    const client = new OpenAI({ baseURL: process.env.OSS_LLM_URL });
    return client.chat.completions.create({
      model: "qwen3:32b",
      messages: [{ role: "user", content: task }],
    });
  }

  private async callProprietary(task: string) {
    return anthropic.messages.create({
      model: "claude-sonnet-4-6",
      messages: [{ role: "user", content: task }],
      max_tokens: 2048,
    });
  }
}

At Shahriar Labs, internal tooling and development use Ollama with Qwen/Llama. Production user-facing features use Claude (quality and reliability guarantees matter for paying customers).

FAQ

Can open-source models handle tool use and function calling? Yes — Llama 4, Qwen 3, and Mistral all support function calling. Quality varies: Claude and GPT-5 are still more reliable for complex multi-tool workflows. Test on your specific tools before switching.

What hardware do I need for self-hosting? 7B models: single RTX 4090 (24GB VRAM). 13B models: 2× RTX 4090. 32B models: single H100 (80GB VRAM). 70B models: 4× A100. Cloud GPU (Lambda, CoreWeave, Vast.ai) is often cheaper than buying for variable loads.

How do I evaluate an open-source model for my use case? Run your golden eval dataset (from Testing LLM Applications) against the candidate model. If pass rate is within 5% of your current model, it's worth considering.

Is Together AI or Groq a good middle ground? Yes — managed inference for open-source models. Groq's LPU delivers sub-50ms latency for Llama models. Together AI offers fine-tuning on top of open-source. Both are cheaper than Claude/GPT for high volume, without self-hosting operational complexity.

Will open-source models keep improving relative to proprietary? Yes — the capability gap keeps closing. Expect parity for most business tasks within 12–18 months. The frontier (highest quality reasoning) will stay with well-resourced proprietary labs longer.