freelm: One Free LLM API Gateway in Python

freelm is a free, open-source Python client that pools six free-tier LLM providers — OpenRouter, Google AI Studio (Gemini), NVIDIA NIM, Groq, Cerebras, and Mistral — behind one OpenAI-compatible call. It rotates keys, fails over across providers on rate limits or errors, paces each tier, and discovers free models live, so your app keeps answering even when one source goes down.

Install it and make a call in two lines:

pip install freelm

import freelm

llm = freelm.FreeLLM.from_env()           # reads provider keys from env
print(llm.text("Explain prompt caching in one sentence."))

Why a free LLM gateway exists

Almost every app now needs an LLM, and tokens cost money — but a lot of capacity is free if you spread requests across providers. Each provider meters its free tier independently, so combining Gemini, Groq, OpenRouter, NIM, Cerebras, and Mistral multiplies your effective free throughput. The problem is operational: six SDKs, six auth schemes, six rate-limit behaviors, and constantly changing free model IDs. freelm collapses that into one client with one mental model, so you write your app once and let the gateway absorb the differences.

What freelm handles for you

The gateway layer owns reliability so your application code stays clean. freelm classifies every failure and reacts: a 429 cools that key and rotates to the next; a 5xx or timeout trips a per-key circuit breaker with backoff; a 401 disables a dead key but keeps serving from others; an unknown model falls through to the next model. Candidates are interleaved across providers, so failover reaches every provider quickly instead of exhausting one provider's many models first.

from freelm import FreeLLM, OpenRouter, GoogleAIStudio, NIM, Groq, Cerebras, Mistral

llm = FreeLLM(
    providers=[
        OpenRouter("sk-or-..."),
        GoogleAIStudio("AIza..."),
        Groq("gsk_..."),
        Cerebras("csk-..."),
        Mistral("..."),
        NIM("nvapi-..."),
    ],
    strategy="quota_aware",   # priority | round_robin | quota_aware | latency
)

Six providers, one OpenAI-compatible call

All six providers expose OpenAI-compatible HTTP, so freelm speaks one protocol to all of them and normalizes the responses. You can ask by intent with virtual models — auto, chat:fast, chat:large — and freelm maps each to a concrete model per provider. If you already use the OpenAI SDK, the drop-in shim swaps in with almost no code change:

from freelm.compat import OpenAI

client = OpenAI()                          # backed by FreeLLM.from_env()
r = client.chat.completions.create(
    model="auto",
    messages=[{"role": "user", "content": "hi"}],
)
print(r.choices[0].message.content)

Live model discovery

Free model IDs churn constantly — providers add and retire :free models every week. freelm does not trust a hardcoded list: on first use it queries each provider's /models endpoint, derives capability tags, caches the result to disk with a TTL, and falls back to a built-in list only when offline. The resolution order is live API → disk cache → hardcoded fallback, so a default call always reaches a model that currently exists.

from freelm import list_free_models

for m in list_free_models()[:5]:           # current OpenRouter free models
    print(m.id, m.tags)

Streaming and async

freelm supports token streaming and async out of the box, both routed through the same failover. It fails over between providers before the first token; once tokens start flowing it commits to that provider so you never get a spliced response.

for chunk in llm.stream("Write a haiku about failover."):
    print(chunk, end="", flush=True)

from freelm import AsyncFreeLLM

async with AsyncFreeLLM.from_env() as llm:
    async for chunk in llm.astream("stream me some tokens"):
        print(chunk, end="", flush=True)

Inspecting the pool

Because reliability is the whole point, freelm exposes live state. health() returns one row per key — whether it is ready, its breaker state, requests used today, last error, and rolling latency — so you can see exactly why the router picked what it did.

for row in llm.health():
    print(row["provider"], row["ready"], row["last_error"])

Frequently Asked Questions

Is freelm really free? freelm itself is MIT-licensed and free. It runs on providers' free tiers, so your actual limits depend on each provider's free quota. Adding more keys or providers raises total throughput.

Which providers does freelm support? Six today, all OpenAI-compatible: OpenRouter, Google AI Studio (Gemini), NVIDIA NIM, Groq, Cerebras, and Mistral. Each is optional — supply only the keys you have.

Do I have to change my OpenAI code? No. Use from freelm.compat import OpenAI for a drop-in replacement, or the native FreeLLM API for finer control over strategy, timeouts, and failover.

How does freelm pick a model? Ask for a virtual model (auto, chat:fast, chat:large) and freelm resolves it per provider from the live model list, deprioritizing slow giant and reasoning models so a default call stays fast.

Does it support streaming? Yes — stream() (sync) and astream() (async) yield content deltas across all providers, with failover before the first token.

---

Written by Shihab Shahriar Antor — AI Engineer & Founder of Shahriar Labs. freelm is open source: install from PyPI (pip install freelm) or read the source on GitHub.