AI Gateways: LiteLLM, OpenRouter, Portkey, Vercel
Once your product talks to more than one model, the direct-SDK approach cracks. Each provider has its own key, its own rate limits, its own outage schedule, and its own bill. An AI gateway is the small piece of infrastructure that sits between your code and every model — Claude, GPT, Gemini, Llama, Kimi, DeepSeek, your local Ollama — and turns "N brittle integrations" into "one endpoint you control." This page compares the four gateways that actually get deployed in production in 2026 — LiteLLM, OpenRouter, Portkey, and Vercel AI Gateway — and shows the killer workflow: point Claude Code at your own gateway so a single proxy handles routing, budgets, logging, and fallback for the whole team.
- Understand what an AI gateway is and the five problems it solves (many providers, one API; fallback; virtual keys; spend caps; observability)
- Compare LiteLLM, OpenRouter, Portkey, and Vercel AI Gateway on latency, pricing, self-hostability, and where each shines
- Wire Claude Code through your own LiteLLM proxy with ANTHROPIC_BASE_URL and a virtual key so the team gets shared limits and logs
- Configure OpenRouter fallback so a Claude outage silently promotes GPT or Gemini instead of showing users a 5xx
- Understand the March 2026 LiteLLM supply-chain incident and how to pin versions safely in production
The problem: one direct SDK per provider does not scale
The first Claude integration is a two-line change: pip install anthropic, set ANTHROPIC_API_KEY, done. The second — say you want to fall back to GPT-5.4 when Anthropic throttles — is where the abstraction breaks. Now you have two SDKs with different request shapes, two dashboards, two bills, two rotation cadences for API keys, and two sets of retry logic. Add a third for Gemini and a fourth for your local Ollama, and every product decision ("cap this team at $500/month", "log every prompt for review", "let a customer bring their own key") becomes N implementations instead of one.
An AI gateway concentrates that plumbing in a single place. Concretely, a production gateway gives you:
- One request shape for every provider. Most gateways speak the OpenAI Chat Completions API (or Anthropic Messages, or both) and translate to the real provider under the hood.
- Fallback and routing. Try Claude first; on 429 or 5xx, retry against GPT or Gemini without the caller knowing. Same for latency ceilings and content-moderation rejections.
- Virtual keys. Issue a per-user or per-service key that maps to a subset of models, its own budget, and its own rate limit — so a rogue script can't drain the whole account.
- Spend caps and logging. Every request is tagged, priced, and stored. You can revoke a key without touching Anthropic or OpenAI, and you can prove to compliance what was sent where.
- Caching. Prompt caching (exact-match) and semantic caching (near-match) turn repeat traffic into free hits.
Not every team needs all five. But the moment two of them are on your roadmap, running a gateway is cheaper than reinventing them per provider.
The four gateways that ship in production
There isn't one "winner" — the four leaders occupy different corners of the design space (self-hosted vs. hosted, open-source vs. proprietary, minimalist vs. control-panel).
| Gateway | Deploy | Pricing model | Best at | Not for |
|---|---|---|---|---|
| LiteLLM | Self-hosted (Docker) or SDK | Free (OSS); Enterprise tier for SSO/audit | Team proxy with virtual keys, budgets, no per-token markup, works with 100+ providers via one config | Teams with no DevOps to run Postgres + Redis |
| OpenRouter | Hosted only | Provider price + ~5.5% credit-purchase fee, no per-request markup | Zero-ops access to 300+ models under one key; ideal for products that let users pick a model | Compliance shops that need self-hosting or data residency |
| Portkey | OSS gateway (npx) or hosted cloud | OSS free; cloud has usage tiers | Sub-ms gateway latency, semantic caching, guardrails, canary testing — the "control panel" angle | Teams who just want the simplest possible key aggregator |
| Vercel AI Gateway | Hosted only | Provider price, no token markup; free with Vercel plans | Devs already on Vercel who want AI SDK v5/v6 + Anthropic Messages + OpenAI Responses APIs unified | Non-Vercel infrastructure or air-gapped deployments |
The important axis to pick on first: self-hosted vs. hosted. If your data cannot leave your VPC (regulated industries, EU residency, enterprise privacy reviews), you need a self-hostable gateway — LiteLLM or Portkey OSS. If you'd rather pay someone to run it, OpenRouter or Vercel AI Gateway is a one-click affair.
The second axis: how much control plane you actually need. If you're a one-person product that just wants to try Kimi K3 and Claude and Grok side-by-side without three signups, OpenRouter is the whole story. If you're a 20-person org where finance wants monthly spend by team, security wants virtual keys with rotation, and platform wants Grafana metrics, you're building on LiteLLM or Portkey.
Killer workflow: point Claude Code at your own LiteLLM proxy
The best-kept secret about Claude Code is that it respects ANTHROPIC_BASE_URL and ANTHROPIC_AUTH_TOKEN. Set them to your gateway and Claude Code stops talking to api.anthropic.com directly — it talks to your proxy, which forwards to Anthropic (or anywhere else) with the auth you control. For a team, this changes three things at once:
- One shared virtual key per developer. You issue and revoke keys in the proxy UI. No shared root credentials in
.envfiles. - Per-developer budgets and logs. The proxy tags every request, so "who spent the $300 yesterday" is a database query, not an incident.
- Model aliasing. You can pin
claude-sonnet-4-6at the proxy so a model deprecation is a one-line config change, not a repo-wide grep.
Start a minimal proxy in three steps:
- In a fresh venv or via uv: uv tool install 'litellm[proxy]'. This pulls in the gateway server (FastAPI + admin UI) alongside the client SDK.
- Model IDs on the left are the ALIAS your callers see (whatever you want); the litellm_params.model on the right is the REAL provider route. Put your ANTHROPIC_API_KEY in the env, not the file.
- Run litellm --config config.yaml (default port 4000). Then set ANTHROPIC_BASE_URL to the proxy URL and ANTHROPIC_AUTH_TOKEN to a virtual key. Claude Code will route every call through the proxy without knowing.
The config file that makes this work:
config.yaml — Claude Sonnet/Opus/Haiku behind LiteLLM
model_list:
- model_name: claude-opus-4-7
litellm_params:
model: anthropic/claude-opus-4-7
api_key: os.environ/ANTHROPIC_API_KEY
- model_name: claude-sonnet-4-6
litellm_params:
model: anthropic/claude-sonnet-4-6
api_key: os.environ/ANTHROPIC_API_KEY
- model_name: claude-haiku-4-5-20251001
litellm_params:
model: anthropic/claude-haiku-4-5-20251001
api_key: os.environ/ANTHROPIC_API_KEY
litellm_settings:
master_key: os.environ/LITELLM_MASTER_KEY
# Optional: enable exact-match prompt caching
cache: true
cache_params:
type: redis
host: os.environ/REDIS_HOSTThen, from any developer's shell:
Point Claude Code at the proxy (per-developer .env)
export ANTHROPIC_BASE_URL="https://llm.internal.example.com" export ANTHROPIC_AUTH_TOKEN="sk-team-alice-9f4c..." # a VIRTUAL key issued by the proxy # now every Claude Code call goes through YOUR gateway claude --model claude-sonnet-4-6
The non-obvious win is the virtual key. The master key is admin-only and never ships to laptops. Each developer gets a virtual key that maps to only the models you allow, has its own monthly budget, and can be revoked in seconds without rotating the underlying Anthropic key. If a laptop is lost, you kill one row in Postgres — not the whole team's access.
Careful: the same env vars work with Anthropic's Bedrock and Vertex integrations, but there are edge cases with experimental beta features. For Bedrock deployments the LiteLLM docs recommend setting
CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1in~/.claude/settings.jsonto avoid header-compatibility issues.
Killer workflow #2: silent fallback with OpenRouter
If you don't want to host anything, OpenRouter's fallback array is the shortest path to "silently retry another model when Claude 429s". You send an ordered list; OpenRouter walks it top-down and returns the first model that answered.
Claude → GPT → Gemini fallback in one request (OpenRouter)
import openai
client = openai.OpenAI(
api_key="YOUR_OPENROUTER_KEY",
base_url="https://openrouter.ai/api/v1",
)
response = client.chat.completions.create(
model="anthropic/claude-sonnet-4.5",
extra_body={
# Ordered fallback. If the first model 429s, is down, or is
# rejected by moderation, OpenRouter tries the next one.
"models": [
"anthropic/claude-sonnet-4.5",
"openai/gpt-5.4",
"google/gemini-2.5-pro",
],
},
messages=[{"role": "user", "content": "Explain B-trees in one paragraph."}],
)
# 'model' in the response tells you which one actually answered.
print(response.model, "->", response.choices[0].message.content)Three things people miss on their first try:
- Billing follows the model that answered, not the one you asked for. If Claude fails and GPT-5.4 answers, you pay OpenRouter's GPT-5.4 rate for that request.
- The fallback triggers on more than 5xx. Rate-limiting, provider downtime, context-length validation errors, and content-moderation refusals all promote to the next model. That last one is the sharpest edge — a "moderation" refusal from one provider can silently route to a more permissive one, which may or may not be what you want. Review your fallback list with the same care as an ACL.
- You cannot mix
modelswithfallbacks. The Anthropic-format Messages endpoint uses a differentfallbacksarray. Sending both keys in the same request returns a 400. Pick the format your client speaks and stick to it.
The March 2026 LiteLLM supply-chain incident: what to actually do
On 24 March 2026 at 10:39 UTC, two malicious PyPI releases of LiteLLM — v1.82.7 and v1.82.8 — were published by an attacker after they stole the maintainer's PyPI credentials via a prior compromise of Trivy, a security scanner running in LiteLLM's CI/CD pipeline. PyPI quarantined the packages at 13:38 UTC (about three hours later). During the exposure window, tens of thousands of downloads occurred. The payload was an infostealer with a persistence mechanism (a litellm_init.pth file that ran on every Python invocation, harvested credentials, and installed a systemd backdoor). Attribution goes to a group tracked as TeamPCP, which also compromised Trivy and Checkmarx KICS.
If you run LiteLLM in any environment, apply this once and then keep it in your platform playbook:
- v1.82.6 and earlier are clean. v1.83.0 and later (published via LiteLLM's rebuilt CI/CD v2 pipeline) are clean. Anything in between should be uninstalled and the environment considered tainted. The official Docker image (ghcr.io/berriai/litellm) was NOT compromised — the incident was PyPI-only.
- Grep site-packages for litellm_init.pth. If it exists, treat the machine as compromised: rotate every credential that was present in env vars or on disk (Anthropic, OpenAI, cloud, DB, SSH, K8s tokens) and forensic-scan for the systemd backdoor.
- From v1.83.0-nightly onward, LiteLLM signs its images. Verifying with cosign before rollout catches a repeat of this incident at the container layer.
- The Docker image escaped the attack; the PyPI wheel did not. That's a durable signal: for a networked service that holds API keys, running the pinned container is safer than a pip-installed venv on a shared host.
- The malware phoned home to models.litellm[.]cloud and checkmarx[.]zone — neither is legitimate. Egress allowlists on production LLM proxies catch this class of attack early.
The wider lesson isn't "don't use LiteLLM" — it's "assume every dependency in your AI stack, including security scanners, can be a delivery vector." Pin versions, sign images, and put your gateway on a network segment that only reaches the model providers.
Choose the right gateway for your situation
- One or two providers with a small team → skip the gateway; direct SDKs are fine. Three+ providers OR a team where 'who has the key' matters → gateway. If you have DevOps and privacy requirements, self-host LiteLLM or Portkey OSS. If you'd rather pay someone else to run it, OpenRouter (hosted-only) or Vercel AI Gateway (great if you already deploy there).
- Yes → LiteLLM (native, mature) or Portkey (native, plus semantic caching). No → OpenRouter or Vercel AI Gateway are lighter.
- OpenRouter's models[] and Vercel AI Gateway's provider-options fallbacks are the shortest path. LiteLLM does it too via fallbacks: in the config, but writes closer to a rules engine than a one-line array.
- Then LiteLLM wins by a mile — it's the only gateway with first-class docs for the ANTHROPIC_BASE_URL + virtual-key pattern, so a team of ten Claude Code users behind one proxy just works.
- Self-hosted only: LiteLLM proxy container or Portkey OSS via npx @portkey-ai/gateway. Egress-allowlist the proxy to the providers it's authorised to reach.
Common combinations that ship in production:
- Solo dev / prototype: OpenRouter direct. One key, 300+ models, done.
- Small team, Claude-first: LiteLLM proxy with Anthropic + one fallback provider, virtual keys per engineer, Redis prompt caching.
- Vercel-native product: Vercel AI Gateway with the AI SDK; add OpenRouter as a
provider-optionsfallback for exotic models. - Regulated / EU: Self-hosted LiteLLM or Portkey OSS in-VPC with Presidio PII masking in front (see Claude + Local Models for the redaction pattern).
- AI-product with heavy repeat traffic: Portkey (semantic caching commonly drives 30–50% cost reduction on chat-style workloads, per Portkey's own case studies — verify on your traffic before believing headline numbers).
What a gateway does NOT solve
Gateways are middleware — they change how you reach models, not which model is right. Two things still need real work:
- Prompt portability. Claude, GPT, and Gemini answer the same prompt differently, and system-prompt conventions vary. A gateway doesn't rewrite your prompt for the fallback provider — that's what Porting prompts across models and Cross-AI translation are for.
- Evals. The gateway makes it easy to A/B two models on the same request. It cannot tell you which one was actually better on YOUR task. Run a real eval (see Evals) before switching defaults.
A common mistake is to install a gateway and consider "multi-model" done. The gateway is the transport layer; portability and evals are the product layer.
Check yourself
0/5- An AI gateway is the missing router between your app and every model — it exists to make virtual keys, budgets, fallback, logging, and caching a single implementation instead of N per provider
- Pick on TWO axes first: self-hosted vs. hosted (LiteLLM/Portkey OSS vs. OpenRouter/Vercel), and minimalist vs. control-panel (OpenRouter/Vercel vs. LiteLLM/Portkey)
- The Claude Code killer workflow: point ANTHROPIC_BASE_URL at your own LiteLLM proxy and issue per-developer virtual keys — team gets shared limits, logs, and one-click revoke without touching the root Anthropic key
- OpenRouter's models[] array is the shortest path to silent Claude → GPT → Gemini fallback, but note that moderation refusals are a fallback trigger too — review the list like an ACL
- After the March 2026 LiteLLM supply-chain attack, pin to v1.82.6 or earlier, or v1.83.0+; prefer the signed Docker image over pip; egress-allowlist the proxy
- A gateway is transport, not product — prompt portability and evals still need real work, no matter how many models you can now reach
Sources & further reading
- LiteLLM — GitHub (BerriAI/litellm) — the source repo and current release notes
- LiteLLM Proxy — official docs — install, config.yaml, virtual keys, budgets
- Claude Code via LiteLLM — official quickstart — ANTHROPIC_BASE_URL setup, verification curl, security notes
- LiteLLM Anthropic provider docs — supported Claude models and options
- Security Update: Suspected Supply Chain Incident (March 2026) — LiteLLM blog — official incident post, safe-version guidance, remediation
- Incident Report: LiteLLM/Telnyx supply-chain attacks — PyPI blog — PyPI's timeline and mitigations
- LiteLLM compromised on PyPI — Datadog Security Labs — malware analysis (litellm_init.pth, egress domains)
- OpenRouter — model fallbacks documentation — the models[] array, triggers, billing rules
- OpenRouter — provider preferences — advanced routing controls
- Portkey AI Gateway — official docs — semantic caching, guardrails, canary
- Portkey Gateway — GitHub (OSS) — self-hostable open-source gateway
- Vercel AI Gateway — official docs — models, providers, BYOK, observability
- Vercel AI Gateway — Anthropic Messages API compatibility — using the Anthropic SDK through Vercel AI Gateway
Related on this site: Claude + Local Models: Hybrid Patterns · Porting prompts across models · Cross-AI translation · Evals · What AI costs across providers · Gray-market AI proxies ("Poison Claude") — the same routing mechanism, weaponized