Skip to main content

AI Gateways: LiteLLM, OpenRouter, Portkey, Vercel

Intermediate

Once your product talks to more than one model, the direct-SDK approach cracks. Each provider has its own key, its own rate limits, its own outage schedule, and its own bill. An AI gateway is the small piece of infrastructure that sits between your code and every model — Claude, GPT, Gemini, Llama, Kimi, DeepSeek, your local Ollama — and turns "N brittle integrations" into "one endpoint you control." This page compares the four gateways that actually get deployed in production in 2026 — LiteLLM, OpenRouter, Portkey, and Vercel AI Gateway — and shows the killer workflow: point Claude Code at your own gateway so a single proxy handles routing, budgets, logging, and fallback for the whole team.

What you'll learn
  • Understand what an AI gateway is and the five problems it solves (many providers, one API; fallback; virtual keys; spend caps; observability)
  • Compare LiteLLM, OpenRouter, Portkey, and Vercel AI Gateway on latency, pricing, self-hostability, and where each shines
  • Wire Claude Code through your own LiteLLM proxy with ANTHROPIC_BASE_URL and a virtual key so the team gets shared limits and logs
  • Configure OpenRouter fallback so a Claude outage silently promotes GPT or Gemini instead of showing users a 5xx
  • Understand the March 2026 LiteLLM supply-chain incident and how to pin versions safely in production

The problem: one direct SDK per provider does not scale​

The first Claude integration is a two-line change: pip install anthropic, set ANTHROPIC_API_KEY, done. The second — say you want to fall back to GPT-5.4 when Anthropic throttles — is where the abstraction breaks. Now you have two SDKs with different request shapes, two dashboards, two bills, two rotation cadences for API keys, and two sets of retry logic. Add a third for Gemini and a fourth for your local Ollama, and every product decision ("cap this team at $500/month", "log every prompt for review", "let a customer bring their own key") becomes N implementations instead of one.

An AI gateway concentrates that plumbing in a single place. Concretely, a production gateway gives you:

  • One request shape for every provider. Most gateways speak the OpenAI Chat Completions API (or Anthropic Messages, or both) and translate to the real provider under the hood.
  • Fallback and routing. Try Claude first; on 429 or 5xx, retry against GPT or Gemini without the caller knowing. Same for latency ceilings and content-moderation rejections.
  • Virtual keys. Issue a per-user or per-service key that maps to a subset of models, its own budget, and its own rate limit — so a rogue script can't drain the whole account.
  • Spend caps and logging. Every request is tagged, priced, and stored. You can revoke a key without touching Anthropic or OpenAI, and you can prove to compliance what was sent where.
  • Caching. Prompt caching (exact-match) and semantic caching (near-match) turn repeat traffic into free hits.

Not every team needs all five. But the moment two of them are on your roadmap, running a gateway is cheaper than reinventing them per provider.

The four gateways that ship in production​

There isn't one "winner" — the four leaders occupy different corners of the design space (self-hosted vs. hosted, open-source vs. proprietary, minimalist vs. control-panel).

GatewayDeployPricing modelBest atNot for
LiteLLMSelf-hosted (Docker) or SDKFree (OSS); Enterprise tier for SSO/auditTeam proxy with virtual keys, budgets, no per-token markup, works with 100+ providers via one configTeams with no DevOps to run Postgres + Redis
OpenRouterHosted onlyProvider price + ~5.5% credit-purchase fee, no per-request markupZero-ops access to 300+ models under one key; ideal for products that let users pick a modelCompliance shops that need self-hosting or data residency
PortkeyOSS gateway (npx) or hosted cloudOSS free; cloud has usage tiersSub-ms gateway latency, semantic caching, guardrails, canary testing — the "control panel" angleTeams who just want the simplest possible key aggregator
Vercel AI GatewayHosted onlyProvider price, no token markup; free with Vercel plansDevs already on Vercel who want AI SDK v5/v6 + Anthropic Messages + OpenAI Responses APIs unifiedNon-Vercel infrastructure or air-gapped deployments

The important axis to pick on first: self-hosted vs. hosted. If your data cannot leave your VPC (regulated industries, EU residency, enterprise privacy reviews), you need a self-hostable gateway — LiteLLM or Portkey OSS. If you'd rather pay someone to run it, OpenRouter or Vercel AI Gateway is a one-click affair.

The second axis: how much control plane you actually need. If you're a one-person product that just wants to try Kimi K3 and Claude and Grok side-by-side without three signups, OpenRouter is the whole story. If you're a 20-person org where finance wants monthly spend by team, security wants virtual keys with rotation, and platform wants Grafana metrics, you're building on LiteLLM or Portkey.

Killer workflow: point Claude Code at your own LiteLLM proxy​

The best-kept secret about Claude Code is that it respects ANTHROPIC_BASE_URL and ANTHROPIC_AUTH_TOKEN. Set them to your gateway and Claude Code stops talking to api.anthropic.com directly — it talks to your proxy, which forwards to Anthropic (or anywhere else) with the auth you control. For a team, this changes three things at once:

  • One shared virtual key per developer. You issue and revoke keys in the proxy UI. No shared root credentials in .env files.
  • Per-developer budgets and logs. The proxy tags every request, so "who spent the $300 yesterday" is a database query, not an incident.
  • Model aliasing. You can pin claude-sonnet-4-6 at the proxy so a model deprecation is a one-line config change, not a repo-wide grep.

Start a minimal proxy in three steps:

Guided walkthrough1 of 3
  1. In a fresh venv or via uv: uv tool install 'litellm[proxy]'. This pulls in the gateway server (FastAPI + admin UI) alongside the client SDK.

The config file that makes this work:

config.yaml — Claude Sonnet/Opus/Haiku behind LiteLLM

model_list:
- model_name: claude-opus-4-7
  litellm_params:
    model: anthropic/claude-opus-4-7
    api_key: os.environ/ANTHROPIC_API_KEY
- model_name: claude-sonnet-4-6
  litellm_params:
    model: anthropic/claude-sonnet-4-6
    api_key: os.environ/ANTHROPIC_API_KEY
- model_name: claude-haiku-4-5-20251001
  litellm_params:
    model: anthropic/claude-haiku-4-5-20251001
    api_key: os.environ/ANTHROPIC_API_KEY

litellm_settings:
master_key: os.environ/LITELLM_MASTER_KEY
# Optional: enable exact-match prompt caching
cache: true
cache_params:
  type: redis
  host: os.environ/REDIS_HOST

Then, from any developer's shell:

Point Claude Code at the proxy (per-developer .env)

export ANTHROPIC_BASE_URL="https://llm.internal.example.com"
export ANTHROPIC_AUTH_TOKEN="sk-team-alice-9f4c..."   # a VIRTUAL key issued by the proxy

# now every Claude Code call goes through YOUR gateway
claude --model claude-sonnet-4-6

The non-obvious win is the virtual key. The master key is admin-only and never ships to laptops. Each developer gets a virtual key that maps to only the models you allow, has its own monthly budget, and can be revoked in seconds without rotating the underlying Anthropic key. If a laptop is lost, you kill one row in Postgres — not the whole team's access.

Careful: the same env vars work with Anthropic's Bedrock and Vertex integrations, but there are edge cases with experimental beta features. For Bedrock deployments the LiteLLM docs recommend setting CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1 in ~/.claude/settings.json to avoid header-compatibility issues.

Killer workflow #2: silent fallback with OpenRouter​

If you don't want to host anything, OpenRouter's fallback array is the shortest path to "silently retry another model when Claude 429s". You send an ordered list; OpenRouter walks it top-down and returns the first model that answered.

Claude → GPT → Gemini fallback in one request (OpenRouter)

import openai

client = openai.OpenAI(
  api_key="YOUR_OPENROUTER_KEY",
  base_url="https://openrouter.ai/api/v1",
)

response = client.chat.completions.create(
  model="anthropic/claude-sonnet-4.5",
  extra_body={
      # Ordered fallback. If the first model 429s, is down, or is
      # rejected by moderation, OpenRouter tries the next one.
      "models": [
          "anthropic/claude-sonnet-4.5",
          "openai/gpt-5.4",
          "google/gemini-2.5-pro",
      ],
  },
  messages=[{"role": "user", "content": "Explain B-trees in one paragraph."}],
)

# 'model' in the response tells you which one actually answered.
print(response.model, "->", response.choices[0].message.content)

Three things people miss on their first try:

  • Billing follows the model that answered, not the one you asked for. If Claude fails and GPT-5.4 answers, you pay OpenRouter's GPT-5.4 rate for that request.
  • The fallback triggers on more than 5xx. Rate-limiting, provider downtime, context-length validation errors, and content-moderation refusals all promote to the next model. That last one is the sharpest edge — a "moderation" refusal from one provider can silently route to a more permissive one, which may or may not be what you want. Review your fallback list with the same care as an ACL.
  • You cannot mix models with fallbacks. The Anthropic-format Messages endpoint uses a different fallbacks array. Sending both keys in the same request returns a 400. Pick the format your client speaks and stick to it.

The March 2026 LiteLLM supply-chain incident: what to actually do​

On 24 March 2026 at 10:39 UTC, two malicious PyPI releases of LiteLLM — v1.82.7 and v1.82.8 — were published by an attacker after they stole the maintainer's PyPI credentials via a prior compromise of Trivy, a security scanner running in LiteLLM's CI/CD pipeline. PyPI quarantined the packages at 13:38 UTC (about three hours later). During the exposure window, tens of thousands of downloads occurred. The payload was an infostealer with a persistence mechanism (a litellm_init.pth file that ran on every Python invocation, harvested credentials, and installed a systemd backdoor). Attribution goes to a group tracked as TeamPCP, which also compromised Trivy and Checkmarx KICS.

If you run LiteLLM in any environment, apply this once and then keep it in your platform playbook:

Guided walkthrough1 of 5
  1. v1.82.6 and earlier are clean. v1.83.0 and later (published via LiteLLM's rebuilt CI/CD v2 pipeline) are clean. Anything in between should be uninstalled and the environment considered tainted. The official Docker image (ghcr.io/berriai/litellm) was NOT compromised — the incident was PyPI-only.

The wider lesson isn't "don't use LiteLLM" — it's "assume every dependency in your AI stack, including security scanners, can be a delivery vector." Pin versions, sign images, and put your gateway on a network segment that only reaches the model providers.

Choose the right gateway for your situation​

Guided walkthrough1 of 5
  1. One or two providers with a small team → skip the gateway; direct SDKs are fine. Three+ providers OR a team where 'who has the key' matters → gateway. If you have DevOps and privacy requirements, self-host LiteLLM or Portkey OSS. If you'd rather pay someone else to run it, OpenRouter (hosted-only) or Vercel AI Gateway (great if you already deploy there).

Common combinations that ship in production:

  • Solo dev / prototype: OpenRouter direct. One key, 300+ models, done.
  • Small team, Claude-first: LiteLLM proxy with Anthropic + one fallback provider, virtual keys per engineer, Redis prompt caching.
  • Vercel-native product: Vercel AI Gateway with the AI SDK; add OpenRouter as a provider-options fallback for exotic models.
  • Regulated / EU: Self-hosted LiteLLM or Portkey OSS in-VPC with Presidio PII masking in front (see Claude + Local Models for the redaction pattern).
  • AI-product with heavy repeat traffic: Portkey (semantic caching commonly drives 30–50% cost reduction on chat-style workloads, per Portkey's own case studies — verify on your traffic before believing headline numbers).

What a gateway does NOT solve​

Gateways are middleware — they change how you reach models, not which model is right. Two things still need real work:

  • Prompt portability. Claude, GPT, and Gemini answer the same prompt differently, and system-prompt conventions vary. A gateway doesn't rewrite your prompt for the fallback provider — that's what Porting prompts across models and Cross-AI translation are for.
  • Evals. The gateway makes it easy to A/B two models on the same request. It cannot tell you which one was actually better on YOUR task. Run a real eval (see Evals) before switching defaults.

A common mistake is to install a gateway and consider "multi-model" done. The gateway is the transport layer; portability and evals are the product layer.

Check yourself

0/5
  1. What is the primary reason to put an AI gateway between your app and the model providers, once you use more than two?
  2. You point Claude Code at a LiteLLM proxy with ANTHROPIC_BASE_URL. What must ANTHROPIC_AUTH_TOKEN be?
  3. OpenRouter's fallback array promotes to the next model when the first one fails. Which of these DOES it trigger on?
  4. You need to install LiteLLM in production. Which versions are safe post the March 2026 incident?
  5. You're a solo dev on Vercel who wants to try Claude, GPT, and Gemini in one afternoon. Best fit?
AI gateways at a glance
Press Enter or Space to flip the card. Use the left and right arrow keys to move between cards.Term shown.
1 / 9
Key takeaways
  • An AI gateway is the missing router between your app and every model — it exists to make virtual keys, budgets, fallback, logging, and caching a single implementation instead of N per provider
  • Pick on TWO axes first: self-hosted vs. hosted (LiteLLM/Portkey OSS vs. OpenRouter/Vercel), and minimalist vs. control-panel (OpenRouter/Vercel vs. LiteLLM/Portkey)
  • The Claude Code killer workflow: point ANTHROPIC_BASE_URL at your own LiteLLM proxy and issue per-developer virtual keys — team gets shared limits, logs, and one-click revoke without touching the root Anthropic key
  • OpenRouter's models[] array is the shortest path to silent Claude → GPT → Gemini fallback, but note that moderation refusals are a fallback trigger too — review the list like an ACL
  • After the March 2026 LiteLLM supply-chain attack, pin to v1.82.6 or earlier, or v1.83.0+; prefer the signed Docker image over pip; egress-allowlist the proxy
  • A gateway is transport, not product — prompt portability and evals still need real work, no matter how many models you can now reach

Sources & further reading​

Related on this site: Claude + Local Models: Hybrid Patterns · Porting prompts across models · Cross-AI translation · Evals · What AI costs across providers · Gray-market AI proxies ("Poison Claude") — the same routing mechanism, weaponized