跳到主要内容

Model Routing Patterns: Cascades, Classifiers, and What Actually Ships

进阶

Once your product does more than one kind of thing, one model stops being the right answer for every request. Classify the ticket, extract the fields, draft the reply, review it — those are four jobs with four different cost/latency/quality budgets. The pattern the whole field has converged on in 2026 is to route each request to the model that fits the job, and to escalate when the cheap model isn't good enough. This page is the editorial guide to those patterns: what they are, when each one ships, the recipes that work, and the failure modes to avoid.

What you'll learn
  • Know the six routing patterns that ship in production and when to reach for each
  • Understand the difference between routing (one decision up front) and cascading (escalate on failure)
  • Build your first classifier router with a copy-paste prompt and a one-week rollout plan
  • Read the cost math: when routing saves real money, and when the overhead eats the savings
  • Recognize the anti-patterns that turn a router into an outage waiting to happen

Why routing is the default pattern in 2026​

The "one big model for everything" era is quietly over. Three forces drove the shift:

  • A wide, well-behaved price curve. Every major provider now ships a family — Anthropic (Haiku / Sonnet / Opus), OpenAI (small / mid / frontier), Google (Flash / Pro), plus open-weight tiers. Cheap tiers are 10–100× cheaper than the frontier and, on narrow tasks, only marginally worse. Leaving the cheap tier idle is money you're setting on fire.
  • Real latency budgets. A classify step in front of a support agent needs to answer in under 300 ms. A frontier model is often the wrong tool for reasons other than cost — it's just too slow for the step.
  • Specialization. Different models genuinely lead on different tasks — one is better at structured JSON, another at long context, another at code, another at multilingual. The rankings shuffle monthly (see Choosing a Model), but the shape — "different tools for different jobs" — is durable.

The result: a router in front of your models, deciding per-request where to send the work. The rest of this page is the design vocabulary for that router.

The two big families: routing vs cascading​

Almost every pattern below is a variation on two ideas. Get the difference right and the rest is naming.

  • Routing makes one decision up front, before any model runs. A classifier (a rule, a small model, or a bigger model) reads the request, picks a target, and hands off. Fast, cheap in the steady state, but wrong when the classifier is wrong — and you don't find out until later.
  • Cascading runs the cheap model first and escalates to a stronger one only when a defined signal says "this answer isn't good enough." Robust because the signal is grounded in the actual output, but every escalation pays both models — so the win only survives when the cheap tier handles most traffic on its own.

They compose. A production system usually does both: route to the right lane based on task type, then cascade inside a lane based on the cheap model's confidence. Anthropic's taxonomy calls the up-front decision "Routing" and the dynamic decomposition "Orchestrator-Workers"; cascading is the industry name for the "try small first, escalate on failure" variant of the same idea.

The six patterns that actually ship​

1. Rule-based routing​

A hand-written rule (regex, keyword, request field, message length) decides the target. No model runs to make the decision.

  • When it wins. The signal is unambiguous and cheap: "if the request contains a code fence, send to the coding model", "if the customer is on the Enterprise plan, send to the frontier tier", "if the input is > 200k tokens, send to the long-context model."
  • When it breaks. The rule silently rots as the domain shifts — new phrasing, new intents, edge cases the regex never anticipated. Every "if X then Y" is a small piece of tech debt.
  • Ship it when. You have one or two high-value carve-outs (a paying tier, a code path, a language) and want zero latency overhead.

2. Classifier routing (LLM-as-router)​

A small, fast model reads the request and returns a label — "billing" | "technical" | "sales" | "other" — which picks the downstream handler. Anthropic gives this as one of their canonical examples: easy questions to Haiku, hard ones to Sonnet.

  • When it wins. You have a small, stable set of categories and each one deserves its own prompt/tool set/model. A single specialized prompt per category beats one giant do-everything system prompt.
  • When it breaks. Categories overlap (many real requests are "billing and technical"), the classifier misroutes silently, or your label set explodes past ~10 categories and picks become noisy.
  • Ship it when. You can enumerate the top 5–8 request types and each one materially benefits from a different handler.

Classifier-router prompt (portable across Claude/GPT/Gemini)

You are a request classifier for a support product.

Read the CUSTOMER MESSAGE below and return a JSON object with:
- "category": one of ["billing", "technical", "account", "sales", "other"]
- "confidence": a number from 0.0 to 1.0
- "reason": one short sentence explaining the pick

If you are less than 0.7 confident, use "other" and say why.
Return ONLY the JSON, no prose, no code fence.

CUSTOMER MESSAGE:
"""
{{message}}
"""

3. Complexity-based routing​

Instead of what the request is about, you estimate how hard it is — length, entity count, whether it references numbers/code, whether tool use is likely — and pick a tier accordingly: cheap for easy, mid for medium, frontier for hard.

  • When it wins. The task type is roughly homogeneous (say, "answer a coding question") but the individual requests vary wildly in difficulty. Instead of paying frontier prices for a two-line syntax question, you pay Haiku prices for those and reserve the frontier for the multi-file refactors.
  • When it breaks. The "difficulty score" is really "prompt length", and long prompts aren't always hard prompts. A user pastes a huge stack trace and asks a trivial question about it — your router upgrades needlessly.
  • Ship it when. You have measurable variance in difficulty within a task type and clear signals (length, presence of code, number of subquestions) that correlate with it.

4. Cascade (cheap-first, escalate on failure)​

Try the cheap model first. Check the answer against a failure signal. If it fails, retry on the stronger model. This is the pattern the industry ships more than any other, because the "signal" is grounded in what the model actually said — not a guess about what it will say.

  • What counts as a signal? Anything cheap and reliable: schema validation on JSON output, a "does this answer the question?" check by an LLM judge, an explicit "needs_help": true field the model can emit when unsure, a downstream test that runs the code, a low logprob on the answer token when the provider exposes it.
  • Cost math. The savings survive only when the cheap tier handles most traffic. If 90% of requests get resolved by the cheap model and 10% escalate, you pay 0.9 × cheap + 0.1 × (cheap + strong) ≈ mostly cheap. If half escalate, you're paying more than always using the strong model.
  • When it wins. Tasks with a clear "did this work?" signal — code that runs or doesn't, JSON that validates or doesn't, extraction where the field either matches the source or doesn't.
  • When it breaks. No cheap failure signal, or the cheap model thinks it succeeded when it didn't (silent failure — the worst case).
  • Ship it when. You can name the failure signal in one sentence and it doesn't require the strong model to check.

5. Ensemble / verify-and-vote​

Run the same request against N models in parallel and reconcile the answers — take a majority vote, take the first schema-valid one, or send all N to a judge model that picks the best.

  • When it wins. Quality matters more than cost or latency: legal research, medical summarization, high-stakes financial extraction, "review this contract for red flags." Also useful for hard math and code, where different models make different mistakes and their intersection is more reliable than any single one.
  • When it breaks. You pay N× the cost and inherit the slowest model's latency for every request. Ensembles are a tax you accept for quality, not a way to save money.
  • Ship it when. The cost of getting the answer wrong is at least an order of magnitude larger than the cost of N model calls — which is almost never true for consumer traffic and often true for enterprise workflows.

6. Fallback (availability, not cost)​

Try the primary; on 429/5xx/timeout, transparently retry on a secondary from a different provider. This is the pattern every serious multi-model app needs even if it uses none of the others.

  • When it wins. Every time your primary provider has an outage or throttles you at exactly the wrong moment. Fallback is a reliability pattern, not a cost-optimization pattern — the secondary is usually a different provider's equivalent tier, not a cheaper model.
  • What to watch. The fallback path is untested code most of the time; it will regress. Run at least a small share of live traffic through the secondary continuously so you find out the response shape drifted before you're relying on it in an outage.
  • Ship it when. Your uptime SLO is higher than any single provider's, or a single-provider dependency is a business risk (contracts, region availability, geopolitics).

Comparison at a glance​

PatternDecision madeAdds latency?Adds cost?Main risk
Rule-basedBefore the model runsNoNoRules silently rot as inputs shift
ClassifierBefore, by a small model+ one fast call+ one cheap callMisroutes when categories overlap
Complexity-basedBefore, by a heuristicNegligibleNegligible"Difficulty" often means "length"
CascadeAfter the cheap model tries+ retry when escalatingMostly-cheap in steady stateSilent success on wrong answer
EnsembleRuns N in parallel, reconcilesSlowest-of-NN×You are buying quality, not saving money
FallbackOnly on primary failure0 in the happy path0 in the happy pathUntested until it matters

Real ship recipes​

The patterns above are Lego. In production they compose. Three recipes we see repeatedly:

  • Customer support agent. Classifier router picks a lane (billing / technical / account / sales), each lane has its own system prompt and tool set, inside the technical lane a cascade tries a cheap model first and escalates to the frontier when a "needs escalation" tool call fires. Fallback across providers wraps everything. Result: 70–90% of traffic never touches the frontier model.
  • Coding assistant. Rule-based routing on file type and diff size sends small edits to a cheap tier and multi-file refactors to a coding-specialist. A cascade on the output — "does the patch apply cleanly and pass the smoke test?" — escalates failures to a stronger model. Compare against the field guide in Claude vs GPT vs Gemini for coding.
  • RAG QA over a document corpus. Classifier picks between "answer from context" (cheap) and "needs cross-document reasoning" (mid). An ensemble of two models cross-checks the answer for high-stakes documents (contracts, filings). Compare with Retrieval-Augmented Generation.

How to build your first router in a week​

Guided walkthrough1 of 7
  1. Instrument the *real* prompts your product already sends. You need at least a few hundred, ideally categorized by outcome (resolved / escalated / wrong). Without ground-truth traffic you're guessing at where the router should send things.

What actually gets deployed (2026 field notes)​

  • Cascades ship far more than learned routers. Research systems like RouteLLM train a router on preference data and can cut cost 2× or more on standard benchmarks (see the RouteLLM paper) — but training and maintaining a learned router is real engineering. Most teams get most of the wins from cheap-first + explicit escalation.
  • The classifier is almost always a cheap chat model, not a fine-tune. Haiku-class or Flash-class models with a good prompt hit the accuracy bar for 5–8 categories. Fine-tune only if you're at scale and the cheap classifier is your bottleneck.
  • The evaluator matters more than the router. A router without a metrics loop is a guess. Instrument every routing decision with request → chosen model → outcome, and review weekly. You can't tune what you don't measure.
  • Infrastructure and design are separable. The patterns above are language- and provider-neutral. The plumbing — one endpoint per provider, virtual keys, per-team spend caps, prompt caching — is what an AI gateway gives you. Once you know the pattern you want, see AI gateways: LiteLLM, OpenRouter, Portkey, Vercel for the concrete choice.

Anti-patterns to avoid​

  • The classifier that calls the frontier model. If picking a lane costs as much as running the strong model would have, you saved nothing and added latency. Classifiers must be cheap.
  • Cascade without a failure signal. "The cheap model returned something, so we're done" is not a signal — it's a bet. Every cascade needs a defined check the cheap model can fail.
  • Rule proliferation. Ten rules is manageable. A hundred is a hand-coded classifier without the tests. When your rule set drifts past ~15 branches, rip it out and use a small model.
  • Ensembles as a cost strategy. Running three models in parallel to save money is a category error — ensembles cost N× to buy quality, not to save cost. If you're using an ensemble to make up for a bad primary model, fix the primary instead.
  • No fallback path. Every model provider goes down eventually. A single-provider product will inherit that outage. See AI gateways for the plumbing.
  • Routing without observability. A router that quietly sends everything to the wrong lane looks fine until quality craters two weeks later. Log every decision with the input hash, the chosen model, the outcome, and the cost — and review weekly.

Check your understanding​

Check yourself

0/3
  1. You have a support product where 90% of requests are answered correctly by a cheap model, but the remaining 10% need the frontier. What pattern gives you the biggest cost win?
  2. Anthropic's 'Building Effective Agents' guide defines Routing as:
  3. You're building a classifier router. Your classifier prompt runs on a frontier model because 'accuracy matters'. What's the problem?

Next​

Sources​