Skip to main content

Cost & Latency Tradeoffs

Intermediate
What you'll learn
  • The cost/quality/speed triangle — why you can never max all three at once
  • The six biggest levers to spend where it matters and save everywhere else
  • How a cheap-first cascade routes 90% of volume to a small model and cuts cost ~70% without dropping quality on the hard cases
  • The latency-specific wins (streaming, parallelism, caching, async) that reshape perceived speed
  • Why 'optimize blind' burns quality — measure first, guard with evals second

Quality, cost, and speed pull against each other. You can't max all three at once — but you can spend each where it matters and save everywhere else.

The triangle​

A bigger model is smarter but slower and pricier; a smaller one is fast and cheap but less capable. Good engineering is routing each task to the right point on this triangle.

The biggest levers (roughly in order)​

Guided walkthrough1 of 6
  1. Don't run Opus for classification. Start with Sonnet, drop to Haiku for simple/high-volume steps, reserve Opus for the hard parts. Biggest single lever — see /docs/api/choosing-a-model.

Worked example: a cheap-first cascade​

Cascading is the lever people skip because it sounds vague. Make it concrete. Say you need to triage 100,000 support emails. The naive approach runs every email through your strongest model. A cascade routes most of the volume to a cheap model and escalates only the hard cases:

Suppose the cheap model resolves 90% of cases on its own and a stronger model costs roughly 5× as much per token. Counting in relative cost units (1 = one cheap pass over one email):

  • All-strong: 100k × 5 = 500k cost units.
  • Cascade: 100k × 1 (every email gets the cheap pass) + 10k × 5 (the 10% that escalate) = 150k cost units.

That's about a 70% cut, and the genuinely hard cases still get your best model. The multipliers and split are illustrative — plug in your own token counts and rates and measure the real escalation rate with evals. The lesson holds regardless: the savings come from how rarely you pay the expensive rate, not from the rate itself.

Latency-specific wins​

Pro tip
  • Stream responses — users see the first tokens instantly, so perceived speed jumps even when total time is unchanged (/docs/api/streaming).
  • Parallelize independent sub-calls — a request that fans out to three tools finishes in max(latency), not sum(latency).
  • Cache repeated work and precompute where you can — the fastest tokens are the ones you don't have to generate.
  • Pick a smaller model for the interactive path; move heavy lifting async so the user isn't waiting on it.
  • Reduce max output tokens for interactive endpoints — long generations are the dominant tail-latency source.

Don't optimize blind​

Measure first: where are the tokens and the seconds actually going? Then optimize the biggest line item. And re-check quality with evals after any cost cut — a cheaper setup that's wrong isn't cheaper.

Check yourself

0/4
  1. Which is USUALLY the single biggest cost lever?
  2. In a cheap-first cascade, where do the savings actually come from?
  3. You want the user to feel a response is faster. Which lever helps MOST without changing model choice?
  4. You cut costs by moving from Sonnet to Haiku on a workflow. What's the required next step?
Key takeaways
  • The triangle is real: quality, cost, and speed pull against each other — engineering means routing each task to the right point, not maxing all three.
  • Right-sizing the model is the biggest single lever; cascades multiply it by only paying flagship rates on the hard minority.
  • Prompt caching, tight max_tokens, and RAG-driven input trimming compound with model choice.
  • Latency-perception is a separate axis — streaming, parallel sub-calls, and async heavy lifting reshape UX without changing raw cost.
  • Every cost cut must be guarded by evals — a cheaper setup that's wrong isn't cheaper.

Next​