Skip to main content

Choosing a Model & Provider

Intermediate

How do you pick among models and providers without getting lost in hype? With a simple, evergreen process — because the specific leaderboard changes monthly, but the way to choose doesn't.

What you'll learn
  • Read a public benchmark as a shortlisting hint rather than a verdict, and know the three reasons it cannot settle your decision
  • Run a bake-off on your own real inputs — the single highest-value habit in model selection
  • Score candidates on the seven factors that actually decide the outcome, not just quality and price
  • Adopt a default posture that keeps switching cheap: mid-tier first, model behind config, re-evaluate on a schedule

Read benchmarks skeptically​

Public benchmark scores are a starting hint, not a verdict:

  • They can be gamed or contaminated (test data leaking into training).
  • They measure generic tasks, not your task.
  • Small score gaps rarely matter in practice.

Use them to build a shortlist, not to make the final call.

The only benchmark that counts: yours​

Run a tiny eval on a handful of your real inputs across 2–3 candidate models. It takes minutes and tells you what no leaderboard can. This "bake-off" is the single best habit in model selection.

Guided walkthrough1 of 6
  1. Use public leaderboards, price lists and capability tables to narrow the field — and stop there. The shortlist is all a benchmark is good for.

Turn a handful of real inputs into a reusable bake-off spec

I want to choose between these candidate models for one specific task: [list candidates].

The task: [describe it in one or two sentences].
Here are [N] real inputs from the workload, including [N] that currently fail: [paste inputs].

Help me build a small eval I can re-run:
1. Write the pass/fail condition for each input, in advance and in plain language.
2. Flag any input where "good" is ambiguous, and ask me to decide before we score.
3. Give me a single prompt to send to every candidate unchanged.
4. Propose a scoring table with one row per input and one column per candidate.
Do not rank the models yet — I will run them and bring the outputs back.

A decision scorecard​

Weigh what actually matters for your use case:

FactorAsk
Quality on your taskDoes the bake-off show it's good enough?
CostPer-token price at your volume (Cost & Latency)
LatencyFast enough for the experience?
CapabilitiesVision? Long context? Tool use? Structured output?
Privacy/complianceData handling, residency, certifications (Privacy)
Reliability & ecosystemUptime, SDKs, docs, support, migration story
Lock-inHow hard to switch later?

Practical posture​

  • Default to a capable mid-tier model and only move up/down on evidence.
  • Abstract the model behind config, not scattered literals, so switching is a one-line change (Errors & Migration).
  • Re-evaluate periodically — the frontier moves fast; today's best may not be next quarter's.

(For the Claude-specific tiers, see Choosing a Claude Model.)

Check yourself

0/3
  1. A new model tops the leaderboard you follow, two points above the one you use. What does this page say to do?
  2. Your bake-off shows candidate A slightly ahead of candidate B on quality. What is still missing before you choose A?
  3. Which practice does this page recommend specifically to keep a future switch cheap?
Key takeaways
  • Leaderboards build a shortlist. Your own eval makes the decision.
  • A bake-off on a handful of real inputs takes minutes and tells you what no public benchmark can.
  • Decide what 'good' means before you read any output, or the bake-off becomes a vibe check.
  • Quality is one of seven factors — cost, latency, capabilities, privacy, reliability and lock-in all decide outcomes too.
  • Mid-tier by default, model behind config, re-evaluate on a schedule: that posture keeps every later choice cheap.

Next​