Evaluating AI Quality (Evals)
Advanced
- Build the smallest useful eval — a golden set of 20–100 real inputs with clear pass criteria
- Pick the right metric per task: deterministic checks, LLM-as-judge, or human review
- Run evals as a gate — before/after every prompt change, model swap, and in CI
- Score every stage (retrieval, tool call, final answer), not just the last one, to localize regressions
If you ship anything built on AI, evals are how you know it works — and how you know a change made it better, not worse. Without them you're flying blind: a prompt tweak that helps one case can silently break ten others. Evals turn "vibes-based" iteration into a measurable loop.
The minimum viable eval
You don't need a framework to start. The whole loop is four steps:
Guided walkthrough1 of 4
- 20–100 real inputs with the correct or acceptable outputs (or clear criteria for what counts as a pass). Cover the easy cases, the tricky ones, and the edge cases that already bit you in production.
- Exact match? Contains a specific key fact? Valid JSON that fits your schema? No hallucinated numbers? On-brand tone? Write the pass criteria down — one line per case is enough.
- Run your current setup against the set and record the score. This is now the number to beat. Save it — future you will need it.
- Prompt tweak, model swap, retrieval change — one variable at a time. If the score goes up and nothing regresses, keep the change. If it doesn't, revert. This is the entire loop.
Choosing metrics
Not every question deserves the same test. Match the metric to the task:
| Task type | Metric | Cost | Trustworthiness |
|---|---|---|---|
| Structured output (JSON, SQL) | Schema validation, exact-match | Free | High |
| Code generation | Run the tests it wrote | Cheap | High |
| Extraction (dates, entities) | String contains / regex | Free | High |
| Classification | Accuracy, precision, recall | Free | High |
| Summaries, drafts, tone | LLM-as-judge with a rubric | Moderate | Medium — must calibrate |
| High-stakes writing, safety | Human review on a slice | Expensive | Highest |
Three rules of thumb:
- Deterministic checks where possible. If the answer is "valid JSON matching this schema" or "the code passes these tests," don't ask an LLM — just check.
- LLM-as-judge for fuzzy quality. Helpfulness, tone, factuality-vs-source. Cheap, fast, but has biases (length, position, self-preference). Validate the judge against human ratings on a sample before you trust its numbers.
- Humans on the highest-stakes slice. Even 10 human-graded cases per release beats 0.
LLM-as-judge starter rubric
You are grading assistant responses against a source document.
For each response, output a JSON object with these fields:
- grounded: true if every factual claim is supported by the source, else false
- complete: true if the response answers all parts of the question, else false
- concise: true if the response contains no filler or repetition, else false
- overall_score: 1-5 (1 = unusable, 5 = ship it)
- reasoning: one sentence explaining the score
Do not consider length. Do not consider whether the response comes first or second.
<source>{{SOURCE}}</source>
<question>{{QUESTION}}</question>
<response>{{RESPONSE}}</response>
Return only the JSON, no preamble.When to run them
- Before/after any prompt or model change. No exceptions. The whole point of the golden set is to catch the change you thought was safe.
- On model migration. New models shift behavior — sometimes silently. Run evals before you flip the model ID. See Errors & Migration.
- In CI, for production systems. Turn a green eval into a merge gate. Regressions get caught before users see them.
- After every reported bug. Add the failing case to the golden set. The set grows as the system does.
Eval every stage, not just the final answer
For RAG and agents, a single "final answer" score hides where things break. Score each stage separately so regressions land in one place:
| Stage | What to check |
|---|---|
| Retrieval | Did the top-k include the doc that contains the answer? |
| Tool selection | Did the agent pick the right tool for this turn? |
| Tool arguments | Were the arguments well-formed and correct? |
| Final answer | Does it match the golden criteria? |
When retrieval score drops but final-answer score holds steady, you know it's a retrieval regression — not a prompt problem. That kind of localization is what makes evals actually useful for debugging, not just scoring.
Common mistakes
- Grading with the same model that generated the answer. Self-preference bias inflates scores. Use a different model — or better, a different family — as judge.
- Judge prompts that leak the "right" answer. If the judge sees the gold answer, it will find reasons to reward matching it. Judge on criteria, not on similarity.
- Golden sets frozen at day one. The set should grow every time production surprises you. A stale set measures the past.
- Optimizing to the eval instead of the task. If you tune prompts against a small set until every case passes, you may have overfit. Hold out ~20% of cases as a validation slice you never look at during iteration.
Check yourself
0/4- 20–100 real inputs + pass criteria + one variable at a time = a working eval loop.
- Deterministic > LLM-judge > human — pick the cheapest metric the task allows.
- Score every stage in a pipeline, not just the final answer.
- The golden set grows with every reported bug. A stale set measures the past.
Next
- Evaluating Your AI Agent — the deeper playbook: trajectory scoring, LLM-judge calibration, CI gate
- Message Batches — Async Jobs at 50% Off — the right way to run a 5,000-prompt eval sweep for half the cost
- Hallucinations & How to Reduce Them
- Building Agents on the API
- Choosing a Model & Provider