跳到主要内容

Supabase Evals — Real-Backend Agent Benchmark

进阶

Coding-agent leaderboards mostly measure pull-request diffs against a test suite — SWE-Bench and its cousins. That's useful, but it's not what a Supabase user actually does: build a schema, fix a broken RLS policy, debug a failed Edge Function, wire up auth. On August 1, 2026, Supabase open-sourced a benchmark that grades coding agents on those real tasks, running them against actual containerized Supabase stacks with real MCP servers and real CLIs. It's Apache-2.0, runs locally on a laptop, and its early scores already say something surprising about how different agents use documentation. This page is the practical field guide to it.

What you'll learn
  • Understand what makes Supabase Evals different from SWE-Bench-style benchmarks
  • Read the cross-agent findings honestly — where Opus 5, Sonnet 5, Kimi K3 and the GPT-5 line actually diverge
  • See the effect of loading skills on the same model (before vs after)
  • Run the benchmark yourself against your own agent and model in under an hour

What Supabase Evals actually is​

Supabase Evals is a benchmark, an evaluation harness and a public leaderboard rolled into one Apache-2.0 repo at github.com/supabase/evals. It exists because SWE-Bench-style tests answer the wrong question for backend work — they measure "can this agent close a filed GitHub issue against a Python test suite" — while most real Supabase work is closer to ops on a running system: your RLS policy started rejecting valid requests, your Edge Function is 500ing in prod, your migration didn't take.

Three things separate it from what came before:

  • Real stacks, not mocks. Every scenario boots a fresh Docker sandbox that mounts the host Docker socket, so supabase start spins up the local stack as sibling containers. The agent talks to a Management-API-compatible surface (@supabase/lite) and real MCP servers and the real Supabase CLI. Nothing is stubbed.
  • Deterministic checks first, LLM-as-judge only where needed. A scorer can query the resulting database, hit the deployed Edge Function, or run repo-root vitest on the produced files. The LLM judge is a backstop for freeform answers ("did the agent's explanation match the actual root cause?"), not the primary grade.
  • One retry, then it's final. No cherry-picked best-of-N. This alone makes the numbers less flattering than most agent PR benchmarks.

The scenario grid​

The August 2026 release ships 38 eval scenarios under evals/<id>/. They're organized on three axes so results can be sliced meaningfully:

  • Nine products — database, auth, storage, edge functions, realtime, cron, queues, vectors, data-api
  • Nine topics — RLS, security, migrations, SQL, SDK, observability, self-hosting, tests, declarative-schema
  • Four stages — build (make something new), deploy (ship it), investigate (figure out what's wrong), resolve (fix it)

Each eval is a directory with a PROMPT.md (the task the agent sees, plus frontmatter metadata), an EVAL.ts scorer, and optional remote/ (seeded hosted-project state — SQL, logs, already-deployed functions) and local/ (starting files copied into the agent's sandbox workspace). The scenarios come from real support tickets and GitHub issues, not synthetic prompts.

Two runtimes, chosen per eval​

Every eval runs in one of two modes, decided automatically by its shape:

  • Tools mode. No sandbox. The agent gets an MCP/tool surface (Management API, SQL, storage, etc.) and its answer is scored by inspecting resulting project state. Used for evals that live entirely inside the platform (no local/ directory, no interface: cli).
  • Local-stack mode. A fresh Docker sandbox with the real Supabase CLI installed. The agent gets a bash tool plus file tools, and can run supabase init/start/db/test against a real local stack that spawns as sibling containers. After the agent stops, the harness docker cps the workspace out and runs vite/vitest from the host to score it. Used when a local/ workspace exists or interface: cli is declared.

The clever bit: the host's toolchain scores what the sandbox produced. Scorers don't need to live inside the sandbox image.

Cross-agent findings (August 2026)​

The initial leaderboard runs pit first-party agents (Claude Code, Codex, Claude API executor experiments) against model-agnostic OpenCode across the Claude Opus 5, Claude Sonnet 5, Kimi K3 and GPT-5.4/5.6 lineups. Numbers below are for the build stage, which is where the initial post reports the sharpest results.

Agent + modelBuild (no skills)Build (with skills)
Claude Code + Opus 5100%100%
OpenCode + Kimi K3100%100%
Claude Code + Sonnet 578%100%
Codex + GPT-5.6 (Sol)89%100%
Codex + GPT-5.4 mini78%89%

Two things stand out:

  • Opus 5 and Kimi K3 max out build with no help. Kimi K3 running through OpenCode matching Opus 5 running through Claude Code is a real signal about open-weight models catching up on backend-shaped work — not a general SOTA claim, but a specific, reproducible one.
  • Skills are a large multiplier on smaller/mid models. Loading the Supabase skill takes Sonnet 5 from 78% to 100% on build, and takes the Postgres-best-practices skill's hit rate from ~10% to 60% across the wider suite. If you're running Sonnet 5 (or any mid model), not loading the relevant skill is leaving 20+ points on the floor.

The finding nobody expected: models use docs very differently​

The single most interesting number in the release isn't a score, it's a behavior:

Codex / GPT-5.6 reads roughly 8 docs pages per scenario. Claude Code reads about 2 — and consults docs in under 40% of scenarios even with skills loaded.

The two agents were staring at the same tasks, the same MCP tools and the same skills. Codex reflexively pulls documentation before acting; Claude Code more often tries from memory and only reads docs when it hits a wall. Neither is "right" — Opus 5 through Claude Code still scored 100% on build. But it's a very different mode of operation, and it explains a lot of anecdotal reports about the two agents feeling different.

★ Insight ─────────────────────────────────────

  • Benchmark reports usually collapse an agent to one number. Supabase Evals is unusually valuable because it instruments the agent's behavior — how often it reads docs, which skills it loaded, how many tools it called — and that behavior is often the story, not the score.
  • The gap here isn't a model gap, it's a harness policy gap. The same base model in a different harness would likely behave differently. See Coding Agent CLIs Compared for why the harness matters at least as much as the model.
  • Deterministic scorers plus one retry (no best-of-N) is what makes the numbers here comparable across agents. Most public agent leaderboards silently allow retries or self-consistency, which flatters slower/weaker models. ─────────────────────────────────────────────────

Three consistent stumbles across every agent​

Regardless of model, every tested agent tripped on the same three patterns:

  • Hand-writing migrations instead of using declarative schemas. The declarative-schema workflow is newer; the training data is heavy with old-style create table migrations, so agents default to those even when the project is set up for declarative.
  • Verifying auth manually instead of using @supabase/server. Agents wire up cookie-parsing and session-checking by hand rather than using the newer server-side helpers that exist for exactly this.
  • Inconsistent docs consultation. As above — the same agent will look up docs in one scenario and try from memory in the next, with no obvious rule for which.

These are the sort of failures that don't show up in pass@1 on a synthetic benchmark, because there's no "wrong but working" answer to grade — the code compiles, it just uses the outdated pattern. Real-container evals catch them because a scorer can inspect the produced project and check "did they use declarative schema, yes/no."

Running it locally​

You need Docker, pnpm, and an API key for whichever provider you want to grade. Ports 54321–54329 must be free (stop any local supabase start stacks first). The whole thing runs on a laptop; a small run finishes in minutes.

Guided walkthrough1 of 5
  1. The skills live in a submodule — a plain clone will leave you missing the Supabase agent skills. Use --recurse-submodules the first time.

Clone the repo with submodules

git clone --recurse-submodules https://github.com/supabase/evals.git
cd evals
pnpm install
cp .env.example .env

Run a single eval against a single experiment

pnpm eval -- \
--eval resolve-dataapi-001-empty-results \
--experiment claude-code-sonnet-5

Head-to-head: same model, skills on vs off

pnpm eval -- \
--experiment claude-code-sonnet-5 \
--experiment claude-code-sonnet-5-no-skills \
--suite benchmark

Full benchmark sweep across all shipped experiments

pnpm eval -- \
--suite benchmark \
--experiment-suite benchmark,no-skills

pnpm export-results
pnpm web

Anatomy of an experiment (so you can add your own)​

An experiment file under experiments/<name>.ts is where you wire up an agent, a runtime and a model into one comparable configuration. That separation is deliberate: it means "Claude Code with Sonnet 5 and no skills" is a different experiment from "Claude Code with Sonnet 5 and skills loaded", and both are different from "OpenCode with Kimi K3". You can hold two of the three axes constant and vary the third.

The -no-skills variants shipped in the repo (e.g. claude-code-sonnet-5-no-skills.ts vs. claude-code-sonnet-5.ts) are what makes the "skills lifted us from 78% to 100%" claim measurable rather than a vibe.

If you're running a custom agent — a homegrown loop, an internal fork, an experimental harness — the pattern is: write an experiment file that declares your agent, opt into localStack: localStackRuntime() if you want the sandbox evals too, and run it against the benchmark suite. Your numbers land in the same table as everyone else's.

Skills, loaded lazily​

Both runtimes use progressive disclosure for skills: only each skill's name and description sits in the system prompt at all times; the agent pulls the full instructions on demand. That's why "skills on" costs only a few extra tokens per turn but can lift a score by 20+ points — the agent only pays for a skill's body when it decides the skill is relevant.

The two runtimes differ only in how the body is fetched:

  • Local-stack (sandbox) mode. Skills are installed into the sandbox with Vercel's skills CLI (baked into the sandbox image, sourced from the local skills/ directory — never the network) under .claude/skills/. When a task matches, the agent reads .claude/skills/<name>/SKILL.md and any files it references with its file tools.
  • Tools mode. No filesystem, so a load_skill tool returns a skill's full instructions when the agent calls it by name.

For context on the emerging cross-agent skills standard this uses, see SKILL.md as an Open Standard.

Where this fits in the AILmanac map​

★ Insight ─────────────────────────────────────

  • Benchmarks like this are most useful directionally, not absolutely. "Sonnet 5 gained 22 points with the Supabase skill loaded" transfers to your own project; "Sonnet 5 scores 100% on build" doesn't necessarily.
  • The right question after reading a leaderboard isn't "which agent won", it's "what does this benchmark test that resembles my work" — and then only the sub-scores on those slices matter. ─────────────────────────────────────────────────

Quick self-check​

Check yourself

0/5
  1. What makes Supabase Evals structurally different from SWE-Bench-style benchmarks?
  2. The August 2026 release reports Codex/GPT-5.6 and Claude Code differ sharply in one *behavior* on the same scenarios. Which?
  3. Loading the Supabase skill lifts Sonnet 5's `build` score from 78% to 100%. What does this most strongly suggest?
  4. You want to compare 'skills on' vs 'skills off' fairly for the same model. What's the right primitive in Supabase Evals?
  5. Which stumble is *not* one of the three consistent failures reported across every tested agent?

Flashcards​

按 Enter 或空格键翻转卡片。使用左右方向键在卡片之间切换。已显示术语。
1 / 7

Sources & further reading​