跳到主要内容

Code Erosion: Why Your Agent Breaks What It Already Built

高级

Almost every coding benchmark you have read about asks the same question: given a task, can the agent solve it? One prompt, one patch, one verdict. That setup is convenient and it hides the failure mode that actually costs you money, because real software is never written once. The spec moves. You extend what you built last week. And the interesting question is not whether the agent can write the feature — it is what the agent does to the code that was already working.

SlopCodeBench exists to measure exactly that, and the results are worse than the discourse suggests. In the paper's own words: "Software development is iterative, yet agentic coding benchmarks hide design issues through their single-shot setup."

Two things made it worth a page this week. First, a third-party campaign report — "SlopCodeBench 10/2026 – Opus 5.5, Astra wins overall, xhigh considered harmful" — reached 260 points and 101 comments on Hacker News on 15 October 2026, following a 405-point post on the same benchmark in July. Second, almost nobody is quoting the numbers in it that matter most. The headline travelled; the regression statistic, the cost-per-pass arithmetic and the cleanliness/correctness inversion did not.

What you'll learn
  • Explain what SlopCodeBench measures that SWE-bench-style benchmarks structurally cannot — iterative specification refinement, with regression tests carried forward across every checkpoint
  • Read the two pass rates correctly: strict and core rank the same ten campaigns almost in reverse, and which one you quote decides your winner
  • Derive the statistic the reports bury — between 82% and 88% of failing tests at the final checkpoint are regressions, not the new requirement
  • Price effort honestly: every medium campaign beat every xhigh campaign on cost per passing checkpoint, with a 9.7x spread from best to worst
  • Hold two findings together without flinching — Opus 5.5 at xhigh wrote materially cleaner code and passed fewer checkpoints than the same model at medium
  • Apply the paper's most actionable result: telling an agent to write clean code improved its first commit by up to a third and did not change the degradation curve at all

What the benchmark actually does​

SlopCodeBench's unit of work is not a task, it is a trajectory. An agent is given a problem, writes a solution, then receives a new requirement and must modify its own existing code. Repeat. The specifications evolve in ways that force architectural decisions while leaving implementation details open, and — this is the mechanism that makes it bite — tests from earlier checkpoints carry forward. To pass a checkpoint you must satisfy the new requirement and keep every previous one working.

The scale, per the paper: 36 problems decomposed into 196 checkpoints total, with 15 coding agents evaluated across open and closed model families.

Two metrics come out:

  • Strict pass — every test passes, including every regression test from every earlier checkpoint.
  • Core pass — the looser bar, which gives credit for satisfying the current requirement.

And the paper names the two degradation modes it tracks. Structural erosion is complexity concentrating — the code curdling into a few overloaded places. Verbosity is redundant code multiplying. These are measured, not vibes.

The result that should be the headline​

From the paper: no agent fully solved any problem end-to-end. Not one of the 15, not one of the 36 problems. The best agent reached 14.8% of checkpoints.

Structural erosion increased in 77% of trajectories; verbosity in 75.5%. Compared against a corpus of 473 open-source repositories, agent-written code showed 2.3× greater verbosity and 2.0× more erosion.

So the baseline claim is not "agents write slightly messier code than humans." It is: across three-quarters of long-horizon trajectories, the code gets structurally worse as the agent works on it, and the end state is roughly twice as eroded as what humans ship.

The October 2026 campaigns​

HumanLayer ran ten complete campaigns on the pinned v1.0 catalog with the stock just-solve prompt, seed 42, any-case runner pass policy, in Docker with Python 3.12 and uv. One run per model-effort combination.

CampaignStrictStrict rateCore rateEst. cost
GPT-6 Astra XHigh32/19616.33%69.39%$758.62
GPT-6 SOL Medium30/19615.31%67.86%$245.69
GPT-6 Astra Medium29/19614.80%66.33%$261.05
GPT-6.1 SOL Medium29/19614.80%65.82%$178.51
Opus 5.5 Medium28/19614.29%73.98%$193.47
GPT-5.6 SOL XHigh28/19614.29%70.92%$753.30
Fable 5.1 XHigh26/19613.27%72.45%$1,550.07
GPT-6 SOL XHigh26/19613.27%68.37%$517.32
Opus 5.5 XHigh25/19612.76%71.94%$817.04
GLM 5.3 XHigh24/19612.24%66.33%$365.13

Look at the strict column first. The entire frontier field lands between 12.24% and 16.33%. The spread from best to worst is 8 checkpoints out of 196. Anyone telling you one of these models is decisively better at long-horizon maintenance is reading noise — especially given one run per cell.

The ranking flips depending on which metric you quote​

This is the part to internalise before you cite any of it.

  • GPT-6 Astra XHigh is 1st on strict (16.33%) and 5th on core (69.39%).
  • Opus 5.5 Medium is 5th on strict (14.29%) and 1st on core (73.98%) — the best core rate in the table.
  • GPT-6.1 SOL Medium is tied 4th on strict (14.80%) and dead last on core (65.82%).

Same ten runs, same data, and the podium reshuffles. The two metrics are measuring genuinely different things: core rate says "how often did it do the new thing," strict rate says "how often did it do the new thing without breaking the old things." A model can be good at one and mediocre at the other, and most of these are.

If you only ever remember one rule from this page: ask which pass rate a coding-agent number refers to. The gap between 14.29% and 73.98% for one model on one dataset is the whole argument.

The statistic nobody quotes: it's regressions, not the new feature​

The report publishes a final-checkpoint table with failed test cases and, separately, how many of those were regression failures. Divide one column by the other and the benchmark's thesis stops being a thesis.

CampaignFinal failed testsRegressionsRegression share
Opus 5.5 Medium2,5082,20688.0%
GPT-6 SOL Medium2,8312,47987.6%
GPT-6.1 SOL Medium2,7572,40487.2%
GPT-5.6 SOL XHigh2,7272,37687.1%
GPT-6 Astra Medium1,7471,51686.8%
Opus 5.5 XHigh2,4462,12386.8%
Fable 5.1 XHigh2,4632,12686.3%
GLM 5.3 XHigh2,7482,31184.1%
GPT-6 SOL XHigh1,6511,36682.7%
GPT-6 Astra XHigh1,4351,17882.1%

The regression share column is arithmetic on the report's published counts, not a figure it states. Across all ten campaigns it sits between 82.1% and 88.0% — remarkably tight for ten different configurations across four model families.

Read that again: when a frontier coding agent fails at the end of a long iterative task, roughly six failures in seven are things that used to work. Only about one in seven is the feature it was actually asked for.

Watch out
  • This is the number that should change your harness, not your model choice. A 4-6 percentage-point difference in strict rate between frontier models is within single-run noise. An 85%-ish regression share is a structural property of how agents edit code — and it is the thing your CI, your test retention policy and your review process can actually act on.
  • It also explains why agents feel better than they measure. Ask for a feature, watch it land, ship it. The regression is three checkpoints away and nobody re-ran the old tests.

"xhigh considered harmful"​

Now the finding that got the post to the front page. Three models were run at both medium and xhigh effort:

ModelStrict: Medium → XHighCore: Medium → XHighCost multiplier
Opus 5.528 → 25145 → 1414.22×
GPT-6 SOL30 → 26133 → 1342.11×
GPT-6 Astra29 → 32130 → 1362.91×

Two of three models got worse on strict passes at higher effort, while costing between 2.11× and 4.22× more. In the author's words: "opus 5.5 did BETTER on medium effort than on xhigh."

To their considerable credit, they refuse to oversell it. The report states the finding "doesn't prove that more reasoning hurts coding" — only that "higher effort was not a reliable way to buy more strict passes." And the methodology caveat is stated plainly: "I ran each model and effort setting once... These single runs don't isolate effort as the cause."

That caveat is real and you should hold onto it. A 3-checkpoint swing out of 196 is well inside the variance you would expect between two runs of the same configuration. The weak claim is what survives: effort is not a dial that reliably buys correctness on long-horizon work. The strong claim — that xhigh actively damages coding — is not established by this data, and the author doesn't make it.

Where the evidence gets much stronger: cost per pass​

The effort table is noisy. The economics are not, because cost differences here are large enough to dwarf the noise. Dividing estimated cost by strict passes:

CampaignEffort$ per strict pass
GPT-6.1 SOL MediumMedium$6.16
Opus 5.5 MediumMedium$6.91
GPT-6 SOL MediumMedium$8.19
GPT-6 Astra MediumMedium$9.00
GLM 5.3 XHighXHigh$15.21
GPT-6 SOL XHighXHigh$19.90
GPT-6 Astra XHighXHigh$23.71
GPT-5.6 SOL XHighXHigh$26.90
Opus 5.5 XHighXHigh$32.68
Fable 5.1 XHighXHigh$59.62

That is my arithmetic on the report's published figures, and the pattern is absolute: every medium campaign is cheaper per passing checkpoint than every xhigh campaign. No overlap. The spread from best to worst is 9.7×.

The single starkest comparison in the whole dataset: Fable 5.1 at xhigh spent $1,550.07 to pass 26 checkpoints. Opus 5.5 at medium spent $193.47 to pass 28. Eight times the money, for two fewer passes.

Pro tip
  • The actionable version: on long-horizon iterative coding, start at medium and make xhigh prove itself on YOUR workload before you pay for it. The defaults already lean this way — Opus 5.5 and Haiku 5.5 default to medium, and medium is the recommended starting effort for agentic coding on Sonnet 5.5.
  • If you do run a comparison, run each configuration more than once. A single campaign at this benchmark's resolution cannot distinguish a 3-checkpoint effort effect from a 3-checkpoint coin flip.

Cleaner code that works less often​

Here is the result that is genuinely hard to explain away, and the most interesting thing in the report.

Opus 5.5 at xhigh passed fewer strict checkpoints than Opus 5.5 at medium — and wrote distinctly better code by the slop metrics. Across 28 matched problems, slop coverage was 77.75% for xhigh against 94.25% for medium (92.90% after filtering); the author characterises xhigh's output as "close to ~5x cleaner."

A note on that figure, because it matters for how you quote it: ~5× is the author's characterisation of their composite slop metric. It is not derivable from the two coverage percentages above, which differ by about 16 points, not 5×. Cite the coverage numbers if you need something checkable; treat "~5× cleaner" as a directional claim about a metric whose internals you haven't seen.

Either way the direction is consistent and the implication is uncomfortable: more reasoning effort bought code that looks better and functions worse. Effort appears to be partly buying architectural taste — refactoring, abstraction, tidiness — and those edits are precisely the ones that break code that already worked. Tidying is a regression risk. A model that leaves the ugly working thing alone scores better on a benchmark that remembers what used to pass.

The report's speculation on the gap between pass rates and code metrics is worth repeating: newer models may be generating novel slop patterns that existing linters do not yet detect. Which is a decent reminder that a clean lint report on agent output is evidence about your linter as much as about your code.

Two opposite architectural styles, visible in the metrics​

One more detail, easy to miss and quietly revealing. Mean cyclomatic complexity across functions and methods:

  • Fable 5.1: 8,967 callables, mean complexity 3.57
  • GPT-6 SOL Medium: 1,083 callables, mean complexity 11.70

Comparable problems, and an 8.3× difference in how many functions got written. Fable decomposes aggressively into many simple units; GPT-6 SOL writes far fewer, far denser ones. Rank them on mean complexity alone and Fable wins by 3×. Rank them on callable count and the story inverts — all those tiny functions are also where verbosity lives.

Neither number means anything alone. This is the general lesson about code-quality metrics on agent output: every single one of them is gameable by an architectural style, so read them in pairs.

What to actually do about it​

The paper tested the obvious intervention and the result deserves to be better known: explicit quality guidance reduced initial issues by up to one-third — and did not affect the degradation patterns.

Telling the agent to write clean code works, once. It improves the first commit. It does not stop the rot. Erosion is a property of the iterative editing process, not of the agent's initial intent, so instructions aimed at intent don't touch it.

Which points the fix somewhere less comfortable than your prompt:

Guided walkthrough1 of 5
  1. An 82-88% regression share is only visible if the old tests still run. If your agent loop validates the current requirement and nothing else, you are measuring core pass rate and shipping strict failures. This is the single highest-leverage change available.

Check yourself

0/6
  1. What does SlopCodeBench measure that single-shot coding benchmarks structurally cannot?
  2. Across the ten October 2026 campaigns, roughly what share of failing tests at the final checkpoint were regressions rather than the new requirement?
  3. GPT-6 Astra XHigh ranks 1st on strict pass rate and 5th on core. Opus 5.5 Medium ranks 5th on strict and 1st on core. What should you conclude?
  4. Which claim about effort does the October 2026 data actually support?
  5. Opus 5.5 at xhigh wrote measurably cleaner code than at medium, yet passed fewer strict checkpoints. What is the most plausible reading?
  6. The paper found that explicit quality guidance reduced initial issues by up to a third. What did it NOT do?
Key takeaways
  • SlopCodeBench measures trajectories, not tasks: the agent extends its own code as the spec evolves, and every earlier test carries forward. No agent has fully solved a single problem end-to-end; the frontier field clusters between 12.24% and 16.33% strict.
  • Strict and core pass rates rank the same ten campaigns almost in reverse. Opus 5.5 Medium is 14.29% strict and 73.98% core. Always ask which one a number refers to.
  • 82-88% of failing tests at the final checkpoint are regressions, not the new requirement — six failures in seven are things that used to work. That is a harness problem, not a model-selection problem.
  • Every medium campaign beat every xhigh campaign on cost per strict pass, with no overlap and a 9.7x spread. Fable 5.1 XHigh: $1,550 for 26 passes; Opus 5.5 Medium: $193 for 28.
  • Effort partly buys tidiness, and tidying working code causes regressions. Opus 5.5 XHigh wrote cleaner code and passed fewer checkpoints — so review refactors harder than additions.
  • Telling the agent to write clean code improved the first commit by up to a third and did nothing to the degradation curve. Erosion lives in the editing process, so fix it with retained tests and per-checkpoint measurement.
  • Read code-quality metrics in pairs: 8,967 callables at mean complexity 3.57 versus 1,083 at 11.70 is two architectural styles, and either metric alone ranks them backwards.

Sources & further reading​

Next​