Is the model nerfed? How to actually tell
On 29 September 2026 the top Claude story on Hacker News was not a launch. It was LiveNerf, a repository whose entire purpose is to answer one question — has Claude Opus 5.5 been nerfed yet? — with 745 points and 306 comments. Seven days after a flagship model ships, the community's most engaged conversation about it is whether it has already been quietly made worse.
This question is older than any current model and it never resolves, because almost everyone arguing about it is arguing from memory. The useful version of the question is narrower and answerable: what, specifically, is allowed to change between the model you used last Tuesday and the model you are using now — and how would you know which one moved?
- Know the one thing that cannot silently change on the Claude API — and why the answer is different on claude.ai
- Recognise the seven mechanisms that really do change output quality without any new model
- Understand why the most-cited 'the model got dumber' result was an eval-design bug, not a capability loss
- Run a measurement that could actually detect a 3-point drop, using a paired design and a control arm
- Triage a bad day in ten minutes before concluding anything
The one thing that cannot change
Start with the claim that the whole argument depends on: they swapped the weights for a cheaper, quantised version and did not tell anyone.
On the Claude API, for the current lineup, this is not a thing that can happen — and the reason is a versioning decision, not a promise about intentions. From Anthropic's models overview:
Every Claude model ID is a pinned snapshot, including the dateless IDs used from the 4.6 generation on.
That sentence is easy to skim past and it is the whole ballgame. Historically
claude-opus-4-5 was an alias that resolved to a dated snapshot like
claude-opus-4-5-20251101 — aliases move, snapshots do not. From the 4.6 generation
onward the dateless ID is itself the snapshot. claude-opus-5-5 is not a pointer
to whatever Anthropic is serving this week; it is one frozen artefact with a published
retirement date (not sooner than 22 September 2027).
So if you are calling the API with a current model ID, "they changed the model" is the one hypothesis you can rule out without measuring anything. Which is useful, because it forces the interesting question: then what did change?
This is a statement about weights, not about serving. A frozen model served through a broken inference stack produces worse output than the same frozen model served correctly — and that has demonstrably happened. See mechanism 4 below.
Seven things that change without a new model
| # | What moves | Where it applies | Visible to you? |
|---|---|---|---|
| 1 | The product system prompt | claude.ai, iOS/Android apps — not the API | Yes, published |
| 2 | Thinking depth (adaptive thinking + effort) | API, Claude Code, apps | Partly |
| 3 | Harness-level model swaps | Claude Code, any client with fallbacks | Sometimes |
| 4 | Serving-stack bugs | Everything | No — until a postmortem |
| 5 | Live incidents | Everything | Yes, status page |
| 6 | Conversation length and compaction | Everything | Rarely |
| 7 | Your own prompts and tasks | You | Almost never |
1. The product system prompt is updated, and the API is not
Anthropic publishes the system prompt that claude.ai and the mobile apps prepend to every conversation, with dated entries per model, and states plainly:
This prompt is periodically updated to improve Claude's responses. These system prompt updates do not apply to the Claude API.
This is the single most under-appreciated fact in the nerf discourse. The same frozen model can behave measurably differently on claude.ai this week than last week, because the several-thousand-token instruction block in front of your message changed — and none of that reaches an API caller. It also means the two populations arguing with each other on Reddit are often not using the same system at all: one group is on a product surface where the framing moves, the other is on a raw endpoint where it does not.
If you want to know whether the app got different, the release notes are the primary source, not your memory.
2. Adaptive thinking means the same prompt does not get the same effort
On the current lineup, thinking is adaptive: the model decides how much to think,
steered by an effort parameter. This is a real, documented source of run-to-run
variance that did not exist in the fixed-budget era.
The defaults are also not uniform, and they moved:
| Model | Default effort | Thinking |
|---|---|---|
| Claude Fable 5.1 | high | Adaptive, always on |
| Claude Opus 5.5 | medium | Adaptive, always on |
| Claude Sonnet 5.5 | high | Adaptive |
| Claude Haiku 4.5 | not supported | Extended |
Anthropic's own note is that Opus 5.5 at medium is calibrated to match Opus 5 at
high. Carry a config forward from an older model without re-reading that and you can
land on less thinking than you had before, on a newer and nominally better model, and
correctly perceive that "it got lazier." Nothing was nerfed. An enum meant something
different.
Claude Code adds a wider ladder — low, medium, high, xhigh, max,
ultracode — settable per session, per model, and overridable by skill and subagent
frontmatter. A skill you installed last month can be quietly capping the effort of
every run it touches.
3. Your harness swaps models on you
This is the mechanism that most often produces a genuine "it got dumber mid-session," and it has nothing to do with the model.
Claude Code has three independent swap mechanisms:
- Usage-limit fallback. Hit the Opus cap and the session continues on Sonnet. Users have filed issues asking for it to stop rather than downgrade, precisely because the drop is felt before it is read.
- Error fallback chains.
--fallback-model sonnet,haikuor afallbackModelarray in settings. It fires on overload and non-retryable server errors — never on auth, billing, rate-limit or request-size errors — and lasts only for the current turn. - Content-flagged fallback. Fable, Opus 5.5, Sonnet 5.5 and Opus 5 run safety
classifiers for cyber and bio content. A flagged message can route to a different
model rather than refuse: Opus 5.5 falls back to Opus 5 for bio and to Opus 4.8 for
cyber; Sonnet 5.5 falls back to Sonnet 5 for cyber and refuses for bio. Toggle it
with
switchModelsOnFlag.
The third one is the sneaky one. Ask about a security topic and your "Opus 5.5 session" may answer on Opus 4.8. The output is not degraded Opus 5.5. It is undegraded Opus 4.8.
The opusplan alias does the same thing by design: Opus for planning, Sonnet for
execution. If your perception is "it plans brilliantly and then implements sloppily,"
check whether you asked for exactly that.
4. The serving stack breaks, quietly, for weeks
The strongest argument that "it's all placebo" is wrong is that Anthropic wrote the counter-example itself. In September 2025 it published a postmortem on three overlapping infrastructure bugs that degraded output quality for weeks:
- A context-window routing error (5 Aug – 18 Sept) sent ordinary requests to servers configured for an upcoming 1M-token context. At peak, 16% of Sonnet 4 requests on 31 August were misrouted, and roughly 30% of Claude Code users hit at least one degraded response. Routing was sticky: once a conversation landed on a wrong server, follow-ups tended to land there too — which is exactly the "it's broken for me and fine for you" pattern people are told is imaginary.
- Output corruption from a TPU misconfiguration (25 Aug – 2 Sept) on Opus 4.1, Opus 4 and Sonnet 4 — including Thai characters appearing mid-English-sentence.
- An XLA:TPU miscompilation of approximate top-k (25 Aug – 12 Sept), a mixed-precision bug in the compiler that translates JAX to TPU instructions, triggered by a change intended to improve token selection.
Three details from that document are worth internalising. First, Anthropic states flatly: "We never reduce model quality due to demand, time of day, or server load" — the specific conspiracy theory is denied, while three unrelated real bugs are admitted. Second, the reason detection took weeks is that the internal evals did not catch it: the benchmarks stayed green while users were right. Third, privacy controls meant engineers could not simply read the bad conversations.
That is the honest shape of this problem. Degradation is real, it is usually infrastructure rather than intent, and aggregate benchmarks are bad at seeing it.
5. There is an incident open right now
Before theorising, check status.claude.com. On 29 September 2026 — the same day LiveNerf hit the front page — Anthropic opened an incident at 14:21 UTC for elevated error rates across claude.ai, Claude Code, Claude Cowork and the API, expanded it at 14:30 to "degraded performance," and applied a mitigation at 14:41. A week earlier, on 22 September, errors hit several models including Opus 5, Mythos 5.1 and Fable 5.1.
A meaningful share of "it got worse today" posts are filed during a window that is already publicly acknowledged.
6. The conversation got long
Every current Claude model advertises a 1M-token context window, and quality is not flat across it. Two compounding effects:
- Long conversations degrade. Attention over a very long prefix is not free, and instructions given 200 turns ago compete with everything since.
- Compaction rewrites your history. Server-side compaction summarises earlier context when it nears the trigger threshold. Claude Code compacts too. After a compaction, the model is working from a summary of the thing you remember telling it. It did not forget. The transcript changed.
Also worth knowing: the tokenizer introduced with Opus 4.7 changed the exchange rate. 1M tokens is roughly 555k words on the current tokenizer, against about 750k words on the older one. The same document is a larger fraction of your window than it used to be.
7. You changed
The least popular mechanism and the most common. Week one you ask for things you
already know the model does well. Week four you ask harder questions, on a larger
codebase, with a longer CLAUDE.md, having stopped writing the careful prompts you
wrote when it was new. The measured quantity — my satisfaction per session — has two
moving inputs and you only instrumented one.
The most-cited proof is an eval bug
If you have seen one piece of evidence that a model got worse, it is probably this: in How Is ChatGPT's Behavior Changing over Time?, Chen, Zaharia and Zou report that GPT-4's accuracy on "is this number prime? think step by step" fell from 97.6% to 2.4% between March and June 2023. It went everywhere. It is still quoted.
Then Arvind Narayanan and Sayash Kapoor looked at the test set. Every one of the 500 numbers was prime.
The March model almost always answered "prime." The June model almost always answered "composite." Against an all-prime set that is 97.6% versus 2.4% — but it measures a bias flip, not reasoning. When they ran 500 composite numbers, the scores inverted. Neither version was checking factors; both listed the divisors to test and then guessed. The behaviour changed. The capability was the same, and it was never good.
The distinction to keep. Behaviour drift — different formatting, verbosity, willingness, default guess — is real, common, and a documented side effect of fine-tuning. Capability loss is a much stronger claim and needs much stronger evidence. Almost every viral nerf claim is the first dressed as the second.
Three statistical traps sit on top of that, and they apply to your own impressions too:
- An all-one-answer eval measures bias, not skill. If your personal test set is all "should refuse" or all "should succeed," a shift in the model's default posture looks like a collapse.
- Selection bias in the evidence. Nobody posts "Opus was fine again today." The feed you are reading is a filter that only passes bad days, and it gets denser as a model gets more popular.
- Regression to the mean. Your baseline is the launch week you were most impressed in — which, being the extreme of your sample, is the thing an unchanged model is least likely to repeat.
How to measure it — the LiveNerf design
LiveNerf is worth studying less for its verdict than for its methodology, because it is the design a credible answer requires. From the repository (662 stars at time of writing, 6 of 30 days collected):
- You cannot make these models deterministic — sampling parameters are gone on the current lineup and adaptive thinking cannot be switched off. LiveNerf's answer, in its own framing: make everything else deterministic. Frozen prompts, a pinned CLI version, exact graders, raw logs kept forever. You are not removing noise; you are making sure the noise is the only thing left that can move.
- 2,336 candidates were screened from GPQA Diamond, MMLU-Pro, competition maths and AIME 2025–26. Opus 5.5 gets roughly 93% of them right first try — so 93% of the pool is useless for detection, because a ceiling cannot show a drop. The panel is the 78 questions the model is only sometimes right on. A question everyone always gets right measures nothing.
- Each run is compared item-by-item against the same items in the launch-week baseline, so question difficulty cancels out instead of becoming a confound. This is what lets a 78-item panel detect anything at all.
- The same GPQA questions go through the same harness on Claude Opus 5 — a different model — every day. If both arms move together, the change is in the platform, the harness or the grader, not in Opus 5.5. Without a control you cannot distinguish 'the model changed' from 'my measurement changed'.
- LiveNerf declares a change only when three conditions hold at once: a 99% confidence interval excluding zero across two consecutive 10-day windows, an effect of at least 3 points, and no matching movement in the control arm. Deciding the threshold after seeing the data is how noise becomes a finding.
- Full prompts, full responses, token usage, latency, CLI version, harness git SHA, task and item hashes, and the plan usage meters before and after each daily run. Thirty days: ten to establish a baseline, then two 10-day windows.
The shortest version of all six: a single bad session is not evidence, and neither is a single good one.
Your own version, cheaply
You do not need 78 GPQA items. You need your own tasks, which is better — you care about your workload, not about MMLU-Pro. The protocol scales down:
Personal nerf-check protocol
1. Collect 20-30 real prompts from your own work. Keep only the ones you sometimes get a good answer to and sometimes don't. Drop anything that always works (ceiling) or never works (floor). 2. Write a grader you can run without thinking about it. Pass/fail is enough. A human rubric you apply inconsistently is worse than nothing. 3. Pin EVERYTHING that is pinnable: - exact model ID (e.g. claude-opus-5-5, never an alias on an older generation) - effort level, explicitly set - never inherited - one frozen system prompt, byte for byte - no tools, or exactly the same tool list every run - fresh conversation per item; no shared history 4. Baseline: run the whole set 5 times over the first week. Save raw outputs. 5. Re-run the identical set on the same schedule later. Compare item by item against the baseline - not set average to set average. 6. Add a control arm: the same items on a DIFFERENT model, same harness, same day. Both move together => your measurement moved, not the model. 7. Decide in advance what counts: e.g. "a drop of 3+ items, twice in a row, with the control flat." Write it down before you look at the numbers.
Two failure modes to avoid. Do not use an alias where the generation still has one —
you will be measuring whatever the alias points at. And do not let effort be
inherited from a session setting, a skill, or subagent frontmatter, or you are
measuring your own config drift.
What the trackers have actually found
As of 30 September 2026, the public trackers do not support a nerf of Opus 5.5. LiveNerf had collected 6 of its 30 planned days with no missed runs and therefore had not reached a verdict — by its own pre-registered rule it cannot, before two full 10-day windows. A second tracker, BridgeBench's NerfBench, was reported in late September to place Opus 5.5 at roughly 99.2% of launch power, comfortably inside its normal-variance band — a figure we could not verify at source (see the note above).
Two honest conclusions from that. The empirical evidence so far points at "no detectable change," and the honest version of "no detectable change" includes the detection floor: a paired 78-item panel with a 3-point minimum effect can miss a real 1-point drop entirely. No evidence of degradation is not evidence of no degradation — and a rigorous tracker will tell you which one it has.
Ten-minute triage
Before you conclude anything, in this order:
- status.claude.com. An open incident ends the investigation. On 29 September 2026 the degraded-performance window and the peak of the nerf discussion were the same afternoon.
- In Claude Code, /model. Look for a usage-limit switch, a fallback chain firing, a content-flag swap, or opusplan doing exactly what it says. The API returns the serving model in the response — read it rather than assume it.
- /effort, plus any effortLevel or per-model modelSettings in settings, plus the frontmatter of any skill or subagent in play. A carried-forward config is the most common cause of 'it got lazier' with no other change.
- If compaction has run, the model is reading a summary of your earlier turns. Start a fresh session with the context restated and see whether the problem survives.
- Adaptive thinking makes single-sample judgements unreliable by construction. If two of three are fine, you sampled a bad draw, not a worse model.
- If you kept no artefact from the good period, you are comparing against a memory of the week you were most impressed - the least reproducible sample you have. Start saving outputs today so the next time this happens you can answer it.
Quiz
Check yourself
0/6Next
- Claude Opus 5.5: the field guide — the model at the centre of this argument, and what actually shipped with it
- Claude Sonnet 5.5: the field guide — effort defaults, thinking-block binding, and the config traps that look like degradation
- How to choose a model — the durable framework, once you have stopped chasing the week's discourse
Sources & further reading
- ninjahawk/livenerf — the paired-design, control-arm, pre-registered tracker for Opus 5.5; read the README for the full protocol
- A postmortem of three recent issues — Anthropic's own account of real degradation: routing, TPU misconfiguration, and an XLA compiler bug
- Models overview — the pinned-snapshot statement, effort defaults, context and tokenizer notes
- Model IDs and versioning — how aliases and snapshots differ across generations
- System prompt release notes — the published, dated product system prompts, and the note that they do not apply to the API
- Claude Code model configuration — fallback chains, content-flagged fallback, effort levels, subagent model precedence
- Claude status — check this before theorising
- How Is ChatGPT's Behavior Changing over Time? — Chen, Zaharia and Zou; the source of the 97.6%→2.4% figure
- Is GPT-4 getting worse over time? — Narayanan and Kapoor's correction: the all-prime test set, and behaviour drift versus capability loss
- anthropics/claude-code#3434 — the silent Opus→Sonnet fallback issue, as users experienced it