Cloudflare's security-audit Skill: Adversarial AI Security Audits with Claude Code (2026)
- Explain why 'ask Claude to find vulnerabilities' produces confident garbage, and what separation of powers between hunters and verifiers fixes
- Walk the six phases and name the artefact each one produces, from architecture.md to NEEDS-VALIDATION.md
- Read a findings.json correctly: what confirmed, needs_validation and rejected each require, and why only confirmed records get a severity
- Run the skill in Claude Code inside a sandbox that meets its rules, validate the outputs mechanically, and re-run for coverage
- Budget it: what the token bill looks like, the rule that makes the skill refuse to launch hunters, and the three ways to keep a run affordable
Cloudflare open-sourced a single skill in mid-September 2026 and it became the fastest-growing repo on GitHub: cloudflare/security-audit-skill sat at 18.5k stars on September 21 after adding 2.4k in one day, and had already drawn a 211-point Hacker News thread on September 17. The reason it matters is not the star count. It is that this skill is the seed of the fleet-wide vulnerability harness Cloudflare described in June, the one that ran across 128 repositories and produced 7,245 actionable findings. The public skill is the single-repo version you can run today with Claude Code.
It is worth being precise about what it is. There is no binary, no rules engine, no model of its own. It is a SKILL.md plus fourteen hunting guides and two zero-dependency Node validators. Everything it does, it does by making your agent spawn isolated sub-agents with narrow briefs and refusing to accept their output until a different agent has tried to break it.
Why "find the vulnerabilities" fails on its own
Cloudflare's June write-up contains the sentence that explains the whole design: an unconstrained agent "will edit the source code so its own exploit works, then triumphantly report the bug it just created." And the companion rule: "If a Hunter is allowed to grade its own homework, it will confidently validate everything it outputs."
That is the failure mode of every one-shot "audit my repo" prompt. The model that spent 80k tokens forming a theory is the last model you want judging that theory. The skill's answer is structural:
- Hunters cannot file findings. They return candidates and are forbidden from editing the target source, the architecture notes, the ledger, or the findings file.
- Verifiers cannot hunt. A fresh verifier receives one candidate, the architecture facts and the schema, and nothing else, not even another verifier's conclusion. Its job is to disprove.
- Nothing is retained unless deterministic code accepts it. The parent agent writes the shared files;
validate-findings.cjsandvalidate-coverage-ledger.cjsreject anything that breaks the schema.
Cloudflare's internal version goes further and uses a different model for validation than for discovery, on the theory that separate weights give an unbiased second opinion. The public skill does not enforce that, but nothing stops you from routing the verifier sub-agents to a different model.
The six phases and what each leaves behind
| Phase | Who runs | Output |
|---|---|---|
| 1. Reconnaissance | 4 parallel research agents (stack and build; principals and auth; entry surfaces and sinks; what is offline-testable) that return facts only | architecture.md (about 1,000 words) and a seeded coverage-ledger.json |
| 2. Coverage-led hunting | Waves of isolated hunters, one or more ledger units each; a coverage critic after every wave | Candidates with proposed_verdict, unit dispositions, hardening notes, newly discovered gaps |
| 3. Candidate validation | One fresh verifier per unique candidate fingerprint | A decision per candidate: confirmed, needs_validation or rejected |
| 4. Structured output | The parent only | findings.json, sorted by fingerprint, schema-validated |
| 5. Independent record verification | One fresh verifier per confirmed or needs_validation record | verified or replace; any material replacement gets a second fresh verifier |
| 6. Target-neutral reporting | The parent, after phase 5 passes | REPORT.md, FINDINGS-DETAIL.md (medium and up only), NEEDS-VALIDATION.md |
Everything lands under ~/security-audit-skill/<repo-name>/run-<N>/, with per-agent agents/<id>/scratch/ and agents/<id>/artifacts/ directories. Only the parent writes the shared files.
The coverage ledger is the mechanism most people miss. A unit is one material combination of entry surface, trust boundary, subsystem and attack class, with starting paths in the repo. Units move through planned, in_progress, blocked, covered or candidate, and each state has rules about which evidence fields must be filled. A coverage critic reads the whole ledger after a wave and looks for unmapped entry points, unchecked parallel paths and unjustified exclusions; it can reopen a unit (archiving the prior attempt) but can never propose a finding. That is what makes re-runs additive: a second run picks up the units the first left blocked or reopened, instead of re-auditing everything.
The three verdicts, and why only one carries a severity
The schema is strict about which fields each verdict may use, and the validator enforces it.
| Verdict | Means | Must have | Must not have |
|---|---|---|---|
confirmed | Independently reproduced in the sandbox from source, minimum observed effect only | root_cause, intended_behavior, trace (entrypoint to sink), evidence, conditions, execution, remediation, severity, confidence | claimed_root_cause, blockers, validation_plan, reason |
needs_validation | Source-grounded, but blocked by one exact fact that cannot be resolved locally (a deployment setting, an owner-only config) | claimed_root_cause, trace, evidence, blockers, validation_plan | severity, execution, remediation |
rejected | The verifier disproved it | claimed_root_cause, trace, evidence, reason | severity, execution, remediation, blockers |
Three consequences follow:
- A
needs_validationfinding has no severity by design. It is not a "medium"; it is a question for the code owner with an exact plan to answer it.NEEDS-VALIDATION.mdexists so that list is not mistaken for a vulnerability list. - Severity requires demonstrated impact, not a checklist miss. The calibration in
SKILL.md: critical is unauthenticated code execution, full data access or account takeover; high is a complete defeat of an explicit control (auth bypass, cross-tenant access); medium is limited blast radius or uncommon preconditions; low is non-secret disclosure; informational is a confirmed observation with minimal impact. "Missing best practice" is explicitly not a finding. - A clean run reports zero. The reporting guide says a proportional report may have no confirmed findings and must say so plainly. If your run "finds" twenty criticals in a small service, suspect the sandbox setup before you suspect the service.
For readers of this section, the AI/LLM hunting guide is the sharpest part. Its opening rule: "Prompt injection alone is not a finding. Require a code-level boundary failure." A hunter must show the attacker, the affected principal, the execution identity, and an unauthorised action or disclosure, with a source trace from attacker-controlled input to the side effect. The patterns it sends hunters after are the ones in our prompt injection and MCP tool poisoning pages, but framed as code defects: tool arguments reaching SQL or a shell without handler-side validation, sub-agents inheriting full credentials instead of least authority, tool descriptions treated as authorisation, model output rendered into HTML or a command without encoding, cross-tenant context bleed from over-broad cache keys. Persuading the model to say something rude does not count.
The sandbox rule is not advice
SKILL.md requires that any target-controlled code (tests, fixtures, the project's own build) run inside an OS-enforced sandbox with no external network, an isolated loopback namespace at most, read-only target source, writes confined to scratch/, and strict CPU, memory, process and file-size limits. Verifiers are told that if the sandbox is unavailable, the candidate simply stays unvalidated. Promotion from scratch/ to retained artifacts/ refuses symlinks, multiply-linked files, archives and recursive copies, and reopens nothing by path.
This is the same threat model as our coding agents under attack page, turned inward: the repository you are auditing may be hostile, and its test suite is code you are about to execute. In Claude Code, that means running the audit with the sandbox and credential masking controls on, not in your daily shell with your real environment variables. One gotcha from Cloudflare's own harness applies if you nest Docker inside a container: it needs seccomp=unconfined and apparmor=unconfined "or it will silently fail."
- npx skills add https://github.com/cloudflare/security-audit-skill --skill security-audit. The skills CLI (vercel-labs/skills) drops it into .claude/skills/ for Claude Code or .agents/skills/ for Codex, Cursor and Gemini CLI; add -g for a global install, -a claude-code to target one agent. Read SKILL.md before you run it: it is the contract you are about to execute.
- Start Claude Code in the target repo with sandboxing and credential masking on, no production credentials in the environment, and network egress limited to your model provider. The skill needs an agent that can spawn parallel sub-agents; Claude Code's Agent tool qualifies.
- Say 'security audit this codebase' or 'find security vulnerabilities in ./src'. A narrower question ('is this JWT check safe?') triggers guidance mode instead, which answers without creating a run directory. For a first run, point it at one bounded directory, not the monorepo root.
- Everything appears under ~/security-audit-skill/<repo>/run-1/. architecture.md should exist within minutes; coverage-ledger.json grows unit by unit; findings.json appears only after phase 4. If nothing has landed after recon, check the agent has a sandbox it can execute in.
- node <skill-dir>/validate-findings.cjs run-1/findings.json and node <skill-dir>/validate-coverage-ledger.cjs run-1/coverage-ledger.json. The skill runs these itself, but running them yourself is the cheapest way to catch a run that was cut short by a session limit.
- Answer each blocker (it is phrased as a question about your deployment) and either promote or drop the record. Then run again: the ledger's prior_status fields make the second run target blocked and reopened units rather than starting over. Cloudflare's own data says one run finds roughly half of what repeated runs find.
Trigger phrases and the validator commands
# Full audit (creates ~/security-audit-skill/<repo>/run-N/) security audit this codebase find security vulnerabilities in ./src # Guidance mode (no run directory, no sub-agent fleet) is the session cookie handling in auth/session.ts safe against fixation? # After a run: mechanical checks, zero dependencies beyond Node node .claude/skills/security-audit/validate-findings.cjs ~/security-audit-skill/<repo>/run-1/findings.json node .claude/skills/security-audit/validate-coverage-ledger.cjs ~/security-audit-skill/<repo>/run-1/coverage-ledger.json # What the schema hard-limits (validator rejects beyond these) # findings.json: 5 MiB, 1,000 findings, 64 nesting levels # coverage-ledger.json: 5 MiB, 10,000 units, 1,000 entries per collection
What it costs, and how to keep it affordable
The HN thread is mostly about the bill. One user reported "I threw 1M tokens for nothing in a medium codebase"; another put the practical ceiling at 150k tokens, "where you start hitting the dumb zone" of degraded attention. Those are anecdotes, not measurements, but they match the shape of the design: four recon agents, a wave of hunters, a verifier per candidate, a critic per wave, then a second verifier per surviving record. Cloudflare's fleet numbers give the honest scale: a standard 30k-line repository took 3 to 4 hours for about 100 raw findings, and across the fleet 58% of raw candidates survived validation (12,057 of 20,799).
Three things in the skill itself keep it from running away:
- The budget-reserve rule. Before each wave the parent must reserve budget for the hunters, the validation pass, the post-wave critic and a final clean critic.
HUNTING.mdsays: if those mandatory reserves do not fit, launch no hunter from that wave, and mark the units deferred with the budget reason. A run that ends with deferred units is a run that respected your budget, not a broken run. - Narrow briefs. Every sub-agent gets one unit or one candidate. Cloudflare's harness kept each agent under 25% of its context window and credits that, more than any prompt wording, for the quality of findings.
- Guidance mode. Most security questions do not need the pipeline. Ask them directly and the skill answers without spawning anything.
On your side: scope the first run to one directory; use a model with a large context and cheap cache reads for the hunters and a stronger one for verifiers; put a task budget on the session; and treat a run cut short by a session limit as partial, not failed, because the ledger records exactly where it stopped.
Where it sits next to Semgrep, CodeQL and AI code review
It is not a replacement for static analysis and does not try to be. Cloudflare's post has a telling detail: they wired Semgrep into the harness and the agents "ignored it completely," preferring to read and run the code. Static tools give you deterministic, cheap, repeatable coverage of known patterns; this skill gives you expensive, probabilistic, reasoning-heavy coverage of boundary failures that pattern matchers cannot express, with a validation step that static tools do not have. Run both. And do not confuse it with PR-time review: Alibaba's open-code-review is a diff reviewer that runs in seconds per pull request; this is a multi-hour audit of a codebase at rest.
Check yourself
0/6Sources & further reading
- cloudflare/security-audit-skill on GitHub — README,
SKILL.md, the phase guides (RECONNAISSANCE.md,HUNTING.md,VALIDATION-AND-REPORTING.md), the fourteen attack-class guides includingAI-AND-LLM.md,report-schema.json, and the two validators; MIT - Build your own vulnerability harness — Cloudflare blog, June 18, 2026 (Grant Bourzikas): how the skill grew into the fleet harness, the 20,799 to 12,057 to 7,245 funnel, the 3–4 hours per 30k-line repo figure, the "grade its own homework" rule, the nested-Docker and 200-OK-with-error gotchas
- Hacker News discussion (item 49736466) — first-hand token reports and the cost pushback
- vercel-labs/skills — the
npx skills addCLI and where it installs skills for each agent - Agent Skills open standard — why one
SKILL.mdruns in Claude Code, Codex, Cursor and Gemini CLI
Next
- Vetting agent skills you install — this skill spawns agents that execute your repo's code; read it before you trust any skill, including this one
- Hardening autonomous runs — the sandbox, secret and egress setup a multi-hour unattended audit needs
- Open Code Review: Alibaba's hybrid AI reviewer — the PR-time counterpart, and the same precision-versus-recall lesson