إنتقل إلى المحتوى الرئيسي

CI/CD for Agent-Written Code

متقدّم
What you'll learn
  • Know why the merge queue — not the runner budget — is the first thing that breaks under agent PR volume
  • Do the batching arithmetic: how a few points of failure rate collapse speculative merging
  • Rank the four levers correctly, because 'add more runners' is the weakest one
  • Ship a gate that proves an agent's test actually fails without the agent's fix
  • Quarantine flakes on evidence instead of vibes, and know the caching gotcha that silently poisons CI

Your coding agents work. That is the problem.

A team that merged 5 pull requests a day and now merges 40 has not changed its test suite, its runner budget, or its branch protection rules. It has changed the arrival rate into a queueing system that was tuned for the old one. Everything downstream — merge queue, flaky-test noise, runner concurrency, review attention — was sized for human throughput, and queueing systems do not degrade gracefully. They hold, hold, hold, then fall over inside a week.

This page is the pipeline side of the verification bottleneck. That page asks who checks the code. This one asks whether your CI can physically absorb the volume, and what to do when it can't.

Three numbers that frame the whole problem​

−7.2%. DORA's 2024 Accelerate State of DevOps report modelled a 25% increase in AI adoption and estimated a 7.2% decrease in delivery stability alongside a 1.5% decrease in throughput. Individual productivity and job satisfaction went up. Delivery-level outcomes went down — the report ties this to larger changesets and weaker batching discipline. The 2025 follow-up reframes it: AI is an amplifier, magnifying whatever your delivery system already is. A pipeline that was fragile at 5 PRs/day becomes unusable at 40.

19%. METR's randomised trial (July 2025) took 16 experienced open-source maintainers and 246 real issues from their own repositories, randomising whether AI tools were allowed. With AI, tasks took 19% longer. The gap that matters for CI planning is the perceptual one: developers predicted a 24% speed-up, and after the slowdown still believed they had been sped up by 20%. Nobody on your team will tell you the pipeline is the constraint, because nobody experiences it as one — they experience it as "waiting", which feels like weather.

1.5% / 16% / 84%. Google's published flakiness data: roughly 1.5% of test runs report a flaky result; about 16% of their 4.2M tests have shown flakiness at some point; and around 84% of observed pass→fail transitions involve a flaky test rather than a real regression. Hold onto that 1.5% — it is small enough to ignore at 5 PRs/day and load-bearing at 40.

Why the merge queue breaks first​

Most teams assume the ceiling is compute. It isn't. It's the merge queue, and the reason is a mechanism most people have never read the docs for.

GitHub's merge queue does speculative checking. Queue PRs #1, #2, #3 and it builds temporary branches: main + #1, main + #1 + #2, main + #1 + #2 + #3 — and tests all of them in parallel. If every check passes, all three land in one round. That is the whole value proposition: batching turns N serial validations into one.

Now read the failure path. When a check fails for the group containing PR #1, GitHub removes #1 from the queue — and every group behind it contained #1's changes, so those groups are invalid and get rebuilt without it. One bad PR does not cost you one slot. It costs you the rest of the batch's CI round.

That makes queue throughput a function of your pass rate, and it's steeper than intuition suggests. Model it: batch size N, each PR independently failing with probability p, and you merge the run of PRs up to the first failure. Expected merges per round is a geometric expectation:

E[merges] = (1 − p) · (1 − (1 − p)^N) / p

With N = 10:

Per-PR failure rate pExpected merges per roundRounds to clear 10 PRs
2%8.9~1.1
5%7.6~1.3
10%5.9~1.7
20%3.6~2.8
30%2.3~4.4

Two consequences most teams get backwards:

  1. A flaky suite is a throughput tax, not a nuisance tax. Flakiness adds directly to p. Going from 5% real failures to 5% real + 5% flake takes you from 7.6 merges per round to 5.9 — roughly 23% of your merge capacity, lost to tests that were never testing anything.
  2. Raising the batch size is the wrong lever when p is high. Bigger N only helps when (1 − p)^N stays near 1. At p = 0.2, a batch of 10 delivers 3.6 merges; the fix is lowering p, not queueing more optimism.

And agent volume pushes p the wrong way — DORA's stability finding is exactly a statement about p rising.

The capacity arithmetic nobody does​

Do this on a napkin before you buy runners. Assumptions stated, because they're what make it wrong for your team:

Guided walkthrough1 of 5
  1. A 45-minute suite fanned out into 15 parallel jobs still costs ~45 job-minutes. Parallelism buys latency, never cost. Take your suite's total job-minutes from a recent run, not from the slowest job.

The takeaway is not "buy Enterprise". It's that the constraint is burst concurrency, and the daily-average number that makes your pipeline look fine is the number hiding the problem.

Four levers, ranked by leverage​

Teams reach for these in almost exactly the reverse of the useful order.

1. Lower p (highest leverage, lowest cost). Every point of failure rate removed pays off nonlinearly in the batching math. Concretely: make agents run the fast suite locally before opening a PR, require a green pre-merge check before a PR is allowed into the queue, and quarantine flakes aggressively (below). This costs no compute.

2. Cut the work per validation — test selection. This is the only lever that changes the exponent rather than the constant. Instead of running everything, run what the change can plausibly break:

  • Develocity Predictive Test Selection builds a model from your own Build Scan history plus cross-project patterns, and ships a simulator so you can measure the savings and the missed-failure rate before enabling it. Two safety details worth stealing even if you never buy it: you can pin must-run tests by class pattern or annotation, and it always selects recently flaky tests, because a flaky test is by definition unpredictable from the change. Pair it with a "remaining tests" run post-merge or nightly so nothing is skipped forever.
  • Bazel / Buck2 get this structurally: the build graph knows which targets depend on the changed files, so bazel test on affected targets is target determination for free.
  • No build graph? Python has pytest-testmon, which records which tests cover which code and re-runs only the affected ones. Jest ships --changedSince out of the box.

Test selection, three ecosystems

# Bazel — test only what the diff can reach
bazel test $(bazel query "rdeps(//..., set($(git diff --name-only origin/main)))" --output=label 2>/dev/null)

# Python — coverage-based impact analysis
pip install pytest-testmon
pytest --testmon          # first run builds the DB, later runs select

# Jest — built in, no plugin
npx jest --changedSince=origin/main

3. Cache aggressively — and know the gotcha. Remote caching stores action results keyed by a hash of inputs, command line and whitelisted environment. Bazel's own docs flag the trap: tools outside the workspace are not tracked, so two machines with different compiler versions can get a cache hit on work that would have produced different outputs. Whitelist what matters with --action_env, and keep write access to CI only — developer machines should read:

Remote cache, CI-writes-only

# CI: read and write
bazel test //... --remote_cache=grpcs://cache.internal:443

# Developer machines: read only, never poison the shared cache
bazel test //... --remote_cache=grpcs://cache.internal:443 \
--remote_upload_local_results=false

4. Add runners (last). Linear cost, linear benefit, and it does nothing about p. Do it after the first three, when you actually know your burst concurrency number.

The AI-specific failure mode: the agent grades its own homework​

Everything above applies to any high-volume pipeline. This one is specific to agent-written code, and it is the gap most CI setups have.

When the same agent writes the fix and the test for the fix, "tests pass" carries much less information than you think. Two failure shapes show up constantly:

  • The test asserts the behaviour of the implementation rather than the requirement — it passes because it was written by reading the code it is testing.
  • The agent, when a pre-existing test fails, edits the test to match the new behaviour instead of fixing the code. The diff is green and the regression shipped. That behaviour has a name and a literature — see reward hacking in coding agents for why models do it and which defenses have measured effect sizes. What follows is the pipeline-side control, not the model-side one.

The cheap structural gate: a new test must fail against the pre-change code. That's the red-green discipline of TDD turned into a machine check, and it's what mutation testing formalises at scale. You do not need a mutation-testing framework to get most of the value — you need one CI job:

CI gate: new tests must fail without the change

#!/usr/bin/env bash
# must-fail-first.sh — run in CI on PRs that add tests
set -euo pipefail

NEW_TESTS=$(git diff --name-only origin/main...HEAD -- 'tests/**' 'test/**')
[ -z "$NEW_TESTS" ] && { echo "no new tests, skipping"; exit 0; }

# Keep the new tests, revert everything else to the base commit
git stash push --keep-index -- $NEW_TESTS
git checkout origin/main -- src/
git stash pop

# The new tests MUST fail now. If they pass, they prove nothing.
if pytest $NEW_TESTS -q; then
echo "FAIL: new tests pass against unmodified source — they do not test the change."
exit 1
fi
echo "OK: new tests fail without the fix."

Two honest caveats. It only works when source and tests live in separate trees, and a test that fails for the wrong reason (import error, missing fixture) passes this gate. It is a floor, not a proof — but it catches the single most common way an agent-authored PR is green and worthless.

The complementary review-side control is a reviewer that doesn't share the author's context: see the hybrid AI code reviewer and the PR-review GitHub Action.

Flake quarantine that survives contact with agents​

Given 84% of pass→fail transitions are flakes, "rerun it" is the default human response and it is the most expensive one available — it burns a full validation on a test that was never going to tell you anything.

Guided walkthrough1 of 5
  1. A flaky test is one that produces different results on the same commit. Run the suite N times on an unchanged main and record per-test failure counts. That is your evidence; developer memory is not.

Gotchas that cost a day each​

  • merge_group is not automatic. GitHub's docs are explicit: you must add the merge_group event to your workflow triggers, or your checks never fire inside the queue and merges fail on missing required checks. This is the single most common merge-queue misconfiguration.
  • The 24-hour queue cancellation is silent. Saturated queue → jobs vanish rather than fail. Alert on queue depth, not just on red builds.
  • Matrix jobs cap at 256 per workflow run. Generated matrices (a common agent-scaffolding pattern) hit this without a useful error.
  • Cross-machine cache hits can be wrong. Untracked external toolchains produce false hits. Pin toolchains, whitelist environment with --action_env.
  • Branch protection with wildcard patterns is incompatible with merge queues. If your protection rules use wildcards, the queue will not engage.
  • Agent PRs touching CI are a security boundary, not a style question. A PR that can edit workflow files can exfiltrate CI secrets — see GitHub issue → CI secrets and hardening autonomous runs.

The honest framing​

There is no configuration that makes a pipeline absorb unbounded agent output. What the levers above buy you is a system whose cost grows with the size of the change rather than the number of pull requests — test selection and caching are sublinear, runners are linear, and merge-queue batching is only sublinear while your pass rate stays high.

Which puts the real constraint back where DORA's 2025 framing puts it: AI amplifies the delivery system you already have. If small batches, fast suites and low flakiness were already your practice, agents make you much faster. If they weren't, agents find out.

Check yourself

0/6
  1. Your merge queue has batch size 10 and your per-PR failure rate rises from 5% to 20%. Roughly what happens to expected merges per round?
  2. A merge group fails because of PR #1. What does GitHub do to the groups queued behind it?
  3. Which lever changes how CI cost scales, rather than just shifting the constant?
  4. Why does Develocity's Predictive Test Selection always run recently flaky tests, even when the diff does not touch them?
  5. An agent's PR adds a fix and a test, and CI is green. What does the 'must fail first' gate check?
  6. Two machines get a Bazel remote cache hit on the same action but would produce different outputs. What is the most likely cause?
اضغط Enter أو مفتاح المسافة لقلب البطاقة. استخدم مفتاحي السهمين الأيسر والأيمن للتنقل بين البطاقات.تم إظهار المصطلح.
1 / 12
Key takeaways
  • The merge queue breaks before the runner budget does. Speculative batching is a bet on your pass rate, and one failing PR invalidates every group behind it.
  • Compute E[merges] = (1 − p)(1 − (1 − p)^N)/p once for your real numbers. It turns 'CI feels slow' into a decision about which lever to pull.
  • Flakiness is throughput, not hygiene. At Google's published rates the noise is bigger than the signal — 84% of pass→fail transitions are flakes.
  • Size for burst concurrency, never daily average. The daily number is the one hiding the problem, and a saturated queue cancels jobs silently at 24 hours.
  • Test selection is the only sublinear lever: Bazel/Buck2 target determination, Develocity PTS with its simulator, pytest-testmon, or jest --changedSince.
  • When the agent writes both the fix and the test, green means less than you think. Gate on the test failing against pre-change source.
  • DORA's framing holds: AI amplifies the delivery system you already have. Small batches, fast suites and low flake were always the advice — agents just made ignoring it expensive.

Sources & further reading​

Next​