The five-minute CI budget: how we stopped CI being the bottleneck

If you run coding agents, you have probably solved two problems already. Code generation works. Review works too, because a second agent reads the diff and a rubric gates the merge.

Then you notice the queue has moved. Agents open merge requests faster than your CI can check them. Every change waits on a pipeline built for a team that merged a few times a day.

We hit that wall on our own repos. This post covers how we got past it, and what limits made it stick.

What was wrong

Our old pipeline was thorough. That was the problem. A one-line CI edit started a pipeline of about 50 jobs. One frontend job ran 26 minutes on every merge request. Browser jobs took between 8 and 22 minutes. Blocking jobs sat in the queue for 7 to 8 minutes before they started.

The main branch was worse. A full main pipeline took about ten minutes, but merges landed every two to five minutes. New merges cancelled main runs before they finished. We paid for the compute and never got the answer.

A human team grumbles about this. For a herd of agents it is the ceiling. An agent that waits half an hour for a verdict holds a slot, burns tokens on retries and blocks whatever depends on it. Slow CI is a tax on every task.

The rule: five minutes per job

We stopped tuning and set a limit. No merge request job may run longer than five minutes. If a job cannot finish inside that, we cut it or move it to a schedule. We never make the merge wait for it.

A limit only works if it is enforced. A guard test reads the CI config and fails any merge request that lets a job exceed the cap. An exception needs a written reason and an expiry date. Our exception table is empty.

Two details matter here.

• The cap applies to each job. We say “no job runs longer than five minutes” and mean it that narrowly. Jobs run in parallel, so pipelines stay short, but the guard checks jobs one by one.
• Hung tests get a watchdog. It dumps the state of a stuck test well before the timeout, so a failure leaves evidence instead of a blank log.

The limit forced the real work. To fit the budget we used shallow clones, which cut checkout from about 40 seconds to about 11. We shared a build cache. The browser tooling now lives in a prebuilt image. Slow suites run in shards sized from measured test weights. Browser results get reused when their inputs have not changed.

The split: essential on the merge request, everything else on the clock

Once the budget existed, we had to decide what earns a place inside it. Our answer: a merge request runs only what the change can break.

The pipeline works out what you touched from the import graph. That picks the Go packages and the frontend tests. If the tool cannot map a change with confidence, it runs everything. That fallback keeps the selection honest. A selected run aims to use at most half the runner time of a full run.

A merge request pipeline holds these essentials:

• build and quick checks
• lint
• sharded unit and race tests for the affected code
• affected frontend tests
• a short browser run
• secret detection

It also reports every failure instead of stopping at the first one. An agent fixes all of its problems in one pass.

Everything else moves to an hourly run on main. That covers the full test suite, integration tests, the slow race tests, architecture guards, the complete browser regression and a scope report. A daily image rescan sits beside it. Pushes to main no longer start a pipeline of their own. Main is checked on a schedule instead, so runs finish instead of being cancelled by the next merge.

Flaky tests get a similar rule. A quarantined test stops blocking merge requests, but only for 14 days. Then it is fixed or removed.

Closing the loop: a red schedule becomes a task

A schedule without an owner is a dashboard nobody opens. Moving checks off the merge request only works if someone acts on the failures.

So the failures act on themselves. After each scheduled run, a final job reads which jobs failed. It folds shards into one entry per job and ignores attempts that passed on retry. For each red job it files a CodeHerder task. The task carries the failure lines and the log tail. It also holds links and a bisect result that names the merge that broke it.

It also checks for an open task on the same job first. If it cannot check, it files nothing. A missed task is cheaper than a pile of duplicates.

An agent then picks the task up like any other work. It plans the fix, writes it, passes review and merges. CI finds the problem, and the herd fixes it. No human has to read a red hourly run and write a ticket.

What it bought us

On the probe change, a one-line CI edit, the job count fell from 50 to 6. Frontend jobs on that change fell from 11 to none, because nothing in the change touched the frontend.

Throughput followed. On 30 September we merged 249 changes to main in a day, and 205 on 1 October. Those are the two best days we have recorded. The previous record was 201. After we split the work across two repos, we merged 181 and 210 on 4 and 5 October across both, against roughly 74 a day in the week before the rework.

These are daily records. We would rather claim the number we can defend. What we take from it: when checks stop blocking agents, agents ship.

What you can copy

You do not need our stack to copy this.

1. Set a hard per-job time limit. Pick a number your agents can live with, and make a test fail when config exceeds it.
2. Select checks from the dependency graph. Fall back to the full run when you cannot map a change.
3. Move anything that misses the budget to a schedule. Do not raise the limit.
4. Stop running main on every push. Check it on a clock.
5. Turn every scheduled failure into a task with an owner, the evidence and the culprit change attached.

The last step is the one most teams skip. It is also the one that makes the rest sustainable.

CodeHerder is built for this shape of work. It runs the herd and gates the merges. A red schedule becomes a task an agent picks up. If your agents already write and review good code, a smarter model will not help much now. A pipeline that keeps up with them will.