10 things we learned running 100 billion tokens through a herd of AI agents
For the last two months we’ve built CodeHerder using CodeHerder. That left us with a dataset we didn’t set out to collect, and three findings in it we weren’t expecting:
- 91.3% of shipped stories needed no rework at all.
- Input is 99.5% of token volume. Agentic coding is a reading workload.
- The quality gate that runs after the merge still rejects 2.3% of stories.
The other seven items are the operational detail behind those three.
What you’re looking at
CodeHerder is an orchestrator for AI coding agents. You file a story, usually a couple of sentences. It walks that story through a fixed sequence of stages:
todo → plan → code → review → merge → verify → doneEach stage goes to a fresh agent, Claude Code or Codex, spawned on one of your own registered machines in an isolated git worktree. The agent does one job and exits. Any stage can send work back to an earlier one, which is how review rejections happen. Nobody sits and watches.
That structure is the only reason this post can exist. Each stage is a recorded object rather than a prompt convention, so every token spent has an address: which story, which stage, which agent, which model, what happened next.
Since the first commit on 26 May, the herd has run 1,166,207 assistant API calls and 109.8 billion tokens across 30,134 agent sessions, finishing 5,742 tasks. This is one herd’s production history, weighted toward the kind of software we happen to build; it isn’t an industry benchmark. (Full pricing and counting methodology is in the Method notes at the end.)
1. A shipped story costs $7.54 and 50 minutes
2,677 user stories went from a one-line description to merged code without a human writing or editing any of it. For the 91.3% that needed no rework:
| percentile | cost |
|---|---|
| p10 | $4.57 |
| median | $7.54 |
| p75 | $10.06 |
| p90 | $14.44 |
The median tells you what a typical story costs, and the p90 tells you whether that number is operationally useful: at 1.9× the median, it is. We’d gone in expecting a fat tail of stories quietly spiralling into three figures, and it isn’t there. Median lead time, first plan to done, is 49.9 minutes; the mean runs higher than the median everywhere in this post ($8.78 against $7.54 on this same cohort), because the tail pulls it up.
One thing to flag, because it’s the most interesting number we’re watching. This median is not stable, and it has been drifting upward over the last few weeks. The obvious suspect is our own codebase getting bigger; a second suspect is the reasoning model each judgment stage runs on. (It was the model, by a wide margin: here is the analysis.)
2. In our default configuration, planning costs more than implementation
Where the money goes on a median story, in workflow order:
| stage | share of cost |
|---|---|
| plan | 31.4% |
| code | 28.6% |
| review | 15.5% |
| merge | 8.9% |
| verify | 15.6% |
Planning is the single most expensive stage, 10% more than the code stage that consumes its output. A common assumption about LLM coding cost is that generation is the expensive part; in our data it isn’t.
But two things are going on here, and one of them is a decision rather than a discovery. The first is model choice, and it’s a real confounder: plan, review and verify run 99.7–99.9% of their calls on the reasoning tier, while code runs 97.9% and merge 99.9% on the faster, cheaper tier. Each stage declares its own model tier in configuration, so a good chunk of that 31% is us choosing to spend the expensive model on judgment.
The second is workload, and it survives the first. Reading enough of a large codebase to plan a correct change appears to cost more than emitting the resulting diff. That’s the part that transfers to your setup regardless of how you’ve configured tiers. If you’re budgeting a herd of agents, budget for comprehension, and check which model your planning stage runs on before you conclude your bill is a code-generation problem. The limit on all of this: it’s one configuration on one codebase.
3. 91% of stories ship with zero rework
2,444 of 2,677 shipped stories cleared every gate on the first attempt: never sent back once, as distinct from eventually passing after a couple of rounds.
A “gate” here is just a stage with the power to reject. The reviewing agent can push the task
back to code, which re-spawns a coding agent with the reviewer’s complaint attached. That loop
is what most of the numbers in this post are measuring.
This cuts against the prevailing wisdom that agent output needs constant babysitting. What it suggests it needs is structure: we think stage isolation and a fresh review context contribute to the rate, but this dataset can’t isolate their individual effects, since we have no matched control where the same stories ran without gates. It’s a correlation on one workload, not a demonstrated cause.
4. 97% of every dollar reached a finished task
Every task that consumed tokens in the priced window, grouped by where it ended up:
| task status | share of spend |
|---|---|
| done | 96.5% |
| blocked | 1.6% |
| cancelled | 1.0% |
| still in flight | 0.9% |
“Agents burn money spinning on nothing” is a fair thing to worry about. Across the whole herd, it’s a rounding error: 96.5% of attributable spend went to a task that finished, at an average cost of $9.43 per finished task of any type.
One caveat on what that measures: it’s a statement about terminal outcomes, not efficiency. A task that eventually finishes can still have burned money on loops that went nowhere along the way, and that spend lands in the 96.5% row. The claim is only that we don’t lose meaningful money on work that never arrives, which is the specific failure mode people ask about.
5. Review approved it. Merge accepted it. It was still broken.
Our stage order is review → merge → verify, which surprises people: verification happens once
the code is on the main branch, not before it lands. That ordering turns out to matter more than
anything else on this list.
| gate | question it asks | resolutions | sent back | reject rate |
|---|---|---|---|---|
| review | is this diff correct? | 3,017 | 233 | 7.7% |
| merge | can this actually land? | 2,902 | 11 | 0.4% |
| verify | does it work now that it’s merged? | 2,728 | 63 | 2.3% |
The reject rate doesn’t decay to nothing. It bottoms out at merge and climbs back to 2.3% after the code has landed, because post-merge verification catches things a diff review structurally can’t: an incomplete implementation, an integration conflict with whatever else merged in the meantime, an environment problem.
(Those counts are stage transitions across all story tasks, not distinct stories, so one story
sent back twice appears twice, and the totals fall at later stages because stories drop out
along the way. Merge’s 0.4% code-reject rate is genuinely tiny; separately, 169 of its
resolutions went to blocked, a protected branch, an external approval, a permission only a
human can grant. That’s the workflow correctly stopping to ask rather than guessing, and those
stories ship once unblocked.)
What’s true regardless of the exact mix is that some failures only become observable against the merged state, and diff review alone can’t reliably reproduce that state. It’s the same reason human teams run CI on main rather than treating a green MR pipeline as proof. Three gates asking three different questions beats one gate asked three times.
6. The gate that catches the most is also the cheapest one to run
Reviewing a story costs $1.16 and 3.5 minutes, 15.4% of the total and half what planning costs. It’s also the gate that catches the most by a wide margin: across all three gates, review accounts for 233 of the send-backs, against 63 at verify and 11 at merge.
The money isn’t in the checking. It’s in the rework. A story review sends back costs $17.93 on average against a clean run’s $8.78, and 126 minutes against 50. Read the two together and review is a fixed 15% premium on every story, set against a cost that runs 2.0× higher on the stories where a defect gets through.
We’re deliberately not turning that into a break-even claim, because the dataset doesn’t contain the comparison it would need: $17.93 already includes the rework review itself triggered, and nobody ran our workflow with the review stage switched off. The defensible version is that review is inexpensive relative to the rework associated with the defects it catches. The companion post works through the arithmetic properly.
What we like about this one is that it’s arguable at all. Normally you defend code review with faith and anecdote and a story about the time someone shipped a null check. Here it’s a line item: 15% of every story, catching one story in thirteen. Once a quality gate has a price, it stops being a matter of engineering culture and becomes an operating cost you can reason about like any other.
7. When review does send work back, one round almost always settles it
204 stories got rejected by review at least once. What happens next is the part we’d have bet against:
| times review sent it back | stories | eventually shipped |
|---|---|---|
| once | 184 | 94.0% |
| twice | 14 | 92.9% |
| three or more | 6 | 100.0% |
90.2% of all rejected stories needed exactly one round of rework, and 97.1% needed no more than two. The failure mode we expected was the endless loop, where a coding agent and a reviewing agent argue past each other until someone kills the task. It’s a thin tail rather than the shape of the distribution.
That’s by design rather than good luck. A stage that keeps getting rejected escalates to a stronger model and then hard-stops at a configured cap, so a story can’t bounce between code and review indefinitely. Read the tight distribution in item 1 as a property of that cap as much as of the models.
8. Agentic coding is a reading workload, not a writing one
Of the 83.7 billion tokens in the priced window, only 0.41 billion were output, or 0.5%. The rest breaks down like this:
| token category | share |
|---|---|
| cache reads | 97.3% |
| cache writes | 2.1% |
| uncached input | 0.1% |
| output | 0.5% |
Cache reads alone are 97.3% of every token spent: agents mostly re-read context rather than generate new code, and per call an agent consumes about 97,543 tokens of input to emit about 491.
We should be precise about what “input” means here: it’s tempting to read it as the agent reading your codebase, but that’s only part of it. Input is everything sent to the model on each call, the system prompt and stage brief, the conversation history so far, tool results including file contents and test output, and the code the agent itself wrote earlier in the session.
The 97.3% cache-read share is the tell. Most input on any given call is context that was already assembled on a previous call and is being re-sent. That explains most of what’s initially confusing about agent economics: where the bill comes from, and why latency behaves the way it does. It’s also why context management is the whole ballgame, and why “the model writes code fast” was never the bottleneck anyone should have cared about.
9. Nearly every dollar resolves to a named stage
95.1% of API calls in the priced window resolve to exactly one workflow stage. That number is the foundation under every other item here: cost is attributed to the stage a session was activated for, not to whatever status the task happened to hold when the call landed. Those differ whenever a call arrives after the task has moved on, and grouping by live status files planning work under coding. Getting that right is the difference between “our agents cost about this much” and a receipt.
The remaining 4.9% belongs to sessions that legitimately span more than one stage. Rather than guess, the cohort tables in this post and the companion piece simply require all five stages attributed, which is why their sample sizes are smaller than the headline story count.
10. It never sleeps
The machines running the agents are just machines you register: laptops, EC2 boxes, whatever has capacity. The orchestrator assigns work whenever registered capacity is available.
| measure | value |
|---|---|
| tasks finished per day, mean | 157.6 |
| tasks finished on the busiest day | 303 |
| quietest hour of the day, as a share of the busiest | 22.8% |
The daily figures cover 24 full days, over which the herd finished 158 tasks a day on average. The number we keep coming back to is the trough: aggregate every call by hour of day across the whole window and the quietest hour still runs at 22.8% of the busiest. A human team’s throughput graph doesn’t look like that; it has a working day in it, and nights that go to zero.
We’re deliberately not converting that into a headcount comparison, because we have no human baseline, no size normalisation and no quality comparison to convert it with. The claim is just the shape: work continues while the person who filed it is asleep, because nothing in the loop waits for anyone to be awake.
The thing that ties these ten together, and the reason we could answer any of them, is that none of it required instrumentation we added for the post. It’s the orchestrator’s own bookkeeping: every call attributed to a run, every run to a workflow stage, every stage transition to an actor.
A single coding agent may well be capable of producing similar individual changes. What a chat transcript can’t give you is the process-level attribution: that planning costs more than coding, and that your cheapest gate is the one doing most of the catching. It’s also what shows that a gate after the merge still finds 2.3%, and that when work does come back it almost always comes back once.
Those aren’t facts about language models. They’re facts about a process, and you only get them if the process is made of things a computer can count. Which is the part we didn’t expect going in: we built the workflow to keep agents from stepping on each other, and what we got as a side effect was a software organisation legible enough to run on numbers instead of vibes.
Next we take the $8 story apart stage by stage: the itemised receipt for a story no human wrote. If you’d rather just try it, CodeHerder is here.
Method notes
A call means one billable request to the model, along with the tool calls it makes. A single model response often arrives as several pieces, some thinking, some text, some tool invocations, and all of those pieces belong to one call and are counted once. The harness also injects synthetic messages that never bill, and those aren’t counted.
Dollar figures come from a narrower slice: 29 June onward, 853,835 calls. Cache writes are priced by how long the cache entry lives, and our per-call record of that starts on 29 June. Before then we can count tokens exactly but can’t price a cache write to the cent, so we don’t quote dollars for that period.
We run agents on flat-rate subscription plans, so nothing here is an invoice we paid. Every figure is notional list price: what those tokens would have cost at published per-model API rates, an estimated equivalent usage cost and nothing more. Each call is priced at its own model’s rate for its own token category, since uncached input, cache writes, cache reads and output all price differently, and cache writes differently again by cache lifetime.
Done is the terminal workflow state, reached only after post-merge verification passes, and shipped means the same thing. A story is one user story; the herd runs other task types alongside them. Currency runs to two decimals, time to one, percentages to one.
