Give your agents their own machine: what CPU contention costs a parallel herd
Concurrency is the whole point of orchestrating agents. One agent working alone is a chat window with extra steps. The value shows up when six stories are in flight at once, each in its own worktree, each unaware of the others.
Which raises a practical question: what should those six agents actually run on? We build from Auckland and run agents on both local machines and EC2 instances in the US, so we have production data on both. The answer turns out to have much less to do with the 12,000 km to the API than with whether anything else is competing for the CPU.
Here’s the measurement, and then what a machine that does this properly costs. Spoiler on the second part: about $62 a month, and it comfortably holds ten concurrent agent sessions.
What’s being measured
CodeHerder is an orchestrator for AI coding agents. You file a story, it walks it through
todo → plan → code → review → merge → verify → done, and each stage goes to a fresh Claude Code
or Codex process spawned on a machine you registered, in an isolated git worktree. The machines
are yours: laptops, EC2 instances, anything that can run the agent.
That gives us the comparison. The same workflow, the same stories, the same models, running on
Auckland developer machines and on m9g.xlarge instances in us-east-1 and us-west-2. Every API
call is recorded against the machine that made it, so the two device classes can be compared
directly.
Window is 29 June to 31 July 2026. Dollar figures for agent usage are notional list price on flat-rate subscription plans: what those tokens would cost at published API rates, not an invoice we paid. AWS figures are real prices, quoted below with their sources.
The geography costs less than you’d think
First, ground truth from the machines themselves rather than from traceroute mythology. Time to
first byte from api.anthropic.com, measured from each location:
| from | TCP connect | TTFB |
|---|---|---|
| Auckland machine | ~0.010 s | 0.30–0.37 s |
| us-east-1 | ~0.005 s | ~0.038 s |
| us-west-2 | ~0.005 s | ~0.027 s |
About 0.26 seconds per API call of genuine geographic disadvantage. That’s real, and it’s the number most people reach for when they explain why their agents feel slow.
Now put it in context. The unit that matters is the gap between calls: the end of one model response to the start of the next, which covers the round trip plus whatever the agent did locally with the result. Across 1.1 million call pairs:
| class | median gap | p90 |
|---|---|---|
| EC2 | 5.98 s | 22.56 s |
| local machine | 6.76 s | 21.61 s |
A 0.78-second difference on a six-second cycle. The Pacific crossing is in there, and it’s measurable, and it is not what decides whether your herd keeps up.
Do the arithmetic on a whole story and the point sharpens. A median story is about 190 API calls. At the measured 0.26 s penalty, that’s 0.8 minutes on a roughly 50-minute story, under 2%. Co-location is worth having if it’s free. It is not the lever.
The load is the lever
Take stories where all the work ran on one class of machine, and measure lead time from the first
plan transition to done:
| class | stories | median lead |
|---|---|---|
| EC2 | 1,757 | 58.2 min |
| local machine | 925 | 67.7 min |
+9.5 minutes, +16.4%. But that aggregate mixes together machines running one session and machines running eight, so it doesn’t isolate anything. Control for that instead: same comparison, bucketed by how many agent sessions were already live on the machine when the story started.
| concurrent sessions | EC2 median | local median | gap |
|---|---|---|---|
| 0–2 | 42.1 min | 45.4 min | +7.8% |
| 3–5 | 43.5 min | 60.8 min | +39.8% |
| 6+ | 96.8 min | 139.9 min | +44.5% |
There it is. On a quiet machine the two classes are close, 7.8% apart, which is the same order as the ~2% the per-call arithmetic predicts plus ordinary noise between two medians. Load them up and they diverge hard: 40% at three to five concurrent sessions, 45% beyond six.
Note also what happens to both columns as load rises. EC2 goes from 42 to 97 minutes; the local machines go from 45 to 140. Concurrency costs something everywhere. It just costs a great deal more on a machine that has other jobs.
Why parallel agents punish shared machines
Look at what a single coding agent actually does between API calls. It reads dozens of files. It greps the tree. It runs the test suite. It builds. It runs a linter. Then it waits about six seconds for the next response and does it again, a hundred and ninety times. In the token breakdown from our production data, input is 99.5% of all token volume, which is the same fact from the other side: this is a read-heavy workload, and reading means local I/O and local CPU.
Now run five of those on your laptop. Five test suites, five build caches, five git worktrees
being checked out, all competing for the same cores and the same disk, and competing with your
editor, your language server, your browser, and whatever your OS decided to index this afternoon.
The agents don’t slow down because the model got slower. They slow down waiting for a CPU to run
go test on.
That is what the +40% at three to five sessions is. It isn’t a network effect, and no amount of co-location fixes it.
Our EC2 boxes show what the headroom looks like. Across the window, the highest number of simultaneous agent sessions reached on single instances:
| device | sessions run | max concurrent | p99 | median |
|---|---|---|---|---|
| EC2 instance A | 2,501 | 17 | 10 | 3 |
| EC2 instance B | 8,380 | 13 | 10 | 3 |
| EC2 instance C | 8,506 | 9 | 8 | 4 |
Ten concurrent sessions on a four-core box is the honest working figure, and it holds because those cores aren’t doing anything else. One caveat on how to read that ceiling: time-weighted mean concurrency on those boxes is only 1.4 to 2.6, so they aren’t saturated most of the time; the busiest stretches are where they run eight to ten. If your backlog is bursty, size for the burst.
What the hardware costs
CodeHerder has a “Launch on AWS” button for exactly this. It’s a CloudFormation template that
stands up a device: a Graviton instance in an auto-scaling group, a persistent EBS data volume for
worktrees, the agent harness installed, and the device registered to your workspace with a
short-lived registration token so nothing long-lived gets pasted into a template. Default instance
type is m9g.xlarge, default to spot, with a fallback ladder through m8g, m7g, m6g and the
r-family equivalents so a spot interruption doesn’t leave you without a machine.
Here’s the cost-out. m9g.xlarge is 4 vCPU and 16 GiB on AWS Graviton. Spot prices are the
trailing seven-day average from the EC2 spot price history at the time of writing; on-demand and
storage come from the AWS price list API. us-west-2:
| line | rate | 24/7 monthly |
|---|---|---|
| m9g.xlarge, on-demand | $0.19568/hr | $142.85 |
| m9g.xlarge, spot (7-day region mean) | $0.0770/hr | $56.21 |
| 70 GiB gp3 (20 root + 50 data) | $0.08/GB-mo | $5.60 |
| all-in, spot, running continuously | $61.81 |
Spot runs 61% below on-demand, and it’s been stable: over the last week the busy AZs ranged $0.0787 to $0.0889, and us-west-2d averaged $0.0597. Quoting the region mean rather than the cheapest AZ keeps the figure honest if you don’t get to pick.
Divide it by what the box holds. At ten concurrent sessions, $61.81 a month is $6.18 per month per concurrent agent session. At a more conservative five, $12.36.
If your backlog only runs during working hours, the arithmetic gets better, because the auto-scaling group can run to a schedule. Twelve hours a day, five days a week is 264 hours: $20.33 of compute plus the $5.60 of storage that persists either way, so $25.93 a month. The EBS volume keeps the worktrees and the device identity, so the machine comes back as the same device rather than a new one needing to re-clone everything.
One thing that isn’t in the table: data transfer out is small for this workload, because the traffic is API calls, and it’s the input that’s enormous, and inbound traffic is free.
For scale, one of those boxes ran over 8,000 agent sessions in five weeks. At $61.81 a month, that’s under a cent of compute per agent session, against notional agent usage that is orders of magnitude larger. The hardware is not the expensive part of running agents. It’s the part that determines whether the expensive part gets to run in parallel.
What to do about it
Give agents their own machine
Not a partition of your laptop, not “I’ll just close Slack.” The load-matched numbers say a dedicated machine at three to five concurrent sessions is 40% faster, and 45% faster beyond six. That’s the largest effect in this entire analysis, and it’s the cheapest one to act on.
Put it near the API if convenient, but don’t reorganise anything to do it
It’s worth single digits at low load: real, measurable, and the smallest number here. If you’re already in AWS, pick a region near the API and take it. If your code, your CI or your compliance boundary wants the machine somewhere else, put it somewhere else.
Size for burst concurrency, not average
Mean concurrency on our boxes is under 3, peak is 9 to 17. Averages will tell you a two-core box is plenty right up until the moment it isn’t.
Watch the load metric, not the latency metric
The tell for a machine that’s holding you back isn’t slow API calls, it’s lead times that inflate when several stories are in flight. Per-call latency looks fine the whole time.
Two ways to try the hardware side of this: launch a device on AWS from the template, or register a machine you already have. Both end up in the same place, which is agents running somewhere that isn’t competing with your editor for CPU.
Method notes
Latency and lead-time figures cover 29 June to 31 July 2026 across the local and EC2 devices in our own workspace, from the orchestrator’s own records. Inter-call gap is the first timestamp of one API call minus that of the previous one within a session, capped at 300 s so that idle stretches between bursts of work don’t dominate. API calls are deduplicated to one row per model response: the harness writes one transcript line per content block, and every line from one response repeats that response’s usage figures.
Concurrent load is the count of sessions live on that machine class when the story started,
counted machine-wide rather than per workspace, from session start and exit intervals. Lead time
is the first plan transition to the first done transition. Stories are attributed to a device
class only when all their work ran on that class, which is 2,682 of the window’s shipped stories.
Spot prices are the trailing seven-day mean of hourly describe-spot-price-history observations
for m9g.xlarge, Linux/UNIX, across all us-west-2 availability zones at the time of writing; they
change. On-demand and gp3 rates are from the AWS price list API for us-west-2. Monthly figures use
730 hours for continuous operation and 264 for a 12×5 schedule. Verify current prices before
budgeting on any of this.
This is one team’s production history on one codebase, not a benchmark.
