Prove a change before the herd runs it
An Evaluation runs two setups on the same finished tasks, grades them with a check the agent cannot see, and reports paired results. Here is how it works.
Every agent run has a price. See it per feature, not just at the end of the month.
Cost, here, means the token spend and compute time behind a shipped change. We track it down to the feature and the task, not just the account total. CodeHerder logs every session's spend so a $4 bug fix and an $80 feature are both visible before they surprise you.
This matters because agent spend does not scale like a headcount. A model upgrade, a bad prompt, or a task that loops turns a five-minute job into an expensive one. A monthly bill will not tell you which task did it. Line-item visibility does.
Read this hub for real cost breakdowns from tasks we ran ourselves. It also covers the levers that move spend the most, and how to set a budget an agent respects.
Request access →An Evaluation runs two setups on the same finished tasks, grades them with a check the agent cannot see, and reports paired results. Here is how it works.
Agents write and review code faster than CI can check it. Here is the budget, the split and the loop that took CodeHerder to its best merge days ever.
One short, vague brief, run 84 times by agents. Median cost about $18, median time 90 minutes. Here is what shipped and what it cost.
Keeping CLAUDE.md small is best practice, but ours hit 231KB across 37 correct commits. Splitting it cut cost per turn 28%, turns per task by a third.
Agent cost doesn't track codebase size. Swapping our judgment stage's reasoning model moved a story's cost 2.4x, and its lead time nearly 3x.
Parallel Claude Code sessions need dedicated hardware. Load-matched data puts one machine 40% ahead, and a spot m9g.xlarge holds ten sessions.
944 user stories shipped in a week, with no human writing the code. Here's what the median one cost, and what each quality gate caught.
Two months of CodeHerder data: 1.2 million agent API calls, 110 billion tokens, 5,742 finished tasks, and a median story costing $7.54, done in 50 minutes.
Token spend is easy to lose track of with more than one agent running. Here's how to keep it visible per task, agent, and model.
File a task. An agent builds it. A person reads every request. You see the cost of each task.