Prove a change before the herd runs it
Every change to how a herd of agents works is a guess until someone measures it. A new model, a higher effort level, a rewritten stage prompt, a new guardrail: each one feels like an improvement when you make it. Whether it is one depends on tasks you did not run both ways.
Our Agent Experience surveys find the paper cuts. They tell you what to try. They do not tell you whether the fix works. That is the job of an Evaluation, and it is the second feature in this series on safe, high-quality output.
An Evaluation is a controlled A/B test. You pick tasks your workspace already finished. You pick two or more setups. CodeHerder runs every setup on every task, grades each result with a check you wrote, and compares the setups in pairs. Below is how it works and what it showed when we ran it on our own tasks. At the end is the loop that improves a herd, with Evaluations as its proof step.
What an Evaluation is
A setup pairs a harness with a model and a credential. You can add an effort level, a launch config, skills, stage prompts, a workflow, wiki pages, guardrails or a pinned ch build. Exactly one setup is the baseline. Everything else is compared to it.
The Evaluation runs each setup on each task, repeats times, from 1 to 50. So the attempt count is setups × tasks × repeats. Four setups on six tasks with one repeat make 24 attempts. Above 24, ch evaluation run asks you to confirm before it spends anything, because every attempt is a real agent session on your own AI credentials.
At creation, CodeHerder freezes the whole definition into a hashed manifest. A model name becomes an exact model. A done task becomes its original text and its starting commit. Each skill becomes a fixed copy, and so do prompts and wiki pages. If you edit your workspace later, the Evaluation does not change. A seed fixes the task sample, the order of attempts and the statistics. The same seed and the same results give the same report.
A fair test
A comparison is only as good as its isolation. If one setup could reach your real repository, or learn what the grader looks for, the numbers would mean nothing. So each attempt runs sealed.
Each attempt is an agent session on an isolated device or in a microVM pool. It talks to a stand-in git host that mimics yours. It holds no CodeHerder token and no git-host credential. The address it would use for CodeHerder points at a sentinel that must see zero connections, and probes for cloud metadata endpoints find nothing. If an attempt reaches a real host, the attempt is invalid. Invalid attempts stay out of every figure, and the report lists each one with its reason.
Scoring happens after the container exits. The hidden check is a command you wrote, such as your test suite. It runs on the merged branch with no network, and the agent never sees it. A pass means the attempt completed and the hidden check passed. Reaching the done stage is not enough.
Each attempt also gets a merge request quality score. The score is blind, and it only observes. It never changes pass.
Reading the report
The report pairs each attempt with the baseline’s attempt on the same task and repeat. It drops any pair where either side is invalid or unfinished. Then it tests ten metrics: pass, turns, tokens, cost, wall time, session time, rework, git-host actions, failed CLI calls and merge request quality.
pass uses an exact McNemar test. The other metrics use a cluster bootstrap with 10,000 resamples. The threshold is 0.05. For each metric you get both averages, the difference, a 95% interval and one of three verdicts:
| Verdict | Meaning |
|---|---|
| Yes | The gap is larger than chance explains on this sample. |
| No | It is not, and the sample is big enough to say so. |
| Too few runs | The sample is too small to tell. The report says how many runs it needs for 80% power. |
We care most about the third verdict. If a report cannot say “not enough data”, you cannot trust its yes either. A comparison with fewer than 6 complete pairs carries a warning. Code writes the plain summary lines at the top from the comparison results. A language model does not.
Before you trust a result, test the test. With --calibrate, CodeHerder runs identical copies of your baseline against each other. The true difference is zero, so any metric that shows a real difference is noise, and the report marks it NOISE. We ran one on 27 October 2026: two identical setups, 3 tasks, 3 repeats. The pass rate showed no difference. Wall time and session time came back significant anyway, and were flagged. So we now read a wall-time gap of under about 1,500 seconds with suspicion at that sample size.
When the last attempt ends, an analyst agent reads the evidence and writes an assessment. It recommends adopt, reject or inconclusive for each setup. Each recommendation has reasons, and each pair gets a note. It runs read-only. It never sets a verdict. The report holds the numbers.
What we found on ourselves
On 28 October 2026 we ran our first A/B with setups that differ in model, effort or prompt. The baseline was base: Claude Sonnet 5.5 at medium effort. We compared it with three others:
opus: the same, with Claude Opus 5.5.high: the same, at high effort.prompt: the same, plus one line in thecodestage prompt, “Before you push, run the focused Go tests for the packages you changed.”
We used 6 real finished tasks with 1 repeat. That is 24 attempts, all valid, and 6 pairs per comparison.
The numbers:
opuscost $3.10 per attempt against $0.86 forbase, about 3.6 times as much. It used about twice the turns. Its merge request quality was higher by 0.13, with an interval of +0.05 to +0.20. Pass did not change.highcost $0.48 more per attempt thanbase. Its quality gain was not significant.promptchanged no metric significantly. One line of prompt text made no measurable difference to cost, speed or quality on these six tasks.- No setup changed pass significantly.
highandprompteach showed +17 points, but the report said it would need about 45 pairs to detect a 17-point difference, and we had 6.
We are not claiming Opus is better. This was one workspace and a small sample, and the tool said so itself. The report did answer the question we asked. Opus buys a measurable lift in merge request quality for roughly 3.6 times the price. For some stages that trade is right. For others it is not. We can now decide that stage by stage, from the figures.
Where it fits: the optimisation loop
You can run an Evaluation by hand. It also works as the proof step in a loop that runs on its own.
This is how the loop is meant to run, and most of it works today.
- Capture: on a schedule, CodeHerder samples a window of finished work. By default that is up to 50 attempts every 24 hours. It keeps the survey responses from the same window.
- Analyst: an analyst reads the sample and may propose one change.
- Candidate: the proposal becomes a candidate that tracks that change. The server checks it against the live target first, so a stale proposal is refused.
- Evaluation: an admin approves the candidate or, in
automode, the loop does. An Evaluation then compares the current setup with the proposed one. - Verdict: the report sets the verdict:
proved,disprovedorinconclusive. Nothing else does. Not the analyst that proposed the change. - Accept: a human admin accepts or declines. Agents cannot.
- Apply: an accepted change becomes a new version of its target, with an audit record.
- Roll back: an admin can write the old version back as a new version. History is never rewritten.
Nothing touches a target before step 6. Each proposal names the metric it should move and a quality guardrail metric that must not get worse. The default guardrail is pass.
Some parts are not there yet:
- The loop can test six kinds of change: stage prompts, models, harnesses, effort levels, launch configs, plus guardrails. Skill and wiki-page candidates cannot start an Evaluation yet.
- Nothing applies itself. In
automode the loop only starts Evaluations, and it stays inside a spend cap. The default cap is $0, which starts nothing. - An Evaluation runs on finished tasks. It does not run on future work. A
provedverdict is evidence. It is not a promise.
Try it
You describe an Evaluation in a YAML file:
name: opus-vs-sonnet
seed: 7
repeats: 3
providers: [gitlab]
tasks:
sample: {count: 10, types: [bug]}
hiddenCheck: {command: make test, timeoutSeconds: 900}
setups:
- name: baseline
baseline: true
harness: claude
model: claude-opus-5-5
credential: pool-a
- name: sonnet
harness: claude
model: claude-sonnet-5-5
credential: pool-a
Then run it:
ch evaluation run --from-file evaluation.yaml --wait
It prints the attempt count first. With --calibrate, it runs the A/A version of the same file. You need an owner or admin role and at least one isolated device. The same page is in the app under Set up, Optimisation, Evaluations.
The full mechanism is in Evaluations and The improvement policy. If you want to know whether your next change to the herd is worth shipping, get started and measure it.