CodeHerderSearch⌘KRequest access →

Setup comparison

Put two setups side by side on the same stages and see which one actually held up.

Stage signals rates a stage overall. ch quality stage-signals --section setups breaks that rate down by which setup ran it. Setup comparison answers the narrower question those two leave open: given exactly two setups, which one is doing better on the same stages?

From the app

Open Work → Reports, then the Quality card under “Quality and delivery”, and find the Setups panel. Pick a row, open its Compare with… menu, and choose a sibling setup — one that ran the same stage on the same task class — to open the comparison with both sides already filled in. The comparison page also has its own Window, Setup A, and Setup B pickers, so you can start from there and choose both sides yourself. Window is the nine-option picker from Choosing a window, and it opens on Last 30 days. All time is the widest choice. The Setup comparison card on Work → Reports opens the same page.

Any workspace member can open it. There’s no extra role or plan requirement.

From the CLI

ch quality compare --setup <a> --setup <b>

Pass --setup exactly twice: the first is setup A, the second is setup B. Each value is a setup selector — at least 8 hex characters of a setup’s own hash, or the full hash.

The easiest way to get both selectors is to skip the hashes: the Compare with… menu and the comparison page’s own pickers fill both sides in for you. To build the command yourself, run ch quality stage-signals --section setups --json and read a cell’s setupHash. The plain table doesn’t print that value — its STAGE-HASH column is a different hash, covering the stage’s prompt alone.

An optional workspace name, slug, slug path, or full ID can come first; without one, the command uses the workspace the CLI is pointed at.

The window flags work the same way they do everywhere else in this report family: --window today|yesterday|week|month, or --since with a duration (3d, 12h) or a full RFC 3339 timestamp — the two are mutually exclusive, and without either the report covers the last 90 days. --since reaches back at most 400 days. The CLI has no end date. Add --json for the raw data instead of the table.

Reading the report

The header names both setups — harness, model, and stage content — then a cost line:

SETUP A: a1b2c3d4  harness=claude model=sonnet stage-content=1a2b3c4d
SETUP B: e5f6a7b8  harness=codex  model=gpt    stage-content=5e6f7a8b

cost/completed-task:  A=$1.42  B=$1.08

The value right after SETUP A: is that setup’s own hash, so you can feed it straight back to --setup on your next run. The stage-content= value is the other hash mentioned above, and won’t resolve as a selector.

Below that, one row per stage, with the same figures for each side:

STAGE   A-RATE  A-REWORKED  A-DECIDED  B-RATE  B-REWORKED  B-DECIDED
code    92%     4           50         78%     4           18
review  95%     3           60         90%     2           20
  • RATE — first-pass rate: the share of decided attempts that came out clean, the same figure ch quality stage-signals --section setups reports for one setup alone.
  • REWORKED — how many of that setup’s attempts at this stage got sent back.
  • DECIDED — the denominator behind RATE: accepted plus reworked attempts. A stage with a small DECIDED count hasn’t produced enough evidence to trust the rate next to it.

Read this before you act on it

  • This adds no new score. Every number here is read straight off the same per-stage, per-class, per-setup cells ch quality stage-signals --section setups already shows — comparison just narrows the view to two setups at a time.
  • A small DECIDED count means the gap between A and B is probably noise, not a real difference. Don’t call a winner off a handful of attempts, and read each pair against its own denominator.

Last updated

Try it on one repository for two weeks.

File a task. An agent builds it. A person reads every request. You see the cost of each task.

Try "pricing", "connect a device", or "who reviews the code"

↑↓ move · ↵ open · esc close