Setup comparison
Put two setups side by side on the same stages and see which one actually held up.
Stage signals rates a stage overall. ch quality stage-signals --section setups breaks that rate down by which setup ran it. Setup comparison answers the narrower question those two leave open: given exactly two setups, which one is doing better on the same stages?
From the app
Open Work → Reports, then the Quality card under “Quality and delivery”, and find the Setups panel. Pick a row, open its Compare with… menu, and choose a sibling setup — one that ran the same stage on the same task class — to open the comparison with both sides already filled in. The comparison page also has its own Window, Setup A, and Setup B pickers, so you can start from there and choose both sides yourself. Window is the nine-option picker from Choosing a window, and it opens on Last 30 days. All time is the widest choice. The Setup comparison card on Work → Reports opens the same page.
Any workspace member can open it. There’s no extra role or plan requirement.
From the CLI
ch quality compare --setup <a> --setup <b>
Pass --setup exactly twice: the first is setup A, the second is setup B. Each value is a setup selector — at least 8 hex characters of a setup’s own hash, or the full hash.
The easiest way to get both selectors is to skip the hashes: the Compare with… menu and the comparison page’s own pickers fill both sides in for you. To build the command yourself, run ch quality stage-signals --section setups --json and read a cell’s setupHash. The plain table doesn’t print that value — its STAGE-HASH column is a different hash, covering the stage’s prompt alone.
An optional workspace name, slug, slug path, or full ID can come first; without one, the command uses the workspace the CLI is pointed at.
The window flags work the same way they do everywhere else in this report family: --window today|yesterday|week|month, or --since with a duration (3d, 12h) or a full RFC 3339 timestamp — the two are mutually exclusive, and without either the report covers the last 90 days. --since reaches back at most 400 days. The CLI has no end date. Add --json for the raw data instead of the table.
Reading the report
The header names both setups — harness, model, and stage content — then a cost line:
SETUP A: a1b2c3d4 harness=claude model=sonnet stage-content=1a2b3c4d
SETUP B: e5f6a7b8 harness=codex model=gpt stage-content=5e6f7a8b
cost/completed-task: A=$1.42 B=$1.08
The value right after SETUP A: is that setup’s own hash, so you can feed it straight back to --setup on your next run. The stage-content= value is the other hash mentioned above, and won’t resolve as a selector.
Below that, one row per stage, with the same figures for each side:
STAGE A-RATE A-REWORKED A-DECIDED B-RATE B-REWORKED B-DECIDED
code 92% 4 50 78% 4 18
review 95% 3 60 90% 2 20
- RATE — first-pass rate: the share of decided attempts that came out clean, the same figure
ch quality stage-signals --section setupsreports for one setup alone. - REWORKED — how many of that setup’s attempts at this stage got sent back.
- DECIDED — the denominator behind RATE: accepted plus reworked attempts. A stage with a small DECIDED count hasn’t produced enough evidence to trust the rate next to it.
Read this before you act on it
- This adds no new score. Every number here is read straight off the same per-stage, per-class, per-setup cells
ch quality stage-signals --section setupsalready shows — comparison just narrows the view to two setups at a time. - A small DECIDED count means the gap between A and B is probably noise, not a real difference. Don’t call a winner off a handful of attempts, and read each pair against its own denominator.
Related guides
- Stage signals — the per-stage rate this report breaks down into two setups
- Stage-attempt detail — the individual attempts behind every cell
- Understanding costs — where cost/completed-task is reported in full
- Cost and rework per completed task — the task-level cost and first-pass numbers this report doesn’t break down by model or harness
Last updated