CodeHerderSearch⌘KRequest access →

Evaluations

Run a controlled A/B test of two or more setups on the same finished tasks, and read a paired report before you switch.

An Evaluation is a controlled test of how CodeHerder runs a task. You pick a set of tasks your workspace already finished. You pick two or more setups, such as a different model, harness, effort level, launch config, skill, stage prompt or workflow. CodeHerder runs every setup on every task, grades each result with a check you wrote, and tells you which setup did better and at what cost.

Use it to answer questions like: does the cheaper model still pass our tests? Does the new code stage prompt need fewer rework rounds? Is this launch config worth its price?

Before you start

  • Roles. Creating, running, cancelling and assessing an Evaluation need a workspace owner or admin. Downloading an attempt’s artifacts needs a human owner or admin. Only the person an attempt runs as can attach to its session with ch session attach. Open an attempt’s session page in the app to see it run. A credential bound to an agent session is refused. Any member can list Evaluations and read their reports. Creating or promoting a case needs a human owner or admin, so an agent credential is refused. Validating a case and publishing a suite need the admin role.
  • A workspace, not a group. An Evaluation belongs to one workspace. A group cannot create one.
  • An isolated device. Attempts run only on a device at isolation level isolated. If none qualifies, CodeHerder refuses the Evaluation. The refusal lists each device and why it did not qualify. A reason can be that the device is not isolated, that it does not cover a setup’s architecture, or that its ch is out of date. If a device goes offline or falls below the release floor after creation, its attempts stay pending. CodeHerder keeps retrying them with no time limit. If the definition lists requiredDeviceCapabilities, the device must also advertise every one of them (see Pin it to devices). See Isolating agent runs on a device to set one up.
  • No credential to pick. An attempt draws its AI credential from the device’s pool, as a normal session does. A capped entry never fails an attempt. See AI credentials on a device.
  • Money. Every attempt is a real agent session on your AI credentials. Work out the attempt count first (see below).

The definition file

You describe an Evaluation in a YAML or JSON file.

name: opus-vs-sonnet
seed: 7
repeats: 3
budget:
  attemptWallSeconds: 3600
providers: [gitlab]
requiredDeviceCapabilities: [region:eu]
tasks:
  sample: {count: 10, types: [bug]}
hiddenCheck: {command: make test, timeoutSeconds: 900}
workflow: story
setups:
  - name: baseline
    baseline: true
  - name: sonnet-code
    overrides:
      - stage: code
        launchConfig: {model: claude-sonnet-5-5}
        prompt: ./prompts/code.md

The top-level keys:

  • name names the Evaluation.
  • seed makes the random parts repeatable: which tasks a sample draws, the order attempts run in, and the report’s statistics. The same seed gives the same report from the same results.
  • repeats is how many times each setup runs each task, from 1 to 50. More repeats smooth out the luck of a single run.
  • budget.attemptWallSeconds caps how long one attempt may run.
  • budget.stallSeconds is optional. It ends an attempt that makes no progress for this many seconds. The default is 20 minutes (1200). See A silent attempt.
  • providers lists the stand-in services each attempt talks to: github, gitlab or jira. Use [gitlab] for ordinary code tasks. A stand-in git host answers in place of your real one. An attempt can still reach most other sites on the internet. It cannot reach CodeHerder or a real git host.
  • requiredDeviceCapabilities is optional. It lists the capability tokens a device must advertise to run an attempt. See Pin it to devices.
  • tasks names the tasks. Use exactly one of:
    • sample draws a seeded random sample of done tasks. count is required. You can narrow the draw with repo, types, workflow, since and until (as RFC 3339 timestamps, for example 2026-09-01T00:00:00Z).
    • ids lists done tasks by ID.
    • suite runs a published version of an eval-library suite, for example suite: core-small@3. You must give the @N version. The object form adds tags (keep the cases that hold every tag) and sample (then draw that many, seeded). Each case runs as a task with its own hidden check. If the case has a hidden patch, CodeHerder applies it to the merged branch before the check. A patch that does not apply fails the check. The preview warns when a case was validated on a different stage image from the Evaluation’s. The report names each task by its case, such as fix-cursor@2.
    • taskSet writes tasks out by hand. Each entry needs a key, a repoUrl and a baseSha. It can add a title, a description, a type, a fields map and its own check.
  • checks maps a verification id (test, lint or typecheck) to a command. CodeHerder writes each command into every matching verification in every setup, so all setups share one set of checks.
  • hiddenCheck is the command that grades an attempt, with an optional timeoutSeconds. The agent never sees it. A taskSet entry’s own check and a suite case’s own check win over it. For ids and sample tasks you can leave it out. CodeHerder then uses the test verification command from the verify stage of the task’s workflow. If CodeHerder finds no check for a task, it refuses the Evaluation.
  • workflow names the workflow under test, such as story. It is required. Every task must run it.
  • setups lists the setups (see below). Exactly one must have baseline: true.

Each setup has a unique name of at most 64 characters. It starts with a letter and uses letters, digits, hyphens or underscores. A setup is the workflow plus overrides. An override changes one node of the workflow. Each override names a stage or an agent, plus one or more of these:

  • prompt or definition replaces a stage’s prompt or its definition. A prompt replaces the whole compiled prompt, so your workspace’s stage overlays no longer apply to that stage.
  • instance patches how the workflow uses that stage, such as its escalation or gate.
  • agent changes which agent runs a stage.
  • launchConfig patches a launch config, for example model, effort, harness or args.
  • workflow on its own replaces the whole composition of the workflow under test.

A setup has no harness, model, effort, launchConfig, stagePrompts or workflow key. CodeHerder refuses a definition that sends one and names the field. Two commands help you write overrides:

ch evaluation tree story --targets
ch evaluation preview --from-file evaluation.yaml
  • tree prints the stages of a workflow as your workspace runs them today, with each stage’s agent, launch config and guardrails. --targets also lists what you can override.
  • preview checks a definition and stores nothing. For each setup it prints the stages and marks the ones the setup adds (+), removes (-) or changes (*). It then prints the attempt count. If CodeHerder would refuse the definition, it prints each problem and exits with status 1. Any member can run it. It skips the device checks that create runs.

These setup keys are optional:

Key What it does
baseline Marks the setup every other setup is compared to.
overrides Lists the node overrides of the setup. A launchConfig override patches a launch config of a stage or an agent. See Agent personas.
skills Lists skill slugs to include.
wiki Lists wiki pages to include.
chBuild Pins the ch build the attempt runs. Use a released version, or sha256:<digest> for a build you uploaded with ch evaluation build upload. ch evaluation build list shows your uploaded builds.
guardrails Lists the guardrails the setup’s attempts run. Each entry is a workspace guardrail slug, or slug with inline files that the workspace does not store. Name each file by its path from the definition file. Keep all of a guardrail’s files in the directory of its GUARDRAIL.md. A setup with no guardrails runs none, even when your workspace has some.

To compare many combinations without writing each one, replace setups with a grid. It takes a template setup, an axes block listing overrides (each names a stage or an agent, a field such as launchConfig.model, and its values), and a baseline that picks which combination is the baseline. CodeHerder generates one setup per combination and names it from its values.

CodeHerder resolves everything at creation: a model name becomes an exact model, a done task becomes its original text and starting commit, and each skill, prompt, workflow and wiki page becomes a fixed copy. A stage’s prompt is the one production would run: the stage’s library prompt with your workspace’s stage overlays applied. A setup that overrides the prompt of a stage runs your text alone, with no overlays. Later edits to your workspace do not change an Evaluation that already exists.

Pin it to devices

requiredDeviceCapabilities is a list of capability tokens. A device runs an attempt only if it advertises every token on the list. A token can be one the device detects itself, such as bin:docker, or a label you set. Use the list when your tasks need something that only some of your devices have, such as a region or a GPU. Set a device’s labels with ch device edit <device> --cap gpu --cap region:eu. --cap replaces the whole set, so list every label you want to keep. See Capabilities.

  • The list holds at most 20 entries. An entry cannot repeat.
  • CodeHerder refuses a bad list and names requiredDeviceCapabilities in the refusal. preview and create both check it.
  • ch evaluation preview prints the list as “required device capabilities”.
  • Without the key, any isolated, eligible device can run an attempt.

If no device covers the list, CodeHerder refuses create. The refusal names each device and the reason, such as “lacks required device capabilities: region:eu”. If a device that covered the list goes away later, its attempts stay pending. ch evaluation placement shows what each device lacks in its DETAIL column, as missing:. Update ch before you use the key. An older ch refuses a definition that carries it.

Run it

ch evaluation run --from-file evaluation.yaml --wait

run first previews the definition, as ch evaluation preview does. It prints each change and how many attempts it will run. The count is setups × tasks × repeats. With a sample, the count is an upper bound (“runs up to N attempts”). Above 24 attempts it asks you to confirm. If you answer no, nothing is created. Pass --yes to skip the question. Then run creates the Evaluation. With --wait, ch follows the Evaluation until every attempt ends and prints the report. Press Ctrl-C to stop waiting. The Evaluation keeps running.

How many attempts run at once

By default every attempt starts as soon as a slot is free. An Evaluation takes about as long as its longest attempt. Each attempt is a session with purpose evaluation. Normal placement puts it on an isolated, eligible device in the workspace scope. A device is eligible when it covers the setup’s architecture and runs a device-server at or above the Evaluation’s frozen release. A microVM pool gives each attempt its own VM. An attempt that finds no eligible device stays pending until one is free.

An evaluation session takes one run slot on its device, as a work session does. Work and evaluations share the slots of a device. An evaluation session does not count against the workspace’s work-concurrency limit. Work never stops an evaluation session to make room.

ch evaluation create --from-file evaluation.yaml freezes the Evaluation the same way. It also previews first, and it asks you to confirm above 24 attempts. Pass --yes to skip the question. Both commands accept - to read the definition from standard input.

To see where an attempt can run right now, use ch evaluation placement <evaluation-id>. It lists each device and whether it is eligible. It also shows the spare run slots and the reason a device takes no attempt, such as an old device-server or a full device.

Follow it

ch evaluation list
ch evaluation show <evaluation-id>
ch evaluation attempts <evaluation-id>
  • list shows your workspace’s Evaluations, newest first, with a state, the number of setups, tasks and repeats, and the name.
  • show prints one Evaluation, with a table of its setups and a table of its tasks. Add --json for the full frozen definition.
  • attempts lists every attempt, oldest first, with its state, task, repeat and setup. The SESSION and SESSION-STATE columns name the attempt’s session and its state. Read that session with ch session show <session-id>. For an attempt that ended invalid, it shows the reason. A POOL ENTRIES column names the credential pool entries that served the attempt. A RETRIES column counts the tries an infrastructure loss cost.

An attempt that a device loses, for example when the device drops offline, returns to pending and starts again. The wait doubles each time: 1 minute, then 2, then 4. An attempt that runs far past its time budget without finishing counts as lost and starts again at once. An attempt gets 4 tries. After the fourth, it ends invalid. An attempt that overruns its own time budget ends as a result, with no retry.

A silent attempt

An attempt that shows no progress gets one reminder, as a normal session does. If it stays silent after the reminder, or for budget.stallSeconds, CodeHerder ends it as stalled. A stalled attempt is a scored result, not an invalid one. It counts against the pass rate. Waiting on a timer, another task, a merge request or an AI provider’s rate limit counts as progress.

Follow a running attempt

ch evaluation attempts <evaluation-id> --live
ch session attach <session-id>
ch evaluation artifact <evaluation-id> <attempt-id> evidence.jsonl -o evidence.jsonl
  • attempts --live re-reads every 5 seconds until no attempt is pending or running. It prints the counts and the attempts table again, with each attempt’s session and session state. It prints a new block only when something changed. Press Ctrl-C to stop.
  • session attach shows the live terminal of an attempt’s session. Take the session ID from the SESSION column. An attempt takes no input, so the attach is read-only. It needs no interactive terminal and sends nothing to the session. Press Ctrl-C to detach. It does not work on Windows.
  • artifact downloads one stored file of an attempt: evidence.jsonl, transcripts.tar.gz, device-server.log, merge-request.diff, comments.jsonl (the task’s comments), terminal.tar.gz (a recording of each session’s terminal) or hidden-check.log (the tail of a failed hidden check’s output). The attempts table lists which files each attempt has. Without -o the file keeps its server name in the current directory.

Each attempt runs as one person: the human who created the Evaluation, or the operator of the agent that created it, or else the workspace owner. Only that person can use session attach on the attempt. An agent caller is refused, and so is an admin who is not that person. Open the attempt’s session page in the app instead (see below). artifact works only for a human owner or admin, outside an agent session.

An Evaluation is frozen while attempts run. It becomes completed once every attempt has ended, and its report is then final. A cancelled Evaluation never completes.

To read what happened inside one attempt, run ch evaluation attempt show <evaluation-id> <attempt-id>. It prints the end cause, the final stage, the wall time and the cost. It then prints a table of sessions for each stage and the hand-off comments and git-host events in time order. Add --transcript <session-id> to read one session’s turns, or --device-log to read the device log. Owners and admins can run it.

To stop one early, run ch evaluation cancel <evaluation-id>. Only a frozen Evaluation can be cancelled. Attempts that have not started are cancelled, and devices are told to stop the attempts that are running. Cancelling a completed or already cancelled Evaluation changes nothing.

Read the report

ch evaluation report <evaluation-id>

The report opens with a plain summary: one line for each setup except the baseline, then one line about the number of runs. For example: “opus: better MR quality (+0.13), 3.6x the cost, slower. No real change in pass rate.” CodeHerder writes these lines from the comparison results, without a language model. The app shows the same lines at the top of the Evaluation page.

Under the summary, the report has one row per setup. Each figure is an average over that setup’s valid attempts.

Column What it tells you
RUNS, INVALID How many attempts counted, and how many did not.
PASS RATE The share of attempts that completed and passed your hidden check on the merged branch. The task does not need to reach done.
MR QUALITY A blind MR score’s average for the final merge request, with its spread and the number of attempts scored. It never changes PASS RATE or any answer.
COST Average spend. Cost shows - when any attempt’s cost is unknown.
RUN TIME Average time from start to finish.
TURNS, TOKENS Average agent turns and tokens.
REWORK Average number of rework rounds.
GIT HOST ACTIONS Average number of git-host actions, such as pushes and merge requests.
FAILED CLI CALLS Average number of failed ch and git-host CLI calls.

Below that table, one row per setup and stage shows the runs, cost, turns, tokens and rework of each stage, with the agent and launch config that ran it. show prints the same stages for each setup, with its changes against the baseline.

An invalid attempt is one that cannot be trusted, for example because it reached the CodeHerder server, or because infrastructure failures used up all 4 tries. A case that fails its check before the agent starts is also invalid (see Qualify a case before each attempt). A refused connection to CodeHerder or a real git host does not void an attempt. Invalid attempts stay out of every figure. The report lists each failed attempt, which counts as not passed. It then groups the failures by cause under Why attempts failed, with the count and the first attempt IDs of each cause. It ends with each invalid attempt and its reason.

Under the table, every setup except the baseline gets a comparison. CodeHerder pairs each attempt with the baseline’s attempt on the same task and repeat. It leaves out any pair where either side is invalid, cancelled or unfinished. For each metric you see both averages, the difference and the likely range of the true difference.

Real difference? answers one question: is the gap larger than chance would explain on this sample? Yes means it is. No means it is not, and the sample is big enough to say so. Too few runs means the sample is too small to say. Runs needed is how many runs the report needs to confirm the difference it saw.

Add --stages to print a per-stage comparison under each setup’s comparison. It uses the same difference, range and Real difference? columns.

Each comparison has one line about the number of runs. With few runs, treat even a Yes as a hint. The report adds a warning only when you must act on it, for example when a comparison has fewer than 6 runs.

For machine-readable output, add --json.

From the app

Open Set up → Optimisation → Evaluations. The page lists your Evaluations with their state, number of setups, tasks and repeats, and when they were created. Open one to see the plain summary. Below it, the page shows the ID, manifest hash, seed, repeats, tasks and setups, then an Assessment section (see below).

The sections after that follow the report: Setups, Comparisons, By stage, Why attempts failed, Attempts and Invalid attempts. Why attempts failed groups the attempts that did not pass by end cause, and each group links to its attempts. A retried attempt reads “Retried N times after an infrastructure loss”. Owners and admins see a Cancel evaluation button while the Evaluation is frozen.

Each row in Attempts has an Attempt link, and a Session link when the attempt has a session. The attempt page shows the end cause and its detail, the final stage, the time and the cost. It has a Stages table with one row per session, a Session panel with Transcript, Terminal and Device log tabs, and an Activity timeline of sessions, hand-off comments and git-host events. The session page lets you watch a running attempt read-only.

Create one in the app

Owners and admins see a New evaluation button on the Evaluations page. It opens a form. You can fill it by hand, or open Import definition, paste a definition as YAML or JSON, and select Fill form.

  1. Enter a Name.
  2. Pick the Workflow to test.
  3. Under Tasks, choose A sample of done tasks or Task IDs. Then set the Repeats, the Seed and the Hidden check. A definition with a hand-written task set shows as The imported task set.
  4. Under Providers, tick the stand-in services the attempts use. Add Checks if every setup should run the same commands.
  5. Read the Baseline table, which shows the stages of the workflow as your workspace runs it.
  6. Add setups. For each one, choose Add override. Pick a Scope of One stage, One agent or Workspace context, then pick the node to change and its value. A setup can also use Redesign workflow to build a different set of stages.
  7. Check the Preview. It shows what each setup runs and how many attempts the Evaluation makes. Refresh preview reads it again.
  8. Select Create evaluation.

If CodeHerder refuses the definition, the refusal lists each problem. A refusal for devices lists each device and why it did not qualify.

The app does not run ch evaluation run, so it asks no attempt-count question. The Preview shows the count instead.

An Evaluation can also start from an improvement candidate. It then names the candidate as its source. See The improvement policy.

The agent-written assessment

When the last attempt ends, CodeHerder queues an analyst agent. The agent reads the evidence and writes an assessment. It gives each setup except the baseline a recommendation: adopt, reject or inconclusive. Each recommendation has its reasons. An inconclusive recommendation also names a next test to run. The agent also writes a note on each compared pair, that is, each attempt and its baseline attempt. The agent runs read-only and cannot change your work.

ch evaluation report <evaluation-id>
ch evaluation assessment show <evaluation-id>
ch evaluation assess <evaluation-id>
  • report prints the assessment under the summary, with its date and a note when the report changed after the agent wrote it. It does not print the pair notes. It prints only the number of pair notes. Add --pairs to print the pair notes.
  • assessment show prints the latest run and its state.
  • assess queues a new run. Owners and admins can run it, but only after the Evaluation completes. A repeat call returns the run in progress.

The run needs an agent CodeHerder can place and enough budget headroom. Until then it stays Queued. In the app, the Assessment section shows Queued, Running (with an Open session link), stopped without an assessment, or the recorded text. Owners and admins also get an Assess evaluation button. Treat the assessment as a reading aid: the report holds the numbers.

Check the test with an A/A run

Before you trust a result, test the test. An A/A run runs identical copies of your baseline against each other. The true difference is zero, so any metric with a real difference is noise.

ch evaluation run --from-file evaluation.yaml --calibrate --wait

--calibrate keeps each setup’s name and baseline mark and copies everything else from the baseline. The report marks each noisy metric with “(noise)”. It adds the line “A/A calibration: every setup is identical, so a real difference here is noise.” and a warning that names them. A few noisy metrics across many are normal. A noisy pass rate or cost means your sample is too small, or your environment is too unstable, to compare setups yet. Add tasks or repeats and run it again.

Save a finished task as a case

A done task can become a stored evaluation case. A case is a frozen copy of the task. An Evaluation can run it again.

ch evaluation case promote <task-id> --key fix-cursor --tag area:cursor

You need to be a human owner or admin. An agent credential is refused. The command reads these facts from the task:

  • The title and description that the first work session saw.
  • The commit that session started on, and its repo.
  • The checks of the task’s workflow.
  • The merged merge request. It becomes the reference solution.

The hidden check is the workflow test command. Use --hidden-check to change it. The expected agent time is the measured time of the task’s work sessions. Use --expected-agent-seconds to change it.

The command refuses a task with the error task_not_promotable. The message names the reason:

  • The task is not done.
  • The first work session did not start on one repo, or did not record the original brief.
  • The task has no merged merge request on that repo.

Use the same key again to store the next version of the case. The same content as the newest version stores nothing.

Check that a case is a fair test

CodeHerder validates each new case on an isolated device. The validation runs three legs, with the stage image and no network:

  1. Every required check passes on the starting commit.
  2. The hidden check fails on the starting commit with the hidden patch applied. If it passes, the case is already solved.
  3. The hidden check passes on the reference solution with the hidden patch applied. This leg shows that the case can be solved. A reference that already holds the hidden patch, such as a merged MR head with its tests, counts as patched.

Read the result with ch evaluation case validations <key>. A passed validation with proven set to no is unproven. The case has no reference solution, so leg 3 did not run. An unproven case is a weaker test. You cannot tell a hard case from an impossible one.

A refused validation names the leg that failed:

  • seed/<check>: a required check fails on the starting commit.
  • seed/hidden: the hidden check passes on the starting commit.
  • hiddenPatch: the hidden patch does not apply to the starting commit, or it neither applies to the reference solution nor is already in it.
  • reference: the reference patch does not apply to the starting commit.
  • reference/hidden: the hidden check fails on the reference solution.

Fix the case and store the next version. A fault on the device, such as a failed fetch of the reference commit, never refuses a case. CodeHerder tries again.

Qualify a case before each attempt

A passed validation records one moment. A case can hold something with a date on it, such as a waiver, a test certificate or a token. If it expires, a required check fails on the case itself, and no agent can pass it.

So before the agent starts, the device runs every check on the case’s reference solution, in the stage image, at the real time. The checks are the ones from the definition’s checks and the task’s own checks. A check that fails ends the attempt as invalid, with the end cause case_unqualified. The app shows it as “Case failed qualification”. The attempt names the failing check. The agent never runs, and the report does not count the attempt as an agent failure. Fix the case or the expired item, and run again.

A task with no reference solution cannot qualify, so its attempts run as before. The preview warns: “task has no reference solution: its mandatory checks are not run on a reference solution before an attempt”.

Publish a suite of cases

A suite is a named list of validated cases, such as core-small@3. You publish it once. An Evaluation can then run the same list again. Write a file:

key: core-small
description: Small bug fixes in the core repo.
cases:
  - fix-cursor@2
  - fix-timeout@1

Publish it with ch evaluation suite create --from-file suite.yaml. You need the admin role. A new publish of the same key stores the next version. The earlier versions never change. The same list again stores nothing.

Every case in a suite must be proven. Its newest passing validation, on any stage image, must show that the hidden check passes on the reference solution. If a case is not proven, the publish stops with case_not_validated and ch lists each case with its state:

  • none: the case has no validation. Run ch evaluation case validate <key@N>.
  • pending or running: wait for the validation to end.
  • refused: a leg failed. The line names the leg.
  • abandoned: the run could not finish. Validate the case again.
  • unproven: the case passed but has no reference solution, so you cannot tell a hard case from an impossible one.

All the cases of a suite must have the same type. If not, the publish stops with suite_mixed_types and lists the type of each case.

Read a suite with ch evaluation suite show core-small@3. It lists the cases in order with their tags and expected agent time. Use ch evaluation suite list for the newest version of each suite, and ch evaluation suite versions core-small for its history.

See how a case does over time

CodeHerder keeps the result of each suite case every time an Evaluation runs it. Read one case with ch evaluation case history fix-cursor@2. You get one table for each setup. A row is one Evaluation. It shows the ch build, the server release, the attempts, the passes and the pass rate. It also shows the mean cost, the mean wall time, the mean MR score and the tries.

CodeHerder joins the runs of one setup by its content, not its name. A renamed setup keeps its history. A setup with a changed prompt, stage or launch config starts a new table. A new ch build does not.

A setup is marked REGRESSED when the Evaluation before the latest one had a passing valid attempt. The latest one must also have valid attempts, and none of them passed. A running Evaluation never sets the mark. An Evaluation where every attempt was invalid does not set it either. ch evaluation case list shows the mark and the last pass rate.

Read a whole suite with ch evaluation suite trend core-small@3. It pools the suite’s cases for each setup and ch build, oldest build first, and lists the cases that regressed. Both reads show at most 20 setups, and 50 Evaluations or builds for each.

Same cases over time

The report of a suite Evaluation also answers one question: did a setup get better or worse on the same cases since its last run? It adds a section “Same cases over time” next to the comparisons.

CodeHerder picks the latest earlier Evaluation that ran the same setup content. That Evaluation must be completed. A running or cancelled one is never used. It must have finished at least 6 of the same case versions. If no earlier run qualifies, the setup has no row and no headline line.

Each case version is one pair. Each side is the mean of that case’s repeats. The likely range resamples by case. For the pass rate, a case counts as passed when most of its repeats passed. A tie does not count as passed. The table shows the same metrics as a setup comparison, with the columns “Last run” and “This run”.

The result headline gains one line for each such setup, for example “opus vs its last run (ch 11.4.1008): no real change.” CodeHerder writes it from fixed rules. A model does not write it. Read it with ch evaluation report. ch evaluation show prints the same lines under “Same cases over time”.

Last updated

Try it on one repository for two weeks.

File a task. An agent builds it. A person reads every request. You see the cost of each task.

Try "pricing", "connect a device", or "who reviews the code"

↑↓ move · ↵ open · esc close