CodeHerderSearch⌘KRequest access →

The improvement policy

Set how often CodeHerder samples finished work, how big a sample is, and how Evaluations of a proposed change run.

One policy document per workspace configures the improvement loop. The loop samples recent work, looks for changes worth trying, and tests each one before a person decides. This page describes the document. Retrospectives describes what the loop finds.

How the loop works

  1. Run. On a schedule, the loop samples work that ended in one window.
  2. Analyst. If the analyst is on, it reads the run and may propose one change.
  3. Candidate. Each proposal becomes a candidate that tracks one change.
  4. Evaluation. A person approves the candidate, or auto mode does it. An Evaluation compares the current setup with the changed one.
  5. Verdict. The Evaluation report sets the verdict: proved, disproved or inconclusive.
  6. Decision. A human admin accepts or declines the candidate.
  7. Apply. An accepted change becomes a new version of its target. An audit row records it.
  8. Roll back. A human admin can write the old version back as a new version.

Nothing changes a target before step 6.

Open the loop in the app at Set up → Optimisation → Improvement. The page has a Configuration section, a list of runs and a list of candidates. Open a run or a candidate for its detail page. The page has the same actions as the CLI. Each action below names its button. A run is the same record as a retrospective.

Read the document, or change it, from the CLI:

ch improvement policy
ch improvement configure --interval-hours 12 --cohort-size 30

configure changes only the fields you name. A workspace owner or admin who is a human can change it. The Configuration section of the Improvement page sets the same fields. An agent cannot. A group workspace has no policy: set it in each workspace that needs one. Pass a workspace name, slug, slug path or full ID as the trailing argument to reach another workspace.

Runs

A scheduled run reads one window of work. The window starts where the last run ended and stops at the last full hour. The first run reads the last Interval hours of work. A window is never longer than 7 days. After a longer wait, the run reads only the last 7 days.

Two scheduled runs never overlap. The second window starts where the first one ended.

The run picks the sample at random from the attempts that ended in the window. The sample holds up to Cohort size attempts. The same workspace and window always give the same sample. Another window gives another sample. The run records the seed, the number of attempts in the window, the number it sampled, and the window. A sample can miss a rare problem.

The run also keeps the survey responses that agents submitted in the window. It keeps them even when their attempt is not in the sample. It keeps at most 200 responses and 128 KiB of free text. A gap names anything it left out. If the workspace has no enabled survey, the response rate reads as unknown, not zero.

An admin can also capture by hand. Run ch quality retrospectives capture [<workspace>]. The app has no button for it. The capture reads the next scheduled window, at the Cohort size. It does not move Last window end or Next run, so the next scheduled run still happens. A human workspace owner or admin can run it.

Settings

Every field has a default. A workspace with no saved policy reads the defaults.

Capture. Whether the loop samples work on a schedule. Default off. Tick Capture work in the Configuration section, or run ch improvement configure --capture on|off.

Analyst. Whether an analyst reads each sample. Default off. Tick Analyse each sample, or run ch improvement configure --analyzer on|off. The analyst starts paid model work.

Interval (--interval-hours). Hours between capture runs. The first run reads this many hours of work. Default 24. Allowed range 6 to 168. If you change it, the next run moves to the last run plus the new interval. If that time has passed, the run starts on the next pass.

Cohort size (--cohort-size). How many attempts one sample holds. Default 50. Allowed range 10 to 100.

Evaluation mode (--evaluation-mode). approval means a person starts each Evaluation. auto means the loop starts it. Default approval.

Evaluation spend cap (--spend-cap-usd). The most the loop may spend on Evaluations, in US dollars, in whole cents. Default 0. Allowed range 0 to 1000. A cap of 0 starts nothing.

Mechanisms (--mechanism, repeatable). The kinds of change the loop may propose and test. Default stage_prompt. The list must have at least one entry, with no repeats. A new list replaces the old one. The loop records only a proposal whose mechanism is in the list. The mechanisms are:

  • stage_prompt: replace one stage prompt.
  • skill: replace the files of one skill. A candidate cannot reach Accept today.
  • wiki_page: replace the body of one wiki page. A candidate cannot reach Accept today.
  • model: change the model of one launch config.
  • harness: change the harness of one launch config.
  • effort: change the effort level of one launch config.
  • launch_config: replace one launch config with another. No Evaluation can test it, so a policy cannot enable it. A candidate cannot reach Accept today.
  • guardrail: replace the files of one guardrail.

An entry must be a mechanism that an Evaluation can test. If it is not, the change is refused with mechanism_not_evaluable. The policy accepts the other seven mechanisms. A candidate of mechanism skill or wiki_page cannot start an Evaluation yet, so it cannot be accepted. See “Why a candidate waits”.

Evaluation template (--from-file). The standing part of every Evaluation the loop starts. It takes providers (required), workflow, checks, tasks.taskSet or tasks.suite, hiddenCheck and budget. tasks.suite names a suite version with @N, for example suite: core-small@3, and can add tags and sample. Write it as YAML or JSON. For example:

providers: [gitlab]
hiddenCheck: {command: make test, timeoutSeconds: 900}
budget:
  attemptWallSeconds: 3600

If the template has no tasks.taskSet and no tasks.suite, the loop takes the tasks from the candidate’s evidence. If it has no workflow, the loop uses the workflow of those tasks, or the type of the suite. A workflow that differs from the suite type is refused. The loop fills in the setups, the name, the repeats and the seed, so the template refuses setups, grid, name, seed, repeats, tasks.sample and tasks.ids. The size limit is 64 KiB. Default none. Remove it with --clear-evaluation-template. A candidate cannot start an Evaluation until the policy has a template.

Last window end (read-only). The end of the last sampled window. The loop writes it after each scheduled run. It is empty until the first run.

Next run (read-only). When the next capture is due. It is empty while capture is off.

Updated (read-only). When the policy last changed.

The workspace activity records each change to a field, as a policy-change entry that lists the changed fields.

Candidates

Each proposal the analyst records becomes a candidate. A candidate has a state: awaiting_approval, evaluating, rejected, proved, disproved, inconclusive, failed, accepted, declined or rolled_back. A candidate that is not yet decided is open.

State Meaning Leaves by Who
awaiting_approval The candidate waits for an Evaluation. Approve to evaluating. Reject to rejected. A human admin. In auto mode, a background pass approves.
evaluating The Evaluation runs. The verdict pass sets proved, disproved or inconclusive. A cancelled Evaluation or the spend cap sets failed. The server
proved, disproved, inconclusive The report set the verdict. The candidate waits for a decision. Accept to accepted. Decline to declined. A human admin
accepted The change is applied. Roll back to rolled_back. A human admin
rejected, failed, declined, rolled_back Closed. No transition leaves them. None None

A closed candidate does not hold its mechanism, target and base version. At most one open candidate holds one mechanism, target and base version.

If a later run proposes the same change, CodeHerder adds its evidence to the open candidate. It does not make a second candidate. If the earlier candidate is closed, the new proposal makes a new candidate.

The server checks each proposal against the live target before it stores it. The target must exist and the base version must be the current version. A stale or missing target is refused, and the refusal names the base version or the target. A proposal names a metric of the Evaluation report and the direction it should move. It also names a quality guardrail metric that must not get worse. The default guardrail is pass.

The candidate page has these sections:

  • State badge. The state, with the reason beside it when there is one.
  • Proposed change. What the change writes, set against the version it replaces.
  • Verdict. The report rows that decided the state: the success measure, the quality guardrail and whether the difference is real.
  • Evidence. The attempts each run cited for and against the change. A gap reads as a sentence that names what the run left out.
  • Estimate. What the Evaluation is expected to cost.
  • State history. Every change of state, with who made it and why.
  • Applications. Each accept and roll back that changed the target.

Read runs and candidates from the CLI:

ch improvement runs
ch improvement candidates --state awaiting_approval
ch improvement candidates show <candidateId>

Survey evidence

A run can carry survey evidence in two legs. The attempt leg counts the survey invitations of the sampled attempts. The window leg counts the survey responses that agents submitted in the window. The counts are issued, completed, expired, voided, pending and decided, and the response rate.

The run detail page shows both legs in its survey coverage. A candidate shows the evidence of the runs that support it. The analyst reads the same evidence.

A survey must be enabled for any of this to exist. A human admin runs ch survey enable <key>. While no survey is enabled, the response rate reads as unknown. An unknown rate is not zero. See Surveys.

Budgets

  • Analyst starts. A fixed daily limit applies to the whole account. Workspaces in the account take turns. One analyst runs at a time, and a run gets two tries. Newer runs go first. A run that has waited 24 hours goes ahead of them. ch improvement policy prints the live limits. A run that waits for this limit reads daily_budget_exhausted.
  • Analyst spend. The optimization budget is a separate dollar cap on analyst sessions. Both limits apply. Either one can hold a run.
  • Evaluations. Only the Evaluation spend cap limits them, as “Spend cap” describes. The optimization budget does not cover an improvement Evaluation. A cap of 0 starts nothing.

Evaluations

An Evaluation tests a candidate. It compares the current setup with the proposed setup on the same tasks. Its report sets the verdict.

Approval. In approval mode a human workspace admin starts each Evaluation. Run ch improvement candidates approve <candidateId>, or press Approve on the candidate page. To close a candidate, run ch improvement candidates reject <candidateId> --reason <text>, or press Reject. The reason is required. Agents and contained sessions cannot approve or reject.

Auto mode. In auto mode a background pass starts the Evaluation of each waiting candidate. The Evaluation runs as the admin who set auto. To change who it runs as, switch to approval and back to auto. The pass checks about once a minute. It retries a candidate only for spend_cap, no_isolated_device and no_evaluation_template. A candidate that waits for any other reason waits for a person.

What the Evaluation holds. The Evaluation runs one workflow: the template’s workflow, or else the workflow of the done tasks that back the evidence. The baseline setup has no change. The proposed setup adds the one change of the candidate. A stage prompt change applies to the stage the supporting attempts ran. A model, harness or effort change applies to the agent that owns the target launch config. It reaches every launch config of that agent. Providers, checks, the hidden check and the budget come from the template. The tasks come from the template’s tasks.taskSet or tasks.suite. The cost estimate counts the expanded cases of a suite. If it has neither, they are the done tasks of the supporting and contrary attempts, at most 8. The repeats give at least twice the pairs the report needs.

Spend cap. Each policy window is one cycle. The estimate is the setups, times the tasks, times the repeats, times the median lifetime cost of the supporting tasks, so one attempt is priced as a whole run of the workflow. Tasks of the frozen workflow price it when they have spend. A start is refused when the estimate is more than the cap minus what the cycle’s other Evaluations already spent or were estimated to spend. If the real spend of a cycle goes over the cap, the pass cancels its running Evaluations. Their candidates become failed with the reason spend_cap.

Why a candidate waits. A candidate that cannot start stays in awaiting_approval. Its state reason names the cause:

The app shows this reason code beside the state badge.

  • no_evaluation_template: the policy has no template, or the template no longer parses. Auto mode retries it.
  • no_tasks: no done task backs the evidence, or none runs the workflow. A person must act.
  • mechanism_not_buildable: the change cannot go into a setup. This is true today for skill and wiki_page, because a setup names those by slug and cannot carry new content. A person must act.
  • no_cost_estimate: no supporting attempt has recorded spend. A person must act.
  • spend_cap: the estimate does not fit the cycle’s cap. Auto mode retries it.
  • no_isolated_device: no isolated device can run the Evaluation. Auto mode retries it.
  • definition_refused: the Evaluation definition did not freeze. A person must act.

Verdict. When the Evaluation completes, the pass reads its report. The candidate becomes proved, disproved or inconclusive. A cancelled Evaluation makes it failed with the reason evaluation_cancelled.

  • inconclusive: the report is an A/A calibration, it has fewer complete pairs than it trusts, the proposal has a legacy metric, or nothing is significant.
  • proved: the proposal’s metric moved significantly in the direction it named, and the quality guardrail did not get significantly worse.
  • disproved: the metric moved significantly the other way, or the guardrail got significantly worse.

The candidate keeps the hash of the report and the rows of the metric and the guardrail that decided it. The Verdict section shows them, and ch improvement candidates show prints them.

Accept, decline and roll back

A person decides what happens to a candidate with a verdict. Only a human workspace admin can. Agents and contained sessions cannot.

Accept. Run ch improvement candidates accept <candidateId>, or press Accept. The change goes to the target as a new version. The audit row and the accepted state commit in the same step. A proved candidate needs no flag. A disproved or inconclusive candidate needs --override and a --reason. In the app, press Accept with override and write the reason.

What each target gets. A skill, wiki_page or launch_config candidate cannot reach Accept today.

  • stage_prompt: a new stage version with the new prompt. A built-in stage gets a local copy first.
  • model, effort, harness: the launch config updates, and its version goes up.
  • guardrail: a new guardrail version.

A moved target. The target must still hold the version the candidate pinned. If it changed, the server answers improvement_target_moved with the action re-evaluate. Nothing is written. Decline the candidate. A later run can propose the change again. If an ancestor workspace owns the target, the server answers improvement_target_not_owned. Edit it in the workspace that owns it.

Decline. Run ch improvement candidates decline <candidateId> --reason <text>, or press Decline. The reason is required.

Roll back. Run ch improvement candidates rollback <candidateId>, or press Roll back. The pre-image goes back as a new version. The history is never rewritten. If the target changed after the accept, the server answers improvement_target_moved with the action force. Run it again with --force --reason <text>, or tick the force box in the app, to write the pre-image over the later change.

Audit. ch improvement candidates show lists each accept and rollback: the versions, the person and the reason. The pre-image stays stored. Neither the app nor the CLI shows it. The Applications section shows the same record.

Safety

  • The loop never applies a change by itself. Only a human admin accepts.
  • The analyst credential is read-only. The analyst records a proposal. It cannot start an Evaluation or apply a change.
  • The verdict comes from the Evaluation report only. The analyst never sets it.
  • A roll back writes a new version. It never rewrites history.
  • An automated check keeps every change behind a human accept.

An Evaluation of a stage prompt runs on tasks that are already done. It does not run on future work. A proved verdict is evidence, not a promise. Read the report before you accept.

Last updated

Try it on one repository for two weeks.

File a task. An agent builds it. A person reads every request. You see the cost of each task.

Try "pricing", "connect a device", or "who reviews the code"

↑↓ move · ↵ open · esc close