# Agent Experience surveys Source: https://codeherder.com/docs/surveys/ Ask your agents a short question set while they work, gate their next move on an answer, and read the results in the web app or with ch survey. CodeHerder can ask your own agents a short set of questions while they work — an **Agent Experience (AX) survey**. You write the questions once. CodeHerder samples sessions, asks the survey, and collects the answers. Build and read a survey from the **Surveys** page in the web app, or from the command line with `ch survey` — both reach the same surveys. ## In the web app Find the **Surveys** page under **Set up** in the sidebar. It lists every survey in your workspace: its title, key, audience, sample rate, channel, question count, and when it was last updated, with a badge marking any survey that’s disabled. ### Creating a survey Click **New survey** to open the create form. Give it a key and a title, pick a channel, an audience, and a sample rate, and add your questions — each with a prompt and a kind: **Single choice**, **Multi choice**, or **Free text**. Enter choices one per line. A multi-choice question can bound how many options someone must pick with a min and a max; a free-text question can cap how many characters it accepts. Add, remove, and reorder questions with the row controls. As you type an audience selector, the form shows a live preview after a short pause — matching nobody, or, once you’ve set a sample rate, roughly how many sessions it would actually invite. You don’t have to start from a blank sheet. Choose **From a template** and pick one from the library — CodeHerder’s own built-in `ax_v1` (“Agent experience”), or any template your workspace has written (see [Your own survey templates](https://codeherder.com/docs/surveys/#your-own-survey-templates) below) — to preview its questions before you commit to it. Picking a template fills in the key and title and carries its questions over as they are; the question editor is hidden while you’re creating from a template, so you can’t tweak them at this step. You can still edit the questions afterwards, from the survey’s own page. ### A survey’s own page Open a survey to see its overview: audience, sample rate, channel, and its full question spec, plus who created it and when. From here you can: - **Edit** the survey — change its title, audience, sample rate, channel, or its questions, the same editor the create form uses. This works whether or not the survey started from a template. - **Enable or disable** it with the button next to the title. This is limited to workspace owners and admins, and only a person can do it — never an agent session. - Read its **Results** — the response rate, per-question answer distributions, and a breakdown by stage, agent, or model. Pick any question from the list to see its answers, choice or free text alike. - Browse its **Responses** — every invitation issued, with its state, stage, and timestamps. Expand a row to read that response’s own answers in full. ### Your own survey templates Besides the built-in `ax_v1`, your workspace can keep its own library of survey templates — reusable question sets you can start a new survey from. Any member can browse them: open **Survey templates** from the Surveys page. Workspace owners and admins can write templates (an agent session can’t): click **New template** and supply a key, title, and question set. A workspace template’s own page has **Edit**, which saves a new version without disturbing surveys already built from an earlier one, and a **Version history** to see what changed. A built-in template has neither — it’s read-only and never copied over. Writing a template under a key that collides with a built-in is refused; writing one with the same content as the current version leaves the version unchanged, and only genuinely different questions add a new one. ### A response’s own page Every response also has its own page, linked from the Responses table. It shows the response’s full set of answers next to who answered, from which task and session, and at which stage. If a response surfaces something worth acting on, click **Convert to task** to open an editable draft — title, description, type, and priority, pre-filled from the response — and submit it to file the task. This only works once per response; converting the same one twice is refused, and names the task that already exists. ## Writing a survey from the command line ### Start from a template List the templates your workspace can build from — the built-in `ax_v1` plus any your workspace has written: ``` ch survey template list ch survey template show ax_v1 ``` Then create your survey straight from one: ``` ch survey create --template ax_v1 ``` `ax_v1` — “Agent experience” — asks six questions: three ratings, on the same four-level scale from “no context” through “thin” and “adequate” to “strong,” covering whether the task gave the agent enough context to start, whether the feedback it got kept it moving, and whether the session was scoped sanely. Each rating is followed by a required free-text answer capped at 600 characters: the first two ask why the agent rated it that way, and the third asks which scoping problem applied. Creating from a template still lets you override its defaults: pass `--key`, `--title`, `--audience`, or `--sample-percent` alongside `--template` to replace the built-in’s own. Manage your workspace’s own templates from the command line too: ``` ch survey template create --from-file template.json ch survey template versions my_template ch survey template versions my_template --version 2 ``` `ch survey template create` takes the whole template document — key, title, and questions — from a file, or pass `-` to read it from stdin. `ch survey template versions` lists every version a template has been through, or shows one specific version. ### Write your own Create one with `ch survey create`: ``` ch survey create \ --key ax_v1 \ --title "Agent experience" \ --audience '*' \ --sample-percent 25 \ --from-file survey.json ``` `--from-file` points at a JSON document with the survey’s questions — or pass `-` to read it from stdin. Never pass the document as an inline argument; a file or stdin keeps quoting and shell expansion out of your questions. Each question has a `key`, a `kind`, and a `prompt`, plus a `required` flag. There are three question kinds: - **`single_choice`** — pick one option from a list of `choices`. A rating scale is just a `single_choice` question with ordered choices like “Poor” through “Excellent” — there’s no separate rating type. - **`multi_choice`** — pick one or more options from a list of `choices`, optionally bounded by `minSelected` and `maxSelected`. - **`free_text`** — open-ended prose, capped at 10,000 characters by default (set a lower `maxChars`, from 1 up to that same 10,000, per question if you want shorter answers). ### Choosing an audience `--audience` decides which sessions are eligible. Use `'*'` for everyone, or a comma-separated list of `facet:value` terms drawn from five facets — `agent`, `model`, `stage`, `harness`, and `task_type`. A `model:` term names a model family — `haiku`, `sonnet`, or `opus` — not one specific model version, so `model:sonnet` matches any Sonnet-tier session regardless of which exact model it ran: ``` ch survey create --key code_stage_check --title "Code stage check-in" \ --audience 'stage:code,harness:claude' --sample-percent 50 --from-file survey.json ``` **Terms are ORed, never ANDed.** A session matches when *any* listed term matches — the example above reaches a session at the `code` stage OR one running the `claude` harness, not only sessions that are both. Intersection (AND) targeting is not supported today. No selector asks for “code stage AND claude harness.” Do not work around this by creating two surveys, one for `stage:code` and one for `harness:claude`. These are two separate, overlapping cohorts, not an intersection. The `stage:code` survey also reaches sessions on other harnesses. The `harness:claude` survey also reaches sessions at other stages. Neither excludes sessions outside the code-stage-and-Claude intersection you actually want, so results from the pair do not answer a question about code-stage Claude sessions. Use `ch survey audience` (below) to see each selector’s real reach before you rely on it. ### Check who it reaches Creating or editing a survey checks that your audience selector is well-formed, but it doesn’t check that it actually matches anyone. Before you rely on a selector, ask CodeHerder how many sessions it would have reached: ``` ch survey audience 'stage:code,harness:claude' ``` This looks back over a window (90 days by default) and reports how many sessions matched, out of how many ran in that window. Read the two counts together — they mean different things depending on which is zero: - **Zero matches, but some sessions ran** means the selector itself is inert: it’s valid, but nothing in that window would ever have been asked. - **Zero matches out of zero sessions** means the window was empty — it says nothing about whether the selector is any good. Look further back with `--since 180d` and check again. ### Sampling `--sample-percent` sets what share of matching, eligible sessions actually get asked — from 0 to 100. A value outside that range is refused before anything is sent. A session that matches your audience isn’t guaranteed a survey; it’s rolled against your sample rate. A session is offered at most one survey invitation at a time, and never a survey it already answered, voided, or let expire. A pending invitation expires automatically once its session ends, so it never sits open waiting on a session that’s no longer there. ## The answer gate When a session is selected, it’s told to answer before it can move on: ``` You've been randomly selected to fill out a quick Agent Experience survey. Read it: ch survey show Answer it: ch survey answer --from-file Then re-run your advance command. ``` A selected agent can’t advance its task to the next stage, and can’t release its sandbox, until it answers. This is the one cost to the agent: one extra command in the same session, not a separate step for you. Abandoning a task — cancelling, blocking, or archiving it — is never gated on a survey. A human operator is never surveyed. ## Reading results from the command line A survey’s Results and Responses sections in the web app cover this same ground. From the command line: ``` ch survey list # every survey in this workspace ch survey list --enabled # only surveys currently collecting answers ch survey show ax_v1 # one survey's questions ch survey show # one invitation and its answers, once answered ch survey responses --survey ax_v1 # completed answers, one row per invitation ch survey report ax_v1 # per-question distributions and the response rate ch survey report ax_v1 --by stage # the same distributions, broken down by stage ``` `ch survey list` shows every survey by default; add `--enabled` or `--disabled` to see only the ones currently collecting answers, or only the ones turned off. `ch survey show` tells a survey from an invitation by the shape of the argument: a survey key is never a full UUID, so a UUID-shaped argument reads an invitation, and anything else reads a survey by key. `ch survey responses` pages completed responses by default. Pass `--state pending`, `--state voided`, or `--state expired` to see other states, and narrow further with `--stage`, `--agent`, or a time window. Free-text answers print in full below the table. Like `ch survey report`, it looks back over the last 90 days unless you pass `--window` or `--since`. `ch survey report` shows, per question, how many sessions picked each choice — every declared choice appears, in declared order, even the ones nobody picked. A free-text question shows only how many were answered versus skipped; read the actual text with `ch survey responses`. It also bounds itself to the last 90 days by default; pass `--window` to narrow that, or `--since` to set your own lower bound. Add `--by stage`, `--by agent`, or `--by model` to see the same distributions broken down along one axis — or leave `--by` off to get all three at once. The report also gives you five counts that always add up to the number issued: **issued**, **completed**, **expired**, **voided**, and **pending**. The response rate is completed divided by (completed + expired), shown as a percentage — or as `-` when nothing has been decided yet, so an idle survey never misreads as a 0% failure. If you edit a survey’s questions after it’s already collecting answers, invitations issued before the edit are still validated against the version of the questions they were sent with. The report reflects this too: responses collected under different versions of the questions render as separate blocks, one block per version, each with its own set of choice counts — so a change to your questions never blends old and new answers together under one set of numbers. ### Where responses are read Every survey has a `channel`, set with `--channel` when you create or edit it. It decides which workspace reads the survey’s completed responses, and which workspace a converted response files its task into. Leave it alone and a new survey’s channel is `workspace`, which keeps both in the workspace that owns the survey. Channel has nothing to do with who gets asked. Your audience selector alone decides that, and a session is only ever offered surveys belonging to its own workspace. ## Changing or retiring a survey Edit a survey from its own page in the web app, or from the command line: `ch survey edit ` updates a survey — its title, audience, sample rate, channel, or questions — without recreating it. Every flag is optional; pass just the ones you want to change. The survey’s key can’t be changed. There’s no delete. Enable and disable from the survey’s own page, or from the command line: ``` ch survey disable ax_v1 ch survey enable ax_v1 ``` **Disabling or editing a survey does not clear an invitation that’s already pending.** A session that was already asked still owes an answer even after you disable the survey. If an agent is stuck on an invitation you no longer want answered, clear that one invitation directly: ``` ch survey void ``` Voiding only works on a pending invitation — one already answered can’t be voided. There’s no screen for this yet; it’s a command-line-only action. ## Turning an answer into a task Convert a response to a task from its own page in the web app, or from the command line, in two steps. First, preview it — this doesn’t create anything: ``` ch survey convert-draft ``` The draft shows a composed description, type, and priority built from the response, but its title is often blank, so you supply one when you actually convert: ``` ch survey convert --title "Stop double-charging on retry" --type feature ``` `--title` is required. `--description` is optional and falls back to the composed body if you leave it off; `--priority` defaults to normal, and `--type` defaults to a plain task if you don’t set it. This only works on a completed invitation, and only once — converting the same response twice is refused, naming the task that already exists. ## Who can do what - **Reading** — any workspace member can list surveys, view questions, read responses, and pull the report, in the web app or from the command line. - **Authoring** — creating, editing, enabling, disabling, or voiding a survey, and writing or editing a workspace survey template, is limited to workspace owners and admins, and only a person can do it — not an agent session. An agent can be asked a survey question, but it can’t write, change, or clear one. - **Converting** — any workspace member can convert a completed response into a task, but again only a person, not an agent session. An agent can’t file a task out of its own survey answer. ## Related guides - [Retrospectives](https://codeherder.com/docs/retrospectives/) — a separate, freeform look back at recent stage attempts - [Using the ch CLI](https://codeherder.com/docs/using-the-cli/) — the shared conventions behind every `ch` command, including passing text bodies through a file