Reading the Observability report
The Observability report in the app and the CLI: summary, rates, trend, time split, breakdowns, slowest calls and drill-downs.
Understanding costs answers “what did this cost?” Observability answers a different question: “where did the time go, and how much of it was actual tool work?” CodeHerder tracks how long every tool call an agent makes actually takes, and rolls that up into a workspace-wide report. It has summary numbers, rates, a trend, attribution coverage, a time split by session, six breakdowns, and the individual slowest calls. You can also drill down from any breakdown row to the sessions behind it, and from a session to its full timeline.
Where to find it
Observability is a report under Work → Reports for a workspace or for a group. A group’s report covers every workspace nested beneath it, the same way a group rolls up any other report in CodeHerder.
Open it from the Observability card under “Time and flow”. The other cards in that group are Task flow, Alert rules and Tool time contributors. The main report also has an On this page list that jumps to each section. On a narrow screen it becomes a jump menu.
If a window has no tool calls, only the summary shows, with the note “No tool time recorded in this window.” Use the page’s clear button to go back to Last 30 days. It reads Show last 30 days when no filter is set.
On a phone-width screen, each table shows five rows. Use Show more to see the rest, and Show less to fold them again.
The summary
The summary shows ten numbers for the window:
- Tool calls: how many calls started and finished.
- Tool time: the total time spent inside those tool calls.
- Average per call: tool time divided by tool calls.
- Tool-call sessions: how many sessions made at least one qualifying tool call in the window. This count is narrower than All sessions in the time-split tables further down the page. All sessions counts every session whose lifetime overlaps the window, including one with no tool calls. The two numbers measure different groups on purpose. If they differ, that is not a bug.
- Tasks: how many tasks those sessions belong to.
- Open calls: calls that started but never reported a result. These are excluded from tool calls and tool time; see How to read these numbers below.
- Tokens and Cost: the recorded token and dollar totals for the window.
- Slowest call: the single longest call in the window.
- Errors: how many calls in the window ended in an error.
Filters and window
A Window control at the top narrows or widens the whole report at once. It offers four choices: Today, Yesterday, Last 7 days, and Last 30 days. These match the four --window keywords of ch costs (see Changing the time window). Unlike the Costs page, this control has no calendar or custom options. The default is 7 days.
Six filters sit alongside it, and you can combine them: Stage, Agent, Model, Category, Tool, and Label. Narrow to one stage to see where a pipeline step spends its time. Narrow to one tool to see every call it made across sessions. The filters and the six breakdowns below answer the same questions from opposite directions. When you change a filter or the window, the panel below dims and shows a spinner while the new figures load. This tells you whether the screen matches your current selection.
Rates
A compact panel converts the summary into per-unit figures: Tool calls per session, Tool calls per task, Tool time per session, Tokens per tool call, Cost per session, and Cost per task. A rate is blank, not zero, when its denominator is zero. Blank means “no measurement”. It does not mean “measured, and it’s zero”.
Trend
Five charts plot the window’s tool activity bucket by bucket: Tool calls per bucket, Duration percentiles per bucket (p50 and p95, each computed from that bucket’s own calls, never averaged across buckets), Failure rate per bucket, Cost per bucket, and Tokens per bucket. A missing bucket shows as a gap, not a zero. A bucket you never measured looks different from one you measured at zero.
Buckets are not always a day long. A Resolution control offers Auto (best fit), Hour, and Day. Auto buckets by hour for a window of 25 hours or less and by day for anything longer, so a short window still shows useful shape. Hour resolution over a long window can produce too many buckets. The report refuses that, so narrow the window or switch to Day.
Two more controls sit alongside Resolution:
- Compare: set it to “Previous period” and the tool calls, cost, and tokens charts each add a second, faint line. It covers the stretch of equal length just before your window, so you can see whether a metric is trending up or down.
- Series: pick Stage, Agent, Model, Category, Tool, or Label and an extra chart appears. It breaks tool calls down by that facet’s busiest few values. A line labelled “Other (N)” folds in everything else.
Drag across the Tool calls per bucket chart to narrow the window to that range. You can also type a range in the From and To fields, which is the keyboard alternative. A Clear range button appears once you set a range. A chosen range overrides the Window control above. Click a single point on the first chart to open its bucket in Selected bucket below. Export CSV downloads the points of the main trend. It does not include the previous-period line or the Series chart.
A point can carry two marks. Incomplete means the report window covers only part of the bucket. That happens at either end: the first bucket when your window starts partway through it, and the last bucket when your window, or the current time, ends partway through it. Low sample means the bucket has some activity, just not much of it — read a rate (like a failure percentage) from a low-sample bucket with that in mind.
Below the charts, a Recorded changes table lists stage-prompt, agent-setup, and release changes that landed during the window. Treat them as a correlation to review. They are never a claimed cause of anything the trend shows. A long list can show as truncated. Narrow the window to see every change.
Selected bucket
Pick a point on the trend and this section appears, listing the slowest tool calls inside that one bucket. Each row opens the session that ran the call. A Clear selection button closes the section and goes back to the whole window.
Attribution coverage
This section answers “what does this window’s data actually cover?” instead of “where did the time go”. It is a companion to the numbers above it, not a seventh breakdown. Three percentages, Tool name, Label, and Model, show what share of this window’s tool calls carry each piece of information. A fourth tile, Coverage rows, counts the rows behind them. A note explains the fallback names you may see in the tables, such as “no stage”, “unrouted”, “unrecorded”, and “unlabelled”. A call can lack a piece for a few reasons. It may have run against no workflow stage, its session may have no routed model recorded, or it may predate the details CodeHerder now tracks. Older records fill in less of this picture than newer ones. That is expected, not a data-quality problem to chase down.
Time split
Three tables — by stage, by agent, by model — split each session’s own wall-clock time into tool work and everything else. Each row shows a proportion bar plus:
- Session time: the session’s own wall clock, clamped to the window (a session still running when the window ends counts only up to the window’s edge).
- Tool time: the union of the session’s tool-call intervals. Two overlapping calls count that overlap once here, unlike the summary’s tool time, which sums every call’s own duration regardless of overlap.
- Other time: session time minus tool time: everything the report doesn’t directly measure, model turnaround and overhead included.
- Parallelism: summed tool-call time over tool-busy wall clock. A value of 2.00x means two tool calls ran at once on average while tools were busy; blank means the row had no tool-busy time to divide by.
- All sessions: every session whose span overlaps the window, including one that made no tool calls at all. This is the larger group that pairs with Tool-call sessions, described under The summary above.
The six breakdowns
Below time split, six tables split tool time along different axes:
- By stage — which workflow stage the time went to (
plan,code,review, and so on), ordered by where each stage falls in the pipeline. - By agent — which agent’s sessions spent the time. A session with no agent behind it (one you started yourself) rolls up under a shared “human session” row.
- By model — which model family was running (for example Opus, Sonnet, or Haiku), or “unrouted” when none was recorded.
- By tool — which tool ran (a shell command, a file read, an editor call, and so on).
- By label — the shape of the command behind the call, with its argument values stripped:
go test | tail, orsleep ; glab ci | tail. A longer chain keeps its pipes and separators, so a label can run to a line of its own. - By category — what kind of work the call was.
shellusually dominates by a wide margin, since it covers every command an agent runs that isn’t recognised as something narrower. The rest split outvalidation(build, test, lint), reads, edits, waits, and a few narrower bookkeeping kinds. This is the same category vocabulary Sessions from the command line uses for one session’s own observations.
Each row carries Tool time, its % of tool time, Average per call, Calls, Sessions, Max (the longest single call in that row), and Errors. By stage, by agent, and by model add exact Tokens and Cost columns. By category, by tool, and by label add Estimated tokens and Estimated cost instead. A category, tool, or label is not billed itself, so the estimate is CodeHerder’s best attribution of cost to that key. A table longer than its cap folds the rest into one “N other …” row at the bottom. That row carries the true totals for everything it folded in, not an average, and its View all rows link opens the complete list.
Not every call arrives with a tool name or a label — an older record, or one CodeHerder can’t summarise, keeps neither. Those calls roll up together into one row, shown as unrecorded in the By tool table and unlabelled in the By label table. The slowest-calls list also shows unrecorded for a missing tool. A missing label shows there as a dash.
By tool, by label, and by category also carry a No tool call row: tokens and cost recorded on turns where the agent didn’t make a tool call at all. It has no tool time, call count, or session count of its own — only tokens and an estimated cost — because there’s no tool call to attribute those figures to.
The slowest calls
Below the breakdowns, a Slowest tool calls table lists the individual calls that took longest in the window as a fixed top 50, longest first. Only calls with both a start and a result are eligible, so an open call never appears here, however long it has run. Each row names the tool, its label, the stage and agent it ran under, how long it took, when it started, and its State. Click the Tool, Duration, or Started header to re-sort. Click a row to open the session that ran the call.
Drilling down: rows, contributors, and traces
Each step keeps your window and filters, so you stay in the context you started from.
- Rows. Under a capped breakdown table, View all rows opens a “Tool time rows by” page for that breakdown. The By label table has the same link for labels. It lists every key in the window, not just the top rows. A search box reaches every row, the columns sort, and Load more pages through long lists. Each row has a View sessions link.
- Contributors. Tool time contributors lists the sessions behind your filters, highest tool time first. The columns are Session, Stage, Agent, Tool time, Calls, Cost, and Started. The time range is fixed when you arrive and matches your window. This page has no Window control. Click a row to open its trace.
- Execution trace. The trace shows a Waterfall of the session’s tool-call spans on one time axis. You can zoom, and a details panel describes the selected span. Below it, a Trace events table lists tool calls, cost turns, and lifecycle events in time order. The waterfall draws at most the first 500 spans. The table is the accessible equivalent.
The command line has the same three steps, described next.
The same report from the command line
ch observability gives you the same report as a family of commands. Each one below takes an optional workspace or group as its first argument, defaulting to your current workspace. Add --json to any of them for machine-readable output.
Most commands take --window with one of today, yesterday, week, or month. The default is week. For a custom span, use --since with a duration such as 30d, 12h, or 90m, or with a timestamp. You cannot combine --window and --since. sparkline, markers, contributors, and trace also take --until.
Apart from trace and markers, every command takes a flag for each of the app’s six filters: --stage, --agent, --model, --category, --tool-name (alias --tool), and --label. They also take --status, a filter the app doesn’t have: it narrows to calls with one result status, success, error, or unknown.
ch observability summary
Prints the same summary figures plus the rates panel, for the last 7 days by default.
ch observability breakdown --by stage,agent
Prints one table per facet you name. --by (or --group-by) takes any of stage, agent, model, category, toolName, label, comma-separated. Leave it off and every facet prints. Ask for the tool facet as toolName; the table it prints is headed by tool.
ch observability time-split --by stage,agent
Prints the time-split tables. --by here only accepts stage, agent, model — a time split belongs to a session, not to a tool call.
ch observability slowest
Prints the same fixed top-50 slowest-calls list the web app shows. There is no --limit. It is always the top 50.
ch observability sparkline
Prints the trend figures as a table, one row per bucket. Buckets follow the same rule as the app’s Auto (best fit): hours for a window of 25 hours or less, days otherwise. Add --resolution hour or --resolution day to choose explicitly. --compare previous prints a second table after the first, headed previous, for the equal-length window just before yours. --by <key> adds a series: list after the tables, with tool calls per bucket for that facet’s top values and an other remainder, the same split the app’s Series control draws.
ch observability markers
Prints the recorded stage-prompt, agent-setup, and release changes in the window, newest first — the same list as the Trend section’s Recorded changes table. It takes only a window (--window, or --since with an optional --until) and no filter flags.
Three more commands drill down from a breakdown row to the sessions behind it, like the app’s drill-down.
ch observability rows --by toolName
Pages through one facet’s complete row set, sorted by tool time by default. --by takes exactly one facet here, spelled the same way as above. It also takes --window or --since, --sort, --dir, --search, and --limit (200 by default and at most) with --cursor for paging.
ch observability contributors --tool-name Bash
Pages through the individual sessions behind whatever you have filtered to. It has no --by. You narrow it with the filter flags above. It takes a --since/--until range instead of --window. It pages 100 sessions at a time by default, up to 200, with --limit and --cursor.
ch observability trace --session <sessionId>
Prints one session’s full timeline of tool calls and cost turns. With --attempt it prints one stage attempt’s timeline instead. Give exactly one of --session or --attempt. It takes neither the filter flags above nor --window. By default it reads the last 24 hours. Use --since and --until to widen that range.
ch observability methods explains how each figure in a report is measured. It answers locally, without calling the server. Task-level flow — how long a task waited versus ran — is a separate report; see Reading the task flow report.
One session’s own tool calls
Every report above rolls up a whole workspace. To see one session’s own calls instead, open that session’s page and look at its Observability section, or run:
ch session tool-calls <sessionId>
Inside a session, you can leave out the session ID. The command prints one row per call, oldest first. Each row has when it started, its tool, label, category, status, and how long it took (or open if it never returned a result). It pages like any other ch list: 50 rows by default, 200 at most, with --cursor to keep going. See Sessions from the command line for the other things a finished session’s record can tell you.
How to read these numbers
A few things are worth knowing before you read tool time as a stopwatch:
- Overlapping calls each count in full, in the summary and the six breakdowns. Tool time there is the sum of every call’s own duration, not a measure of wall-clock time — if a session runs two calls at once, both durations count, so a window’s total tool time can exceed how long the window actually lasted. Time split’s own Tool time column is the one place overlap is not double-counted — see Time split above.
- Open calls are excluded from tool time. A call that started but never reported a result contributes to the open-calls count, never to tool calls or tool time, since there’s no known duration to add.
- A result recorded after the session ended isn’t counted either. Occasionally a session is stopped but the process behind one of its tool calls keeps running, and its result gets recorded hours later. That call is treated the same as an open call — it’s never counted as a real duration, so it can’t inflate tool time or land in the slowest-calls list.
- The record keeps the shape of a command, never its output. A call’s label strips out argument values and keeps the programs and operators behind it —
go test | tail, orgit status | head ; echo ; ch task. One thing survives that stripping: a script invoked by its path is named by that path, so a label can read./scripts/with-test-postgres.sh | tail. Nothing a tool printed is ever stored. Read a label as a sketch of what ran, and check the By label table yourself before you export or share it.
Who can see it
Any workspace member can read the Observability report, in the app or from the CLI — there’s no separate permission to grant.
Related guides
- Understanding costs — the spend side of the same story: what a task or agent cost, not where the time went
- Sessions from the command line — one session’s own tool calls, observations, and execution summary
- Reading the task flow report — where a task’s own elapsed time went, and why ready work isn’t running
- Alert rules — watch this report’s own queue-age figure and open an incident when it breaches
Last updated