Alert rules
Watch a metric on a rolling window, open an incident when it breaches, and route the alert to your activity feed or a webhook.
An alert rule watches one metric over a rolling window and opens an incident when the metric stays above its threshold for long enough. It’s opt-in: nothing in your workspace generates an incident until you create a rule for it.
Where to find it
To read your rules and their incidents, use the command line. Any workspace member can run these, and they’re the quickest way in:
ch observability alerts
ch observability alerts incidents --state open
See From the CLI below for the full set.
To create or change a rule, you need the app page. It lives at your workspace’s own address, https://app.codeherder.com/<slug path>/-/reports/observability/alerts — see Using the ch CLI for what a slug path is. To open it from the app, go to Work → Reports and click the Alert rules card under “Time and flow”.
The page has a Configured rules table with each rule’s name, metric, breach condition (for example > 30m for 15m), last verdict, last evaluation and failure count. Once at least one rule exists, a Recent incidents table sits below it. Open a rule to see its own Evaluation timeline and the Incidents it raised, with a window picker for how far back to look.
What a rule watches
A rule can watch one of two metrics:
- Task queue age (p95) — the same p95 figure the task flow report shows for how long queued work has been waiting.
- Git host delivery gap (repos) — how many repos have a webhook or GitHub App connected, yet show host activity that no delivery announced. It uses the same verdict as the repo page’s Integration health section. The rule’s Window is the silence that counts as a gap. Set the threshold to
0to alert on any repo with a gap. A past time cannot be replayed, so the evaluation timeline preview shows no data for this metric.
A rule breaches when the observed value rises above its threshold — there’s no “falls below” direction. On each evaluation tick, CodeHerder samples the metric over the rule’s window and compares it to the threshold. If the breach holds for the rule’s sustain period, an incident opens.
Creating a rule
Only an owner or admin can create, edit, or delete a rule. On the Alert rules page, click New alert rule. The form opens on its own page and asks for:
- Name — a label for the rule.
- Metric — which metric to watch.
- Threshold — the value that counts as a breach. For Task queue age (p95), enter a duration such as
30mor1h. For Git host delivery gap (repos), enter a number of repos. - Window — how far back each evaluation looks. Default
1h. - Sustain — how long a breach must hold before an incident opens. Default
15m. The same hold applies to recovery, so a brief dip back under threshold doesn’t resolve an incident early. - Cooldown — the quiet period after a resolution before the rule can fire again. Default
1h. - Minimum volume — below this many samples, the rule reports insufficient data instead of judging the metric. A sample is an open queued task, or for the delivery gap a push-connected repo. Default
1. - Scope — see below.
- Destination — see below.
Enter every duration as a number with a unit, such as 30m, 1h or 1h 30m.
Before you save, set a threshold and click Preview rule to replay it against your workspace’s recent history and see what it would have done. Preview writes nothing — no incident, no event, no notification.
Click Create alert rule to save. CodeHerder opens the new rule’s page. Cancel takes you back to the list. If you aren’t an owner or admin, the page tells you that only an owner or admin can create alert rules.
Scope
A rule’s scope decides which workspaces it evaluates:
- Self — only the workspace that owns the rule.
- Subtree — the owning workspace and every workspace nested beneath it.
Destination, and the webhook precondition
A rule’s destination decides where its fired and resolved events land, but both destinations emit the same underlying event — destination isn’t a second delivery path, it’s a precondition:
- Activity — the event reaches your workspace’s activity feed and anything already subscribed to it. No setup needed.
- Webhook — the event also needs a subscribed webhook. Before you can save a rule with destination Webhook (whether you create it, edit it or enable it), your workspace must already have an enabled webhook subscription carrying the
workspace.alert_firedevent type. The subscription can sit on the workspace itself, or on a parent group with Include descendant workspaces turned on. See Watching a whole group. If neither exists, the save is refused.
This is the easiest thing to get wrong when setting up a rule: set up your webhook subscription and its workspace.alert_fired event type first, then save the alert rule.
Evaluation verdicts
Each evaluation tick reaches one of five verdicts:
- OK — measured, and under the threshold.
- Breaching — measured, and over the threshold.
- Insufficient data — too few samples to judge at all.
- Stale data — the evidence behind the metric doesn’t reach back far enough to trust.
- Read failed — the metric couldn’t be read this tick.
Insufficient data and stale data are not the same as OK, and neither one resolves an open incident. If a rule can’t get a trustworthy reading, an incident it already opened stays open — a data outage should never look like recovery.
Incidents
An incident is open or resolved. While it’s open, CodeHerder tracks the first, peak, and most recent value it observed. An incident resolves once the metric holds back under threshold for the rule’s own sustain period.
Editing, silencing and deleting a rule
A rule’s own page has Edit, Disable/Enable and Delete. The list has Disable/Enable on each row, with Delete in the row’s actions. All of them need an owner or admin. Any workspace member can view the rule list, a rule’s own page, and its incidents.
Edit changes any setting except the metric. To watch a different metric, create a new rule.
Disable silences a rule. It stops evaluating and can open no new incidents, but it stays configured with all its settings intact. Editing an unrelated field on a disabled rule doesn’t re-enable it; you have to click Enable. If it already had an incident open when you disabled it, that incident stays open until you re-enable the rule and it resolves normally.
Delete asks you to confirm, then removes the rule and its incident history. To stop a rule from firing and keep that history, disable it instead.
From the CLI
Reading rules and incidents from the command line is open to any workspace member. Rule writes — create, edit, delete — are app-only; there’s no ch command for them.
ch observability alerts
ch observability alerts --disabled
ch observability alerts show <ruleId>
ch observability alerts incidents --state open
ch observability alerts incident <incidentId>
ch observability alerts timeline <ruleId> --window month
ch observability alertslists the workspace’s rules. Add--enabledor--disabledto narrow to enabled or disabled ones; leave both off to see all.ch observability alerts show <ruleId>shows one rule’s full configuration and its last evaluation.ch observability alerts incidentslists incidents, optionally filtered by rule (--rule <uuid>) or state (--state open/--state resolved).ch observability alerts incident <incidentId>shows one incident in full.ch observability alerts timeline <ruleId>replays a rule’s evaluation history alongside the real incidents it raised. It covers the last week by default. Use--window today|yesterday|week|monthor--since 7dto change that.
Related guides
- Reading the task flow report — where the Task queue age (p95) metric comes from
- Reading the Observability report — the report Alert rules sits alongside
- Webhooks — set up the subscription a webhook-destined rule needs
Last updated