Self-hosted service objectives
Measure each customer journey and feed on a self-hosted server, see the operating ranges CodeHerder states, and set your own targets from them.
This page helps you set service objectives for a self-hosted CodeHerder server. It states the timings the product produces. It shows how to measure each journey. It names the dependency failures that affect each one.
CodeHerder commits only to the supported ranges on this page. You set the target. Every target cell says “You set the target”.
A range comes from a setting or a fixed value in the server. It is not a promise about your network, database, or identity provider. Where no fixed value exists, this page says so. It does not invent a number.
Journeys
Each journey has an indicator to record. Record it from your proxy log, from a scheduled check, or from the worker board. Do not use the health endpoint alone. GET /v1/health proves the process and the database answer. It does not prove a journey works.
API request
- How to measure: Read
statusanddurationfor each request in your proxy access log. See Self-hosted logs. The server also sends aServer-Timingheader on each response. SetCH_SERVER_TIMING=falseto turn it off. - Indicator to record: The share of requests that return a status below 500. The 95th percentile of
duration, grouped by route. - Supported range: CodeHerder states no latency range for your setup. The load test in Sizing a self-hosted deployment measured the 95th percentile under 13 ms at up to 40 times the launch workload, on its own test hardware. Measure your own baseline.
- Failure classes: Database (slow queries, pool wait,
GET /v1/instance/metrics/db). Identity provider (sign-in requests only). - Target: You set the target.
SPA load
- How to measure: Time a page request through your proxy. Run a synthetic browser check against the sign-in page from outside your network.
- Indicator to record: The time until the page returns
200. The share of checks that pass. - Supported range: CodeHerder states no latency range yet. Time it from outside your network, through your proxy.
- Failure classes: Proxy or app host. The page itself needs no AI provider, git host, or device.
- Target: You set the target.
Sign-in
- How to measure: Run
ch instance checkon a schedule. See Self-hosted synthetic checks. Thesign_initem proves that the credential you run the command with authenticates against your server. Themailandsecret_decryptionitems check the server’s side. They shownot_runwhen sign-in fails. The command also checksdevice_connectionandrelease_retrieval. The check does not sign in through your identity provider. To measure browser sign-in, run your own synthetic browser sign-in. - Indicator to record: The result of each item, and the time the command takes.
- Supported range: CodeHerder states no latency range yet. Sign-in time depends mostly on your identity provider.
- Failure classes: Server or database (
sign_infails, so the credential cannot authenticate). Identity provider (browser sign-in fails; see Self-hosted Cognito sign-in). Mail (mailfails, so one-time codes do not arrive). Database. - Target: You set the target.
Session attach
Attach is the live terminal view of a running session. It uses a WebSocket.
- How to measure: Open a session from a scheduled check and record whether the socket connects. Count WebSocket closes in your proxy log.
- Indicator to record: The share of attach attempts that connect. The number of unplanned closes per hour.
- Supported range: The server pings each socket every 25 seconds. It gives a write 10 seconds to finish. It checks the caller’s access again every 50 seconds. A revoked credential stops working within 60 seconds. The browser reconnects after 1 second, then doubles the wait up to 30 seconds. It resets the wait after a connection holds for 5 seconds.
- Failure classes: Device (the session runs on a device, so a lost device ends the attach). Proxy (an idle timeout below 25 seconds cuts the socket). Database (the access check fails).
- Target: You set the target.
Task queue to session start
- How to measure: Compare the time a task became ready with the time its session started. Both stamps are on the task and session records.
ch task showprints the wait reason for a task that is not running. See Why isn’t my task moving?. - Indicator to record: The wait from ready to session start, per workflow. The count of tasks with a wait reason for longer than your target.
- Supported range: Task creation and stage change staff work at once when a device has room. The backstop reconcile pass runs every 30 seconds for staffing and capacity. A slower pass runs every 5 minutes. So the backstop retries a missed task about every 30 seconds. A device runs at most 4 sessions by default (
CH_MAX_DEVICE_SESSIONS). A failed spawn retries after about 30 seconds. The wait doubles with each failure, up to a cap of 10 to 15 minutes. The cap depends on the kind of failure. - Failure classes: Device (none online, or none with the needed capability). AI provider (the wait reason names it, or spawns fail and retry). Git host (clone fails, so the spawn retries). Database.
- Target: You set the target.
Device-loss recovery
- How to measure: Stop a test device and time how long its session takes to read
lost. PollGET /v1/instance/metrics/workersfor the sweeper that marks sessions lost. See Monitor background workers. - Indicator to record: The time from the last device frame to the
loststate. The time until the task has a new session on another device. - Supported range: A device sends a heartbeat every 30 seconds. The server checks every 30 seconds for a tunnel silent for more than 75 seconds, then drops it. The sweeper marks a session
lostwhen its device tunnel has been offline for 90 seconds. It also runs every 30 seconds. The 90 seconds start when the tunnel drops. A device that closes its connection cleanly readslost90 to 120 seconds after the last frame. A device that goes silent, for example on power or network loss, readslost165 to 225 seconds after the last frame. If the tunnel state is unclear, a session with no device contact for 900 seconds (CH_SESSION_STALE_CONTACT_SEC) readslost. A session lease lasts 300 seconds (CH_SESSION_LEASE_TTL_SECONDS). Work prefers its old device for 5 minutes (CH_DEVICE_AFFINITY_GRACE). After that any device can take it. A device reconnects after 1 second and doubles the wait up to 60 seconds. - Failure classes: Device (power, network, or process). Proxy (a proxy that drops idle sockets looks like device loss). Database (sweepers stall, so no session is marked
lost). - Target: You set the target.
Feeds
A feed is data that flows out of the server after an event. Record its delay, not only its availability.
| Feed | Delay the product produces | Cause | Where data can be lost | How to detect |
|---|---|---|---|---|
| Activity | Under a second on a connected browser. After a dropped socket, up to 30 seconds of reconnect wait, then a full refetch. | The server pushes each event after its database commit. An event written inside a larger transaction is pushed only once that transaction commits. The browser refetches every live list on reconnect. | None. The database row is the record. A missed push shows on the next refetch. | Compare an event’s stored time with the time it shows in the SPA. |
| Audit export: OpenTelemetry traces | About 10 seconds in normal use (CH_OTEL_EXPORT_INTERVAL). After a receiver outage, the export catches up on the next tick that succeeds. |
The server sends each batch of audit spans on every export tick. A failed send stays queued and is sent again on the next tick. | Not on an outage alone. The export keeps its read position until a send succeeds. A server restart or update while the export fails loses the queued spans: about one batch of up to 500 events. The export does not send them again. Retention also removes events that wait too long (CH_EVENTS_MAX_AGE_DAYS, default 180 days). |
Alert on the audit trace sweep failed log line. The server writes it on each failed export tick. The audittrace: export rollup line stops during an outage. When the export succeeds, that line shows exportFailures and cursorLagSec. See Alarm on trace export failures. |
Execution traces (CH_OTEL_EXECUTION_TRACES, off by default) |
About 5 seconds in normal use. Up to about 65 seconds while the receiver is down. | The exporter batches spans and sends every 5 seconds. Each send has a 10-second timeout (CH_OTEL_TRACES_TIMEOUT). |
Yes. The queue holds 2048 spans. A full queue drops spans. The exporter retries a failed send for up to 1 minute. Then it drops the batch and counts the failure. | Poll GET /v1/instance/metrics/trace-export. Alert when the failure count rises. |
| Audit export: webhooks | Seconds for a delivery that succeeds first time. | A delivery is tried up to 6 times. The wait starts at 30 seconds and doubles up to 1 hour. A subscription turns itself off after 15 failed deliveries in a row. | Yes. A delivery that fails 6 times does not reach your receiver unless you replay it. | Check the digest chain for a gap. See Verify the audit export. Digests arrive about every 15 minutes and cover deliveries older than 5 minutes. |
| Device status | A device that closes cleanly shows offline at once. A silent device shows offline 75 to 105 seconds after its last frame. Load metrics are at most 30 seconds old while the device is healthy. | The server checks every 30 seconds and drops a tunnel silent for more than 75 seconds. A device sends load metrics every 30 seconds (--metrics-interval). |
None. Status is current state, not history. | The device list marks metrics stale after 5 minutes. It marks capabilities stale after 45 minutes. |
The webhook export and the execution trace export can lose data. The audit trace export loses data in two cases: a server restart while the export fails, or retention that removes events first. If you can, fix the collector before you restart the server. Set your freshness target for the receiver, not for the server. Alert on a failing audit trace export and on a rising execution trace failure count. Check the webhook digest chain on a schedule.
You set the target for each feed.
Dependency failure classes
Classify each incident by the dependency that failed first. The class tells you which check to read.
| Dependency | Journeys affected | Where to look |
|---|---|---|
| Database | All journeys. Sweepers stall, so device loss goes unnoticed. | GET /v1/instance/metrics/db, /v1/instance/metrics/slow-queries, and the worker board. |
| Identity provider | Browser sign-in. | Your own synthetic browser sign-in. ch instance check does not reach the identity provider. |
| AI provider | Task queue to session start. Running sessions stall. | The task wait reason in ch task show. |
| Git host | Task queue to session start. Merge steps. | The task wait reason. |
| Device | Session attach. Session start. Device-loss recovery. | ch instance check, item device_connection. The device list. |
Related guides
Last updated