Alarm on trace export failures
Watch audit and execution trace export on a self-hosted server. Learn how long an undelivered event survives and set alarms.
A self-hosted CodeHerder server can export audit and execution spans to your collector over OTLP/HTTP. The export runs in the background. The server keeps answering requests when the collector is down. This page shows how to see a failed or late export, and how long you have before events are lost.
The server refuses to start under CH_ENV=production when the export settings are unsafe. It refuses a collector URL that is not https, the CH_OTEL_TRACES_INSECURE flag, and a malformed CH_OTEL_TRACES_HEADERS entry. Read the server log for the reason. The message names the setting and the entry position. It never prints a header value.
What to watch
| Signal | Where | Meaning |
|---|---|---|
audit trace sweep failed |
Server log, level WARN, one line for each failed audit export tick |
The collector refused or did not answer the audit export, or the sweep failed for another reason such as the database. Either way the export is stuck and retries on the next tick. |
audittrace: sdk reported an export error |
Server log, level WARN |
The collector refused or did not answer an execution trace export. |
exportFailures |
audittrace: export rollup log line, level INFO, about every 5 minutes |
The number of failed exports since the last rollup line. It counts failed audit exports. With CH_OTEL_EXECUTION_TRACES on, it also counts failed execution trace exports. While the audit export fails, the server writes no rollup line. The first rollup line after the export recovers shows every failure of the outage. |
cursorLagSec |
The same rollup line | The age of the newest event that the export sent on its last successful tick. A large value means the export is slow and behind. The value does not change while the export fails. |
exportFailures, cursorLagSeconds, lastSweepAt |
ch instance trace-export, or GET /v1/instance/metrics/trace-export |
The same counters for the execution trace export. |
runAuditTraceSweeper, runExecTraceSweeper |
ch instance workers |
Whether each export worker still finishes its ticks. See Monitor background workers. |
The ch instance trace-export command reports only when CH_OTEL_EXECUTION_TRACES is on. With it off, the command shows enabled: false and zero failures, even when the audit export fails. Use the log lines for the audit export.
The rollup line is itself a heartbeat. The export writes it about every 5 minutes, even when nothing failed. The export writes it only after a tick that succeeds. A missing rollup line therefore has two causes: the collector or the database is failing, or the export worker stopped. Look for audit trace sweep failed first.
How long an undelivered event survives
The audit export and the execution trace export behave differently.
Audit export. It does not drop spans when the collector is down. It can lose spans when the server restarts during the outage.
- Send. The export sends each batch of spans on its own tick, every
CH_OTEL_EXPORT_INTERVAL(default 10 seconds). It waits for the collector to accept the batch. - Retry. A failed send keeps the spans queued. The read position does not move. The next tick sends them again.
- Restart. The failed spans stay queued in server memory only. If the server restarts or updates while the export fails, the queued spans are lost. This is about one batch of up to 500 events. After the restart, the export reads those events again but does not send their spans again.
- Source rows. The events that the spans came from stay in the database. Retention removes them later:
CH_EVENTS_MAX_AGE_DAYS(default 180 days). - Stopped export. When the export worker stops, its position stays in the database. It resumes from that position. Events that are still inside the retention window are then exported. Events that retention already removed are lost. If the worker stopped because the server restarted during a failed export, the spans of the failed batch are also lost.
- Open spans. A stage or session span that never gets its closing event is closed after
CH_OTEL_SPAN_MAX_AGE(default 48 hours). It is markedcodeherder.span.unterminated.
Execution trace export. This export runs only when CH_OTEL_EXECUTION_TRACES is on. It keeps a failed export in memory for a short time only. Then it drops it.
- Retry. The exporter retries a failed batch with back-off for at most 1 minute. After that, the exporter drops the batch. It does not send the batch again.
- Queue. The exporter holds at most 2048 spans in memory. When the queue is full, it drops new spans.
- Cursor. The export advances its read position when it hands spans to the exporter. It does not wait for the collector to accept them. A dropped span is therefore not read again.
- Source rows. Retention removes the source rows after
CH_EXECUTION_OBSERVATION_RETENTION_DAYS(default 90 days).
So the loss limit differs. An audit export loses events in two cases. A server restart during a failed export loses about one batch of up to 500 events. Events that pass the retention age are also lost. Alarm well before that age. If you can, fix the collector before you restart the server. An execution trace export that loses its collector for more than 1 minute loses the spans in flight at that time.
Alarm recipe for CloudWatch
This recipe uses the log shipping in Ship and redact logs. Vector writes each server log line to the CloudWatch group in sink-cloudwatch.yaml. Vector parses the line into an app object. The log text is in message.
Create three metric filters on that log group. Give each one a metric in your own namespace. Then create the alarms.
| Filter name | Filter pattern | Metric value |
|---|---|---|
TraceExportError |
{ $.app.msg = "audit trace sweep failed" } |
1 |
TraceExportRollup |
{ $.app.msg = "audittrace: export rollup" } |
1 |
TraceExportLag |
{ $.app.msg = "audittrace: export rollup" && $.app.cursorLagSec > 900 } |
1 |
Test each pattern before you rely on it. Run aws logs test-metric-filter with a sample rollup line. Vector keeps values from the server log as text. If TraceExportLag does not match the sample, do not rely on that alarm. Tell the support contact. Watch the cursorLagSec value in the log by hand until it is fixed.
Set these alarms. Send each one to an SNS topic.
| Alarm | Condition | Action |
|---|---|---|
| Export failing | TraceExportError sum is above 0 in 2 consecutive 5-minute periods |
Page your on-call responder. |
| Export behind | TraceExportLag sum is above 0 in 2 consecutive 5-minute periods |
Page your on-call responder. This alarm catches a slow export. It does not fire while the collector is down, because no rollup line is written then. TraceExportError catches a down collector. |
| Export stopped | TraceExportRollup sum is 0 for 3 consecutive 5-minute periods, with missing data treated as breaching |
Page your on-call responder. A failing collector also stops the rollup line, so check for audit trace sweep failed first. If that line is absent, run ch instance workers and check runAuditTraceSweeper. |
Only alarm on the last row when the export is on. A server with no CH_OTEL_TRACES_ENDPOINT writes no rollup line.
Escalate to the support contact
- Your on-call responder checks the collector first. Is it up? Does its certificate still verify? Did its ingest key change?
- If the collector is fine and the alarm lasts longer than your own threshold (______), contact the support contact through the channel in your support agreement. Put the alarm name, the time it started, and the last rollup line in the message.
- If the lag is above 86400 seconds (24 hours), treat it as SEV2 and contact the support contact at once. This is well before the 48-hour span limit and the 90-day and 180-day retention limits. See Self-hosted support and incident response.
- Do not paste header values, tokens or the collector URL with its key into the message.
- After the fix, the first rollup line still shows the failures of the outage. Its
windowSeccan be much more than 300. Read the second rollup line after the fix. ItswindowSecmust be near 300,exportFailuresmust be0, andcursorLagSecmust fall.
Your support agreement sets the SEV2 answer time. CodeHerder runs no on-call service for self-hosted installs. Your responder owns the first response.
Last updated