Self-hosted synthetic checks

A server that answers a health probe can still be broken. Sign-in, mail, stored secrets, device connections and updates can each fail while the server stays up. The ch instance check command tests all five and exits non-zero when any check does not pass. Run it on a schedule and alarm on the exit code.

What each check proves

Check	It passes when
sign_in	The credential you run the command with authenticates against your server.
mail	The configured mailer accepts one test message. The message goes to your own verified email address.
secret_decryption	The host keys (CH_WEBHOOK_SECRET_KEY and CH_INTEGRATIONS_SECRET_KEY) open a secret already stored in the database. The check never writes.
device_connection	At least one device tunnel is live on the server process that answered.
release_retrieval	The update manifest downloads, and its signature verifies against the release key built into ch. This check runs on the machine that runs ch.

The mail check fails when the server uses the fake mailer, because that mailer sends nothing. Set CH_MAILER=ses, CH_MAILER_FROM and CH_SES_REGION to deliver mail.

The device_connection check fails when no device is connected. If your install has no devices, skip it.

Before you start

1. Make sure your email address is listed in CH_INSTANCE_OPERATORS on the server. Verify that address in your account.
2. Create an API key for the account that runs the check. See Credentials and profiles.
3. Install ch on the machine that runs the schedule. Set CH_ADDR to your server and CH_TOKEN to the API key.

Run it

ch instance check

The command prints one line for each check:

sign_in            pass     the caller's credential authenticated
mail               pass     the mailer accepted a test send to the operator's own address
secret_decryption  pass     the host key decrypts stored secrets in 2 key slot(s)
device_connection  pass     3 device tunnel(s) live on this server process
release_retrieval  pass     the update manifest fetched and its signature verified (version 3.117.34)

The exit code is 0 when every check passes. It is 1 when any check fails. If sign-in fails, the server checks show not_run.

Add --json for output that a monitoring agent can read. Add --skip <check> to leave one check out. Repeat --skip to leave out more.

The device_connection check counts tunnels on one server process. If you run more than one server process, a run answered by a process with no devices fails. Alarm on repeated failures, not on one.

Schedule it

Do not send a test message every few minutes. Run the cheap checks often and the full set less often.

Add these lines to a crontab on a machine you trust:

MAILTO=oncall@example.com
CH_ADDR=https://codeherder.example.com
CH_TOKEN=<your-api-key>

*/5 * * * * ch instance check --skip mail --skip release_retrieval
0 * * * *   ch instance check

Cron sends the output to MAILTO when the command prints anything. A passing run still prints, so use chronic from moreutils, or wrap the command in a script that prints only on failure.

With systemd, run the command from a timer. Set OnFailure= on the service to start a unit that pages your team.

With a monitoring agent, run the command with --json. Alarm when the exit code is not 0, or when ok is false. Alarm also when the check does not run at all, because a silent scheduler looks like a healthy install.

When a check fails

• sign_in fails. The API key is wrong, revoked or expired, or the server is down. Run ch whoami to see the error.
• mail fails. Check CH_MAILER, CH_MAILER_FROM and CH_SES_REGION. Check that SES has verified the sender and that your account is out of the SES sandbox. Also check that the operator email address is verified.
• secret_decryption fails. The server does not hold the key that sealed its stored secrets. This happens after a restore onto a new host without the old keys. Put the original CH_WEBHOOK_SECRET_KEY and CH_INTEGRATIONS_SECRET_KEY back and restart. If the original keys are lost, enter each webhook and integration secret again.
• device_connection fails. No device is connected to this server process. Check that the device server is running and that it can reach your server.
• release_retrieval fails. The machine cannot reach the release source, or the manifest signature is wrong. Check the network path and CH_CLI_MANIFEST_URL. Do not update from a manifest that fails signature verification.

Related guides

• Self-hosted deployment — install, configure and upgrade the server
• Self-hosted logs — ship server logs off the host