# Self-hosted synthetic checks

Source: https://codeherder.com/docs/self-host-checks/

Run one command on a schedule to prove sign-in, mail, secrets, devices and updates work on your self-hosted server.

A server that answers a health probe can still be broken. Sign-in, mail, stored secrets, device connections and updates can each fail while the server stays up. The `ch instance check` command tests all five and exits non-zero when any check does not pass. Run it on a schedule and alarm on the exit code.

## What each check proves

| Check | It passes when |
| --- | --- |
| `sign_in` | The credential you run the command with authenticates against your server. |
| `mail` | The configured mailer accepts one test message. The message goes to your own verified email address. |
| `secret_decryption` | The host keys (`CH_WEBHOOK_SECRET_KEY` and `CH_INTEGRATIONS_SECRET_KEY`) open a secret already stored in the database. The check never writes. |
| `device_connection` | At least one device tunnel is live on the server process that answered. |
| `release_retrieval` | The update manifest downloads, and its signature verifies against the release key built into `ch`. This check runs on the machine that runs `ch`. |

The `mail` check fails when the server uses the fake mailer, because that mailer sends nothing. Set `CH_MAILER=ses`, `CH_MAILER_FROM` and `CH_SES_REGION` to deliver mail.

The `device_connection` check fails when no device is connected. If your install has no devices, skip it.

## Before you start

1. Make sure your email address is listed in `CH_INSTANCE_OPERATORS` on the server. Verify that address in your account.
2. Create an API key for the account that runs the check. See [Credentials and profiles](https://codeherder.com/docs/credentials/).
3. Install `ch` on the machine that runs the schedule. Set `CH_ADDR` to your server and `CH_TOKEN` to the API key.

## Run it

```
ch instance check
```

The command prints one line for each check:

```
sign_in            pass     the caller's credential authenticated
mail               pass     the mailer accepted a test send to the operator's own address
secret_decryption  pass     the host key decrypts stored secrets in 2 key slot(s)
device_connection  pass     3 device tunnel(s) live on this server process
release_retrieval  pass     the update manifest fetched and its signature verified (version 3.117.34)
```

The exit code is 0 when every check passes. It is 1 when any check fails. If sign-in fails, the server checks show `not_run`.

Add `--json` for output that a monitoring agent can read. Add `--skip <check>` to leave one check out. Repeat `--skip` to leave out more.

The `device_connection` check counts tunnels on one server process. If you run more than one server process, a run answered by a process with no devices fails. Alarm on repeated failures, not on one.

## Schedule it

Do not send a test message every few minutes. Run the cheap checks often and the full set less often.

Add these lines to a crontab on a machine you trust:

```
MAILTO=oncall@example.com
CH_ADDR=https://codeherder.example.com
CH_TOKEN=<your-api-key>

*/5 * * * * ch instance check --skip mail --skip release_retrieval
0 * * * *   ch instance check
```

Cron sends the output to `MAILTO` when the command prints anything. A passing run still prints, so use `chronic` from moreutils, or wrap the command in a script that prints only on failure.

With systemd, run the command from a timer. Set `OnFailure=` on the service to start a unit that pages your team.

With a monitoring agent, run the command with `--json`. Alarm when the exit code is not 0, or when `ok` is `false`. Alarm also when the check does not run at all, because a silent scheduler looks like a healthy install.

## When a check fails

- **`sign_in` fails.** The API key is wrong, revoked or expired, or the server is down. Run `ch whoami` to see the error.
- **`mail` fails.** Check `CH_MAILER`, `CH_MAILER_FROM` and `CH_SES_REGION`. Check that SES has verified the sender and that your account is out of the SES sandbox. Also check that the operator email address is verified.
- **`secret_decryption` fails.** The server does not hold the key that sealed its stored secrets. This happens after a restore onto a new host without the old keys. Put the original `CH_WEBHOOK_SECRET_KEY` and `CH_INTEGRATIONS_SECRET_KEY` back and restart. If the original keys are lost, enter each webhook and integration secret again.
- **`device_connection` fails.** No device is connected to this server process. Check that the device server is running and that it can reach your server.
- **`release_retrieval` fails.** The machine cannot reach the release source, or the manifest signature is wrong. Check the network path and `CH_CLI_MANIFEST_URL`. Do not update from a manifest that fails signature verification.

## Related guides

- [Self-hosted deployment](https://codeherder.com/docs/self-hosting/) — install, configure and upgrade the server
- [Self-hosted logs](https://codeherder.com/docs/self-host-logs/) — ship server logs off the host