# Self-hosted key custody

Source: https://codeherder.com/docs/self-host-keys/

Keep the four at-rest keys outside the host, reuse them on a new host, set the production key rules, and protect and recover the KMS key.

A self-hosted CodeHerder server depends on two kinds of key. You hold both. This guide covers custody: who keeps each key, which settings make the server strict, and what to do if a key is lost.

## The keys you hold

| Key | Where it lives | What it protects |
| --- | --- | --- |
| `CH_WEBHOOK_SECRET_KEY` | App host | Webhook secrets |
| `CH_INTEGRATIONS_SECRET_KEY` | App host | Integration secrets |
| `CH_OTP_HMAC_KEY` | App host | Sign-in codes in flight |
| `CH_INTEGRATIONS_OAUTH_STATE_KEY` | App host | OAuth flows in flight |
| KMS key (`CH_KMS_KEY_ID`) | Your AWS account | Every value stored with `ch variable` as a secret |

See [What to back up](https://codeherder.com/docs/self-host-backups/#what-to-back-up) for the cost of losing each one.

## Keep a copy outside the host

Store the four values in a secret store that does not run on the app host. Keep one offline copy for the case where the store itself is lost. Do this before the first boot. The steps are in [Escrow the four keys before first boot](https://codeherder.com/docs/self-host-backups/#escrow-the-four-keys-before-first-boot).

Keep a custody record. Write down:

- the name of the store;
- the people who can open the offline copy;
- the date of the last read-back check.

At each check, read a value back from the store. Compare it with the file on the host.

## Reuse the keys on a new host

A new host, a rebuild and a restore all take the escrowed values. Never make new keys for an existing database. The server reads existing rows at boot. With `CH_ENV=production`, a wrong key stops the boot with a fatal error. The check is `verifyAtRestKeyCanaries` in `cmd/codeherder/at_rest_guards.go`. See [Never mint a new key on an existing database](https://codeherder.com/docs/self-host-backups/#never-mint-a-new-key-on-an-existing-database).

To change a key on purpose, rotate it. See [Rotating self-hosted at-rest keys](https://codeherder.com/docs/self-host-key-rotation/).

## Production settings

Set these two values on the app host:

```
# /etc/systemd/system/codeherder.service.d/production.conf
[Service]
Environment=CH_ENV=production
Environment=CH_REQUIRE_STRONG_AT_REST_KEY=1
```

The repository ships this file as the `production.conf` drop-in of the API host. Then run `systemctl daemon-reload` and restart the server.

With these values, the server enforces these rules at boot:

- `CH_WEBHOOK_SECRET_KEY` and `CH_INTEGRATIONS_SECRET_KEY` are 64 hex characters.
- `CH_OTP_HMAC_KEY` and `CH_INTEGRATIONS_OAUTH_STATE_KEY` are at least 32 bytes.
- The server never invents a key that is missing.
- The database connection uses TLS. See [Self-hosted database encryption and TLS](https://codeherder.com/docs/self-host-database/).
- A setting that weakens security, such as `CH_DEV_ORIGINS`, makes the server refuse to start. Leave it unset.

A deployment that runs a 32-character raw key today must rotate that key to the 64-hex form first. Setting the flag alone does not convert it. See [Rotating self-hosted at-rest keys](https://codeherder.com/docs/self-host-key-rotation/).

## Who holds the KMS key

The KMS key lives in your AWS account. Your key administrators manage it. These are the principals in `key_admin_arns`, or the account root if you leave that empty. Only the server role may use it for encrypt and decrypt. CodeHerder staff have no access. See [Self-hosted KMS keys and attachments bucket](https://codeherder.com/docs/self-host-kms-and-attachments/#create-the-secret-store-key).

## Where the key material comes from

The Terraform module makes a key with the origin `AWS_KMS`. KMS makes the key material inside its hardware security modules. Nobody can export it. This holds for you and for CodeHerder.

The server never holds the key material either. For each secret it stores a wrapped data key, not key material (`Cipher.Encrypt` in `internal/variables/cipher.go`). So there is no copy of the KMS key to escrow. Protect the key from deletion and plan recovery instead.

Do not import your own key material unless your policy requires it. If you do, you own the backup of that material.

## Protect the key from deletion

KMS has no deletion-protection flag. Use these controls:

- **A deletion window.** KMS waits 30 days before it deletes a key. The module sets `deletion_window_in_days` to 30.
- **Named administrators.** Allow `kms:ScheduleKeyDeletion` and `kms:DisableKey` only to `key_admin_arns`. Add a service control policy if your organization uses them.
- **An alarm.** Alert on `ScheduleKeyDeletion` and `DisableKey` events in CloudTrail.
- **A cancel step.** Inside the window, run `aws kms cancel-key-deletion --key-id <key>`, then enable the key again.

## Recover

The supported strategy is to keep the key alive, not to restore it. You cannot export or re-import the key material, so a deleted key is gone.

| Event | What to do |
| --- | --- |
| Key disabled | Enable it again. No data is lost. |
| Key pending deletion | Cancel the deletion inside the window. |
| Key deleted | Every secret stored with `ch variable` is lost. Create a new key, set `CH_KMS_KEY_ID`, and enter each secret again. The four env keys, the database and the attachments are not affected. |
| Region lost | Use a replica of a multi-Region key. You choose this when you create the key. See below. |

To allow a replica, set `multi_region = true` in the module when you create the key. AWS cannot change this later. Then follow [Keep recovery copies in a second region](https://codeherder.com/docs/self-host-backups/#keep-recovery-copies-in-a-second-region).

After a key rotation, keep the old key until every backup that needs it has expired. See [Rotate the KMS key](https://codeherder.com/docs/self-host-key-rotation/#rotate-the-kms-key).

## Related

- [Self-hosted backup and recovery](https://codeherder.com/docs/self-host-backups/)
- [Rotating self-hosted at-rest keys](https://codeherder.com/docs/self-host-key-rotation/)
- [Self-hosted KMS keys and attachments bucket](https://codeherder.com/docs/self-host-kms-and-attachments/)
- [Check your KMS permissions](https://codeherder.com/docs/self-host-kms-check/)
- [Self-hosted database encryption and TLS](https://codeherder.com/docs/self-host-database/)