# Rotating self-hosted at-rest keys

Source: https://codeherder.com/docs/self-host-key-rotation/

Rotate any of the four at-rest keys or the KMS key on a running self-hosted server, reseal every stored secret, and drop the old key safely.

A self-hosted CodeHerder server holds four at-rest keys and one optional AWS KMS key. Rotate a key on a schedule, or at once after a leak. Each rotation uses a previous-key slot, so the server keeps working while you switch. Your organization runs the rotation. CodeHerder does not do it for you.

## The keys and what each protects

| Key | What it protects | What you do after you switch |
| --- | --- | --- |
| `CH_WEBHOOK_SECRET_KEY` | Outbound webhook signing secrets and inbound webhook endpoint secrets | Run `codeherder reseal` |
| `CH_INTEGRATIONS_SECRET_KEY` | Integration connection secrets and pending OAuth connections | Run `codeherder reseal` |
| `CH_OTP_HMAC_KEY` | Sign-in and email-change codes that are in flight | Wait at least 10 minutes |
| `CH_INTEGRATIONS_OAUTH_STATE_KEY` | OAuth flows that are in flight | Wait at least 10 minutes |
| `CH_KMS_KEY_ID` | Values stored with `ch variable` as a secret | Run `codeherder reseal` |

Each of the first four keys has a previous-key slot. The name of the slot is the key name with `_PREVIOUS_KEY` in place of `_KEY`:

| Current key | Previous-key slot |
| --- | --- |
| `CH_WEBHOOK_SECRET_KEY` | `CH_WEBHOOK_SECRET_PREVIOUS_KEY` |
| `CH_INTEGRATIONS_SECRET_KEY` | `CH_INTEGRATIONS_SECRET_PREVIOUS_KEY` |
| `CH_OTP_HMAC_KEY` | `CH_OTP_HMAC_PREVIOUS_KEY` |
| `CH_INTEGRATIONS_OAUTH_STATE_KEY` | `CH_INTEGRATIONS_OAUTH_STATE_PREVIOUS_KEY` |

The server encrypts and signs with the current key only. It reads and verifies with the current key, then the previous key. Set a previous key only during a rotation. The server refuses to start when a previous key has a bad shape or equals the current key.

## Before you start

1. Back up the database. See [Self-hosted backup and recovery](https://codeherder.com/docs/self-host-backups/).
2. Generate the new key with `openssl rand -hex 32`.
3. Put the new key in escrow before you use it. A key you lose cannot be rebuilt. See [Escrow the four keys before first boot](https://codeherder.com/docs/self-host-backups/#escrow-the-four-keys-before-first-boot).
4. Keep the old key. You need it until the last step.

Rotate one key at a time. You can rotate all four in one window, but a single change is easier to check.

## Rotate an encryption key

Use these steps for `CH_WEBHOOK_SECRET_KEY` and `CH_INTEGRATIONS_SECRET_KEY`.

1. On every replica, set the new key as the current key. Set the old key as the previous key. For example: `export CH_WEBHOOK_SECRET_KEY =< new key > export CH_WEBHOOK_SECRET_PREVIOUS_KEY =< old key >`
2. Restart every replica. Wait until all replicas run the new setting. Do not run the next step before this is true. A replica that still uses the old key writes new secrets under the old key.
3. Run the reseal command with the same environment as the server: `codeherder reseal --db "<the --db value of your server>"` Add `--dry-run` first if you want counts with no writes. You can run the command while the server serves traffic.
4. Read the output. It prints one line for each table, then a total. Run the command again until it prints `result: ok` with `resealed=0` and `unreadable=0`. A `changed_concurrently` count means a live write changed a row during the run. That row is safe. Run the command again.
5. Unset the previous key on every replica, then restart every replica.
6. Read the boot log. It must not show an `at-rest key mismatch` error. If it does, see [Never mint a new key on an existing database](https://codeherder.com/docs/self-host-backups/#never-mint-a-new-key-on-an-existing-database).
7. Erase the old key from your environment files. Keep it in escrow for as long as your backups of the database need it. A restored old backup holds secrets sealed under the old key.

If the command prints `unreadable` rows, stop. No configured key opens those rows. Do not drop the previous key. Find the key that sealed them, or enter those secrets again through the API.

### What the reseal command does

- It rewrites each row that only the previous key opens, under the current key.
- It changes one row at a time, and only if the row is unchanged since it was read.
- It leaves rows that the current key already opens. A second run changes nothing.
- It rewrites each legacy `aesgcm-v1` row as `aesgcm-v2`, bound to its workspace and row id.
- It skips `plain` rows and counts them. A `plain` row has no authentication, so sealing it would trust bytes nobody verified. Enter that secret again through the API.
- It refuses to run when the current key is missing or invalid. It never seals to plain text.

## Before you upgrade past the release that drops aesgcm-v1

This release stops reading `aesgcm-v1` rows. The release also tightens the database so no new `aesgcm-v1` row can exist. Reseal first, then upgrade.

1. Back up the database.
2. Download the new release. Do not start it yet. The old server keeps serving.
3. Run `codeherder reseal --dry-run` with the new binary and the same environment keys as the server. Read the counts.
4. Run `codeherder reseal`. Run it again until it prints `result: ok` with `resealed=0` and `unreadable=0`. The old server reads `aesgcm-v2`, so this is safe beside live traffic.
5. Upgrade the server.

If you skip step 4, the new server refuses to start. The migration stops with `aesgcm-v1 secret rows remain` and names the tables and counts. It applies nothing, so the old binary still runs on that database. Run `codeherder reseal`, then start the new server again.

If a row is `unreadable`, no configured key opens it. Enter that secret again on the old release. Rotate the webhook secret, or reconnect the integration. Then run `codeherder reseal` again.

## Rotate a signing key

Use these steps for `CH_OTP_HMAC_KEY` and `CH_INTEGRATIONS_OAUTH_STATE_KEY`. These keys sign short-lived codes and OAuth state. They do not protect stored secrets, so there is nothing to reseal.

1. On every replica, set the new key as the current key. Set the old key as the previous key.
2. Restart every replica.
3. Wait at least 10 minutes after the last replica runs the new key. Sign-in codes and OAuth state expire after 10 minutes. Until then the server accepts the old key for them.
4. Unset the previous key on every replica, then restart every replica.

A person who starts a sign-in before step 2 can finish it during the 10 minutes. After the previous key is gone, that person must ask for a new code.

`codeherder reseal` prints a reminder of this window for each signing key. It does not change anything for them.

## Rotate the KMS key

Use these steps to move sealed variables to a new AWS KMS key.

1. Create the new KMS key, or choose one. It must be a symmetric encryption key in the region of `CH_KMS_REGION`.
2. Grant the server role `kms:Encrypt`, `kms:Decrypt`, `kms:GenerateDataKey` and `kms:DescribeKey` on the new key. Keep `kms:Decrypt` on the old key. See [Self-hosted infrastructure](https://codeherder.com/docs/self-host-infrastructure/).
3. Set `CH_KMS_KEY_ID` to the new key on every replica. Restart every replica. New secrets use the new key at once. Old secrets still open, because the server reads the key ID stored with each secret.
4. Run `codeherder reseal --db "<the --db value of your server>"`. It wraps the data key of each stored secret again under the new key. It does not change the secret value or its version.
5. Run the command again until the `variables (kms)` line shows `resealed=0` and `unreadable=0`.
6. Keep the old KMS key enabled. Do not schedule its deletion. A backup of the database taken before the rotation holds data keys that only the old KMS key opens. Delete the old key only after every such backup has expired.

AWS automatic rotation of the same KMS key needs no action. The key ARN does not change, and AWS keeps the old key material to open old data. The reseal command counts those rows as already current.

## Check that it worked

- `codeherder reseal` prints `result: ok`, and a second run shows `resealed=0`.
- The server starts with the previous key unset, and the boot log has no key-mismatch line.
- Run the synthetic checks. See [Self-hosted synthetic checks](https://codeherder.com/docs/self-host-checks/).

## Related

- [Self-hosted backup and recovery](https://codeherder.com/docs/self-host-backups/)
- [Self-hosted support and incident response](https://codeherder.com/docs/self-host-support/)
- [Replacing a compromised self-hosted server](https://codeherder.com/docs/self-host-compromise/)