Rotating self-hosted at-rest keys

Rotate any of the four at-rest keys or the KMS key on a running self-hosted server, reseal every stored secret, and drop the old key safely.

A self-hosted CodeHerder server holds four at-rest keys and one optional AWS KMS key. Rotate a key on a schedule, or at once after a leak. Each rotation uses a previous-key slot, so the server keeps working while you switch. Your organization runs the rotation. CodeHerder does not do it for you.

The keys and what each protects

Key	What it protects	What you do after you switch
`CH_WEBHOOK_SECRET_KEY`	Outbound webhook signing secrets and inbound webhook endpoint secrets	Run `codeherder reseal`
`CH_INTEGRATIONS_SECRET_KEY`	Integration connection secrets and pending OAuth connections	Run `codeherder reseal`
`CH_OTP_HMAC_KEY`	Sign-in and email-change codes that are in flight	Wait at least 10 minutes
`CH_INTEGRATIONS_OAUTH_STATE_KEY`	OAuth flows that are in flight	Wait at least 10 minutes
`CH_KMS_KEY_ID`	Values stored with `ch variable` as a secret	Run `codeherder reseal`

Each of the first four keys has a previous-key slot. The name of the slot is the key name with _PREVIOUS_KEY in place of _KEY:

Current key	Previous-key slot
`CH_WEBHOOK_SECRET_KEY`	`CH_WEBHOOK_SECRET_PREVIOUS_KEY`
`CH_INTEGRATIONS_SECRET_KEY`	`CH_INTEGRATIONS_SECRET_PREVIOUS_KEY`
`CH_OTP_HMAC_KEY`	`CH_OTP_HMAC_PREVIOUS_KEY`
`CH_INTEGRATIONS_OAUTH_STATE_KEY`	`CH_INTEGRATIONS_OAUTH_STATE_PREVIOUS_KEY`

The server encrypts and signs with the current key only. It reads and verifies with the current key, then the previous key. Set a previous key only during a rotation. The server refuses to start when a previous key has a bad shape or equals the current key.

Before you start

Back up the database. See Self-hosted backup and recovery.
Generate the new key with openssl rand -hex 32.
Put the new key in escrow before you use it. A key you lose cannot be rebuilt. See Escrow the four keys before first boot.
Keep the old key. You need it until the last step.

Rotate one key at a time. You can rotate all four in one window, but a single change is easier to check.

Rotate an encryption key

Use these steps for CH_WEBHOOK_SECRET_KEY and CH_INTEGRATIONS_SECRET_KEY.

On every replica, set the new key as the current key. Set the old key as the previous key. For example:
```
export CH_WEBHOOK_SECRET_KEY=<new key>
export CH_WEBHOOK_SECRET_PREVIOUS_KEY=<old key>
```
Restart every replica. Wait until all replicas run the new setting. Do not run the next step before this is true. A replica that still uses the old key writes new secrets under the old key.
Run the reseal command with the same environment as the server:
```
codeherder reseal --db "<the --db value of your server>"
```
Add --dry-run first if you want counts with no writes. You can run the command while the server serves traffic.
Read the output. It prints one line for each table, then a total. Run the command again until it prints result: ok with resealed=0 and unreadable=0. A changed_concurrently count means a live write changed a row during the run. That row is safe. Run the command again.
Unset the previous key on every replica, then restart every replica.
Read the boot log. It must not show an at-rest key mismatch error. If it does, see Never mint a new key on an existing database.
Erase the old key from your environment files. Keep it in escrow for as long as your backups of the database need it. A restored old backup holds secrets sealed under the old key.

If the command prints unreadable rows, stop. No configured key opens those rows. Do not drop the previous key. Find the key that sealed them, or enter those secrets again through the API.

What the reseal command does

It rewrites each row that only the previous key opens, under the current key.
It changes one row at a time, and only if the row is unchanged since it was read.
It leaves rows that the current key already opens. A second run changes nothing.
It rewrites each legacy aesgcm-v1 row as aesgcm-v2, bound to its workspace and row id.
It skips plain rows and counts them. A plain row has no authentication, so sealing it would trust bytes nobody verified. Enter that secret again through the API.
It refuses to run when the current key is missing or invalid. It never seals to plain text.

Before you upgrade past the release that drops aesgcm-v1

This release stops reading aesgcm-v1 rows. The release also tightens the database so no new aesgcm-v1 row can exist. Reseal first, then upgrade.

Back up the database.
Download the new release. Do not start it yet. The old server keeps serving.
Run codeherder reseal --dry-run with the new binary and the same environment keys as the server. Read the counts.
Run codeherder reseal. Run it again until it prints result: ok with resealed=0 and unreadable=0. The old server reads aesgcm-v2, so this is safe beside live traffic.
Upgrade the server.

If you skip step 4, the new server refuses to start. The migration stops with aesgcm-v1 secret rows remain and names the tables and counts. It applies nothing, so the old binary still runs on that database. Run codeherder reseal, then start the new server again.

If a row is unreadable, no configured key opens it. Enter that secret again on the old release. Rotate the webhook secret, or reconnect the integration. Then run codeherder reseal again.

Rotate a signing key

Use these steps for CH_OTP_HMAC_KEY and CH_INTEGRATIONS_OAUTH_STATE_KEY. These keys sign short-lived codes and OAuth state. They do not protect stored secrets, so there is nothing to reseal.

On every replica, set the new key as the current key. Set the old key as the previous key.
Restart every replica.
Wait at least 10 minutes after the last replica runs the new key. Sign-in codes and OAuth state expire after 10 minutes. Until then the server accepts the old key for them.
Unset the previous key on every replica, then restart every replica.

A person who starts a sign-in before step 2 can finish it during the 10 minutes. After the previous key is gone, that person must ask for a new code.

codeherder reseal prints a reminder of this window for each signing key. It does not change anything for them.

Rotate the KMS key

Use these steps to move sealed variables to a new AWS KMS key.

Create the new KMS key, or choose one. It must be a symmetric encryption key in the region of CH_KMS_REGION.
Grant the server role kms:Encrypt, kms:Decrypt, kms:GenerateDataKey and kms:DescribeKey on the new key. Keep kms:Decrypt on the old key. See Self-hosted infrastructure.
Set CH_KMS_KEY_ID to the new key on every replica. Restart every replica. New secrets use the new key at once. Old secrets still open, because the server reads the key ID stored with each secret.
Run codeherder reseal --db "<the --db value of your server>". It wraps the data key of each stored secret again under the new key. It does not change the secret value or its version.
Run the command again until the variables (kms) line shows resealed=0 and unreadable=0.
Keep the old KMS key enabled. Do not schedule its deletion. A backup of the database taken before the rotation holds data keys that only the old KMS key opens. Delete the old key only after every such backup has expired.

AWS automatic rotation of the same KMS key needs no action. The key ARN does not change, and AWS keeps the old key material to open old data. The reseal command counts those rows as already current.

Check that it worked

codeherder reseal prints result: ok, and a second run shows resealed=0.
The server starts with the previous key unset, and the boot log has no key-mismatch line.
Run the synthetic checks. See Self-hosted synthetic checks.

Last updated 3 October 2026