Self-hosted key custody

Keep the four at-rest keys outside the host, reuse them on a new host, set the production key rules, and protect and recover the KMS key.

A self-hosted CodeHerder server depends on two kinds of key. You hold both. This guide covers custody: who keeps each key, which settings make the server strict, and what to do if a key is lost.

The keys you hold

Key	Where it lives	What it protects
`CH_WEBHOOK_SECRET_KEY`	App host	Webhook secrets
`CH_INTEGRATIONS_SECRET_KEY`	App host	Integration secrets
`CH_OTP_HMAC_KEY`	App host	Sign-in codes in flight
`CH_INTEGRATIONS_OAUTH_STATE_KEY`	App host	OAuth flows in flight
KMS key (`CH_KMS_KEY_ID`)	Your AWS account	Every value stored with `ch variable` as a secret

See What to back up for the cost of losing each one.

Keep a copy outside the host

Store the four values in a secret store that does not run on the app host. Keep one offline copy for the case where the store itself is lost. Do this before the first boot. The steps are in Escrow the four keys before first boot.

Keep a custody record. Write down:

the name of the store;
the people who can open the offline copy;
the date of the last read-back check.

At each check, read a value back from the store. Compare it with the file on the host.

Reuse the keys on a new host

A new host, a rebuild and a restore all take the escrowed values. Never make new keys for an existing database. The server reads existing rows at boot. With CH_ENV=production, a wrong key stops the boot with a fatal error. The check is verifyAtRestKeyCanaries in cmd/codeherder/at_rest_guards.go. See Never mint a new key on an existing database.

To change a key on purpose, rotate it. See Rotating self-hosted at-rest keys.

Production settings

Set these two values on the app host:

# /etc/systemd/system/codeherder.service.d/production.conf
[Service]
Environment=CH_ENV=production
Environment=CH_REQUIRE_STRONG_AT_REST_KEY=1

The repository ships this file as the production.conf drop-in of the API host. Then run systemctl daemon-reload and restart the server.

With these values, the server enforces these rules at boot:

CH_WEBHOOK_SECRET_KEY and CH_INTEGRATIONS_SECRET_KEY are 64 hex characters.
CH_OTP_HMAC_KEY and CH_INTEGRATIONS_OAUTH_STATE_KEY are at least 32 bytes.
The server never invents a key that is missing.
The database connection uses TLS. See Self-hosted database encryption and TLS.
A setting that weakens security, such as CH_DEV_ORIGINS, makes the server refuse to start. Leave it unset.

A deployment that runs a 32-character raw key today must rotate that key to the 64-hex form first. Setting the flag alone does not convert it. See Rotating self-hosted at-rest keys.

Who holds the KMS key

The KMS key lives in your AWS account. Your key administrators manage it. These are the principals in key_admin_arns, or the account root if you leave that empty. Only the server role may use it for encrypt and decrypt. CodeHerder staff have no access. See Self-hosted KMS keys and attachments bucket.

Where the key material comes from

The Terraform module makes a key with the origin AWS_KMS. KMS makes the key material inside its hardware security modules. Nobody can export it. This holds for you and for CodeHerder.

The server never holds the key material either. For each secret it stores a wrapped data key, not key material (Cipher.Encrypt in internal/variables/cipher.go). So there is no copy of the KMS key to escrow. Protect the key from deletion and plan recovery instead.

Do not import your own key material unless your policy requires it. If you do, you own the backup of that material.

Protect the key from deletion

KMS has no deletion-protection flag. Use these controls:

A deletion window. KMS waits 30 days before it deletes a key. The module sets deletion_window_in_days to 30.
Named administrators. Allow kms:ScheduleKeyDeletion and kms:DisableKey only to key_admin_arns. Add a service control policy if your organization uses them.
An alarm. Alert on ScheduleKeyDeletion and DisableKey events in CloudTrail.
A cancel step. Inside the window, run aws kms cancel-key-deletion --key-id <key>, then enable the key again.

Recover

The supported strategy is to keep the key alive, not to restore it. You cannot export or re-import the key material, so a deleted key is gone.

Event	What to do
Key disabled	Enable it again. No data is lost.
Key pending deletion	Cancel the deletion inside the window.
Key deleted	Every secret stored with `ch variable` is lost. Create a new key, set `CH_KMS_KEY_ID`, and enter each secret again. The four env keys, the database and the attachments are not affected.
Region lost	Use a replica of a multi-Region key. You choose this when you create the key. See below.

To allow a replica, set multi_region = true in the module when you create the key. AWS cannot change this later. Then follow Keep recovery copies in a second region.

After a key rotation, keep the old key until every backup that needs it has expired. See Rotate the KMS key.

Last updated 3 October 2026