Self-hosted KMS keys and attachments bucket

Create the secret-store KMS key and an encrypted attachments bucket in your own AWS account, and set the four settings that connect them to your server.

This page shows how to create the two AWS resources a self-hosted server can use for data protection. The first is a KMS key for the secret store. The second is an S3 bucket for attachments. Both live in your AWS account. CodeHerder staff have no access to them.

CodeHerder provides a Terraform module for each resource, kms and attachments-bucket, in the self-host Terraform bundle. Each module has no vendor backend and names no vendor account. You supply the state backend, the provider and the credentials.

What needs a key or a bucket

Setting	Purpose	If it is unset
`CH_KMS_KEY_ID`	Key id, key ARN or alias of the secret-store key.	The secret store answers `503`.
`CH_KMS_REGION`	Region of that key. Set it when the key is in a different region from `CH_AWS_REGION`.	The server uses `CH_AWS_REGION`. If that is also unset, the server uses `us-west-2`.
`CH_S3_BUCKET`	Name of the attachments bucket.	Attachment uploads answer `503`.
`CH_S3_REGION`	Region of the attachments bucket. Set it to the `bucket_region` output.	The server uses `CH_AWS_REGION`, then `us-west-2`. A bucket in a different region then fails.

At boot, if CH_KMS_KEY_ID is set, the server calls kms:DescribeKey once. If the call fails, the server logs an error and turns the secret store off. It answers 503. It never stores a secret in plaintext. It never crashes.

One key per deployment

A self-hosted server serves one organization. One secret-store key is enough. CodeHerder does not offer a key for each workspace. Each stored secret still gets its own data key, and the encryption context binds each secret to its scope and name.

Create the secret-store key

Add a module block to your own Terraform. Use the path of your bundle copy.

module "codeherder_kms" {
  source = "./terraform/selfhost/kms"

  # The IAM role your server runs as. No default.
  server_principal_arn = "arn:aws:iam::<account>:role/<server-role>"
}

output "ch_kms_key_id" {
  value = module.codeherder_kms.key_arn
}

Set CH_KMS_KEY_ID to the key_arn output. The module makes these choices:

One principal. The key policy grants kms:GenerateDataKey, kms:Encrypt, kms:Decrypt and kms:DescribeKey to server_principal_arn only. The server calls kms:Encrypt only from codeherder reseal. That command re-wraps each data key under a new key after a key rotation.
Administration. The account root administers the key through IAM. Set key_admin_arns to name your own administrators instead.
Rotation on. KMS rotates the key material every year.
Deletion window. The window is 30 days. Set deletion_window_in_days between 7 and 30.
Single Region by default. Set multi_region = true when you create the key if you want a replica in a recovery Region. AWS cannot change this later. See Self-hosted key custody.

KMS is dual control. The server role needs an IAM policy that allows the same four actions on the key. The key policy alone is not enough, and the role policy alone is not enough. Name the role in both.

If you do not use Terraform, create a key with rotation on and this key policy. Replace the placeholders in angle brackets.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "KeyAdministration",
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::<account>:root" },
      "Action": "kms:*",
      "Resource": "*"
    },
    {
      "Sid": "ServerEnvelopeOps",
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::<account>:role/<server-role>" },
      "Action": ["kms:GenerateDataKey", "kms:Encrypt", "kms:Decrypt", "kms:DescribeKey"],
      "Resource": "*"
    }
  ]
}

Create the attachments bucket

Add a second module block.

module "codeherder_attachments" {
  source = "./terraform/selfhost/attachments-bucket"

  bucket_name          = "example-codeherder-attachments"
  log_bucket_name      = "example-codeherder-attachments-logs"
  server_principal_arn = "arn:aws:iam::<account>:role/<server-role>"
  app_origin           = "https://codeherder.example.com"
}

Set CH_S3_BUCKET to the bucket_name output. Set CH_S3_REGION to the bucket_region output. The module applies these settings:

SSE-KMS with its own key. Default encryption uses a new KMS key with rotation on. The module uses a separate key from the secret-store key. A bucket key cuts the number of KMS calls. The server sets no encryption header on an upload, so the bucket default applies. The key policy grants use of the key to the server role only, and only through S3 in the bucket’s region.
Block public access. All four block settings are on. Ownership is BucketOwnerEnforced, so ACLs are off.
TLS only. A bucket policy denies every request that does not use TLS. The statement is below.
Access logging. S3 writes server access logs to the log bucket under the attachments/ prefix. S3 cannot write logs to a bucket that uses SSE-KMS. The log bucket therefore uses SSE-S3 (AES256). It has the same public block and TLS deny. Logs expire after log_retention_days (default 365).
Lifecycle. The bucket aborts incomplete multipart uploads after one day. It also expires old object versions and expired delete markers. It has no expiry for current objects, because that would delete live attachments. The server removes its own pending uploads.
CORS. The web app uploads straight to S3 with a presigned URL. The bucket allows PUT from app_origin only.
Versioning off. The server erases an attachment by deleting its key. With versioning on, S3 keeps the old bytes after a delete. Leave versioning off so an erasure removes the data. If you turn it on, the noncurrent-version rule expires old versions after noncurrent_version_expiration_days (default 30).

The TLS deny policy is:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyInsecureTransport",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::<bucket>",
        "arn:aws:s3:::<bucket>/*"
      ],
      "Condition": { "Bool": { "aws:SecureTransport": "false" } }
    }
  ]
}

Allow the server role to use the bucket

The bucket key policy names the server role. The role’s IAM policy must name the key too. Add this statement to the role, and use the kms_key_arn output.

{
  "Effect": "Allow",
  "Action": ["kms:GenerateDataKey", "kms:Decrypt"],
  "Resource": "<kms_key_arn output>"
}

A presigned upload needs kms:GenerateDataKey. A presigned download and a server read need kms:Decrypt. The role also needs s3:PutObject, s3:GetObject and s3:DeleteObject on the bucket objects. The server never lists the bucket. See Self-hosted infrastructure for the rest of the role policy.

Alert on a decrypt by another principal

The server role is the only principal that should call kms:Decrypt on the secret-store key. The alert fires on any other caller. That includes administrators, the account root and a cross-account caller. It also fires on a denied attempt. The event shows which secret the caller tried to read. The server puts the secret’s scope and name in the encryption context of every call, and CloudTrail records them.

Before you turn the alert on, check three things:

A CloudTrail trail logs read management events in the key’s region.
The trail does not exclude AWS KMS events.
The EventBridge rule lives in the key’s region.

kms:Decrypt is a read-only event. A rule in the ENABLED state never matches it. The rule must use the state ENABLED_WITH_ALL_CLOUDTRAIL_MANAGEMENT_EVENTS. The module sets it for you.

Create an SNS topic yourself. Then set decrypt_alert_topic_arn in the module block:

module "codeherder_kms" {
  source = "./terraform/selfhost/kms"

  server_principal_arn    = "arn:aws:iam::<account>:role/<server-role>"
  decrypt_alert_topic_arn = "arn:aws:sns:<region>:<account>:<topic>"
}

The topic policy must let EventBridge publish:

{
  "Sid": "AllowEventBridgePublish",
  "Effect": "Allow",
  "Principal": { "Service": "events.amazonaws.com" },
  "Action": "sns:Publish",
  "Resource": "arn:aws:sns:<region>:<account>:<topic>"
}

If the topic uses a customer-managed key, that key policy must also let events.amazonaws.com use the key.

If you do not use Terraform, read the decrypt_alert_event_pattern output, or copy the pattern below. Create an EventBridge rule on the default bus with this pattern and the state ENABLED_WITH_ALL_CLOUDTRAIL_MANAGEMENT_EVENTS. A SIEM can use the same match. Replace the placeholders in angle brackets.

{
  "source": ["aws.kms"],
  "detail-type": ["AWS API Call via CloudTrail"],
  "detail": {
    "eventSource": ["kms.amazonaws.com"],
    "eventName": ["Decrypt"],
    "resources": { "ARN": ["<key_arn output>"] },
    "$or": [
      { "userIdentity": { "arn": [{ "anything-but": { "prefix": "arn:aws:sts::<account>:assumed-role/<server-role-name>/" } }] } },
      { "userIdentity": { "arn": [{ "exists": false }] } }
    ]
  }
}

The prefix is the server role’s assumed-role ARN. An assumed-role ARN has no role path. If your role ARN is arn:aws:iam::<account>:role/svc/codeherder-server, use codeherder-server as the role name. The second branch catches a caller with no userIdentity.arn, such as a cross-account caller. anything-but does not match a missing field.

Test the pattern before you rely on it. Save the pattern as pattern.json. Save this event as event.json. It is a decrypt by a different role.

{
  "source": "aws.kms",
  "detail-type": "AWS API Call via CloudTrail",
  "detail": {
    "eventSource": "kms.amazonaws.com",
    "eventName": "Decrypt",
    "userIdentity": {
      "type": "AssumedRole",
      "arn": "arn:aws:sts::<account>:assumed-role/other-role/session"
    },
    "resources": [{ "ARN": "<key_arn output>", "type": "AWS::KMS::Key" }]
  }
}

Run aws events test-event-pattern --event-pattern file://pattern.json --event file://event.json. The answer must be "Result": true. Change the arn to arn:aws:sts::<account>:assumed-role/<server-role-name>/session. The answer must then be false.

Alert on an at-rest key mismatch

At boot the server proves that the configured at-rest keys open the secrets the database already holds. See Back up the database for the keys. A wrong key does not stop a non-production server, or a server started with CH_AT_REST_KEY_CANARY_RESEAL set. The server then reports the fault in two places.

The health endpoint. GET /v1/health needs no token. The data.atRestKeys field holds one of these values:

Value	Meaning
`ok`	Every checked key opens the stored secrets.
`mismatch`	A key does not open the secrets this database holds.
`unreadable_rows`	The keys are proven, but some stored secrets do not open.
`not_checked`	The boot did not run the check. This happens in plaintext development mode.

The value is set at boot. It stays until you restart the server with the correct key. The HTTP status stays 200, so a check on the status alone misses the fault. Alert when the field is not ok. Poll once a minute from a cron job, a systemd timer or an uptime monitor:

curl -fsS https://<your-server>/v1/health | jq -e '.data.atRestKeys == "ok"'

The command exits non-zero when the value is not ok. An uptime monitor with a keyword check can look for "atRestKeys":"ok" and alert when it is missing.

The log. The server writes one of three lines to the codeherder.service journal. Alert on a match for any of them:

at-rest key mismatch
at-rest key proven, but some stored secrets cannot be decrypted
accepting an at-rest key mismatch because CH_AT_REST_KEY_CANARY_RESEAL is set

journalctl -u codeherder.service --grep 'at-rest key (mismatch|proven, but some stored secrets)|accepting an at-rest key mismatch'

Forward the journal to your SIEM and match the same text there. Self-hosted support lists these lines under “Security signals”.

Check that it works

Restart the server. Look for a KMS health-check error in the log. There should be none.
Upload an attachment to a task in the web app. Open it again. An upload that fails with an access error usually means the role policy lacks the bucket key statement.
Write and read back a secret. Use ch variable. A 503 means CH_KMS_KEY_ID is unset or the health check failed. An access-denied error means the key policy or the role policy does not name the role.
Confirm the object uses your key. Run aws s3api head-object on it. The response shows aws:kms and your key ARN.

To protect the keys, see Self-hosted backup and recovery.