Hardening a self-hosted server

The production settings, systemd unit, drop-ins and reverse-proxy config to run a self-hosted CodeHerder server hardened, with a way to check the result.

Use this page after you install a self-hosted server from Self-hosted deployment. It lists every security setting with its required production value. It also gives a hardened systemd unit, drop-ins and a reverse-proxy config to copy.

Set these first

Set CH_ENV=production before the server takes real traffic. The server then checks its own settings at boot and refuses to start on a weak one. A setting the server refuses appears in the second table.

Variable	Required value	Why
`CH_ENV`	`production`	turns on the boot checks below, at-rest encryption, strong keys and a TLS database DSN
`CH_REQUIRE_STRONG_AT_REST_KEY`	`true`	requires 256-bit keys for the four at-rest key settings
`CH_SIGNUP_DISABLED`	`true`	closes self-service signup
`CH_SSO_ALLOWED_EMAIL_DOMAINS`	your email domains, never `*`	limits which identities can sign in

With CH_ENV=production, boot refuses a weak value of each knob below:

Name	Kind	Required production value	Means
`CH_ALLOW_INTERNAL_GIT_HOSTS`	Server + device	false	allow internal/loopback/metadata git remote hosts (air-gapped / internal-mirror setups)
`CH_ALLOW_PLAINTEXT_SECRETS`	Server	false	local-dev escape hatch: opt out of at-rest secret encryption (never overrides a production signal)
`CH_ALLOW_UNISOLATED_PLANNING`	Device	false	allow a non-writable planning stage to run without its own isolated worktree
`CH_ANALYTICS_STATEMENT_TIMEOUT_MS`	Server	greater than zero	server-side statement_timeout on the analytics pool: a scan past it is cancelled by PostgreSQL and fails; 0 disables the bound outside production; production requires a value above 0
`CH_APP_BASE_URL`	Server	an https:// URL	the server’s own public SPA base URL; required unless the listen address is loopback, and an https:// URL under CH_ENV=production
`CH_BASE_URL`	Server	an https:// URL	the server’s own public API base URL; required unless the listen address is loopback, and an https:// URL under CH_ENV=production
`CH_DB`	Server	TLS DSN for a non-local host, not the compose default password	a postgres://\|postgresql:// DSN — the only backend a server binary can open (opens the pgx backend and applies pending migrations before returning); a file-path value is refused at startup with a fatal, helpful error
`CH_DEV_ORIGINS`	Server	false	admit localhost/dev origins for CORS and WebSocket upgrades
`CH_MIGRATE_DB`	Server	TLS DSN for a non-local host, not the compose default password	optional postgres:// DSN of the table-owner role; when set, startup migrations run as that role and CH_DB can be a role with no UPDATE or DELETE on events (docs/database-roles.md); unset migrates on CH_DB
`CH_PPROF_ADDR`	Server	unset, or a loopback host:port	loopback address to expose net/http/pprof diagnostics on; unset disables it. Never bind non-loopback.
`CH_READ_STATEMENT_TIMEOUT_MS`	Server	greater than zero	server-side statement_timeout on the reader pool: a query past it is cancelled by PostgreSQL and fails; 0 disables the bound outside production; production requires a value above 0
`CH_REQUEST_TIMEOUT_SECONDS`	Server	greater than zero	per-request handler timeout; 0 disables it
`CH_RETENTION_DB`	Server	TLS DSN for a non-local host, not the compose default password	optional postgres:// DSN of the retention role, the only role that deletes events rows (docs/database-roles.md); unset prunes events on CH_DB
`CH_STAGE_ALLOW_ROOT_IMAGE`	Device	false	allow a –docker-executor stage whose custom image’s own default user is root (uid 0) to run as root against the stage’s writable host bind mounts; default off refuses such a spawn
`CH_TRUSTED_EXECUTION`	Device	false	operator consent to run without a hostile-agent boundary: required to start a –docker-executor with –stage-egress=false or CH_STAGE_ALLOW_ROOT_IMAGE, required to start the privileged –docker launcher, and admits bypassed-approval spawns under CH_REQUIRE_ISOLATED_EXECUTION on process, launcher or agent-user-only devices; never serves git credentials
`CH_WRITE_STATEMENT_TIMEOUT_MS`	Server	greater than zero	server-side statement_timeout on the writer pool: a statement past it is cancelled by PostgreSQL and fails; migrations are exempt; 0 disables the bound outside production; production requires a value above 0
`EGRESS_PROXY_EXTERNAL`	Server + device	false	this process sits behind the hard git-egress sidecar proxy, so its own DNS/IP resolve check is redundant

Boot also logs the effective value of each of these settings. It hides secrets in that log.

Create the service user

The unit runs as user codeherder. Create that user and give it one writable directory:

sudo useradd --system --home-dir /var/lib/codeherder --shell /usr/sbin/nologin codeherder
sudo install -d -o codeherder -g codeherder -m 0750 /var/lib/codeherder
sudo install -d -m 0755 /etc/codeherder

The systemd unit

Install this file as /etc/systemd/system/codeherder.service. It confines the server to one writable directory. It removes every Linux capability and denies privileged system calls, kernel access, extra address families and executable memory mappings.

# CodeHerder server — reference systemd unit for a self-hosted install.
# Install as /etc/systemd/system/codeherder.service. See README.md.

[Unit]
Description=CodeHerder server
After=network-online.target postgresql.service
Wants=network-online.target

[Service]
Type=simple
User=codeherder
Group=codeherder

# No -db flag: CH_DB comes from /etc/codeherder/db.env (20-secrets.conf).
# Binds loopback only. The reverse proxy terminates TLS on :443.
ExecStart=/usr/local/bin/codeherder -addr 127.0.0.1:7878

Restart=on-failure
RestartSec=2s
StandardOutput=journal
StandardError=journal

# Filesystem confinement. ReadWritePaths is the one writable hole.
ProtectSystem=strict
ReadWritePaths=/var/lib/codeherder
ProtectHome=true
PrivateTmp=true
NoNewPrivileges=true

# No core dumps: a dump holds the four at-rest keys and DB credentials.
LimitCORE=0

# Kernel and privilege restrictions. `systemd-analyze security` scores the set
# below at 1.6 OK (scripts/check-systemd-units.sh fails above 2.0).
# - No @resources denial: the Go runtime raises RLIMIT_NOFILE at start.
# - AF_UNIX stays for sd_notify; AF_NETLINK stays for Go interface lookups.
CapabilityBoundingSet=
AmbientCapabilities=
PrivateDevices=true
ProtectKernelTunables=true
ProtectKernelModules=true
ProtectKernelLogs=true
ProtectControlGroups=true
ProtectClock=true
ProtectHostname=true
ProtectProc=invisible
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK
RestrictNamespaces=true
RestrictRealtime=true
RestrictSUIDSGID=true
LockPersonality=true
MemoryDenyWriteExecute=true
SystemCallArchitectures=native
SystemCallFilter=@system-service
SystemCallFilter=~@privileged
SystemCallErrorNumber=EPERM
RemoveIPC=true
UMask=0027
KeyringMode=private

[Install]
WantedBy=multi-user.target

The drop-ins

Install these files in /etc/systemd/system/codeherder.service.d/. Each one holds one concern, so you can change it without touching the unit.

The production drop-in sets CH_ENV and the strong-key setting.

# Production mode. The server refuses to start under CH_ENV=production when a
# security knob has a weak value. See the hardening profile.
[Service]
Environment=CH_ENV=production
Environment=CH_REQUIRE_STRONG_AT_REST_KEY=true

The secrets drop-in reads two files that only root can read. Create /etc/codeherder/secrets.env and /etc/codeherder/db.env with mode 0600. The database address must use TLS.

The origins drop-in sets the three public origins. Production refuses to start unless CH_BASE_URL and CH_APP_BASE_URL are https:// URLs. The server does not check CH_PREVIEW_ORIGIN. You own it.

# Public origins. Replace the example hosts with your own.
# Production refuses to start unless CH_BASE_URL and CH_APP_BASE_URL are
# https:// URLs. The server mails links built from them.
[Service]
Environment=CH_BASE_URL=https://codeherder.example.com
Environment=CH_APP_BASE_URL=https://app.example.com
# The preview must be its own origin, never the app origin. The server does not
# check this value. You own it. Keep it equal to previewOrigin in the web app's
# config.js and to the host of the preview Caddyfile block.
Environment=CH_PREVIEW_ORIGIN=https://preview.example.net

# Secrets stay out of unit files. Create both files as root, mode 0600.
#   /etc/codeherder/secrets.env  CH_WEBHOOK_SECRET_KEY, CH_INTEGRATIONS_SECRET_KEY,
#                                CH_OTP_HMAC_KEY, CH_INTEGRATIONS_OAUTH_STATE_KEY
#   /etc/codeherder/db.env       CH_DB=<a postgres:// DSN with TLS>
# The licence file is owner root, group codeherder, mode 0640.
[Service]
EnvironmentFile=/etc/codeherder/secrets.env
EnvironmentFile=/etc/codeherder/db.env
Environment=CH_LICENSE_FILE=/etc/codeherder/license.token

The identity drop-in closes self-signup. Fill in your Cognito values if you use Cognito.

The mail drop-in selects real delivery through Amazon SES. Production refuses the fake mailer.

# Real mail delivery. Production refuses the fake mailer and refuses to start
# when SES fails to initialise. SES credentials come from the instance role or
# from /etc/codeherder/secrets.env. The From domain must be a verified SES
# identity with SPF and DKIM set up. You own that setup.
[Service]
Environment=CH_MAILER=ses
Environment="CH_MAILER_FROM=CodeHerder <no-reply@example.com>"
Environment=CH_AWS_REGION=us-east-1

# Identity. Close self-signup and sign in through your identity provider.
[Service]
Environment=CH_SIGNUP_DISABLED=true
# Cognito. Replace the placeholders with your own user pool.
#Environment="CH_COGNITO_USER_POOL_ID=us-east-1_EXAMPLE"
#Environment="CH_COGNITO_REGION=us-east-1"
#Environment="CH_COGNITO_DOMAIN=login.example.com"
#Environment="CH_COGNITO_SPA_CLIENT_ID=EXAMPLE"
#Environment="CH_COGNITO_CLI_CLIENT_ID=EXAMPLE"
#Environment="CH_COGNITO_MCP_CLIENT_ID=EXAMPLE"
# Limit sign-in by email domain. Never set this to *.
#Environment="CH_SSO_ALLOWED_EMAIL_DOMAINS=example.com"
# Instance operators run the instance checks, the support bundle and the
# restore-fence commands. List their verified email addresses.
#Environment="CH_INSTANCE_OPERATORS=ops@example.com"

The capacity drop-in sets two limits. Tune them for your fleet.

# Capacity limits. These values are the hosted defaults. Tune them for your fleet.
[Service]
Environment=CH_MAX_DEVICE_SESSIONS=50
Environment=CH_SESSION_CONCURRENCY_TARGET=16

Then reload systemd and start the server:

sudo systemctl daemon-reload
sudo systemctl enable --now codeherder

The reverse proxy

The server trusts the X-Forwarded-For header from its proxy. It reads the entry its trusted proxy wrote, counted from the right (CH_TRUSTED_PROXY_HOPS, default 1). The proxy must drop any client-supplied X-Forwarded-For and set the real client address. Caddy does this by default. This holds only while the config sets no trusted_proxies. Never add it.

The server serves the API only. This config also serves the web app and the prototype preview as two more hosts, and it sets their security headers. The server does not set them. You own them. Build the web app and the preview with cd app && npm ci && npm run build && VITE_SPA_ORIGIN=https://app.example.com npm run build:preview. Set VITE_SPA_ORIGIN to your web app origin: the viewer takes it at build time, and stays inert without it. Copy app/dist to /srv/codeherder/app and app/preview-dist to /srv/codeherder/preview. Write /srv/codeherder/app/config.js so the web app finds the API and the preview origin:

window.__CH_CONFIG__ = {
  apiBase: "https://codeherder.example.com",
  previewOrigin: "https://preview.example.net",
};

This config also redacts credentials from the access log. See Self-hosted deployment for what it redacts.

# Reference Caddyfile for a self-hosted CodeHerder server.
# Replace the three example hosts with your own. Install as /etc/caddy/Caddyfile.
#
# THREE HOSTS: the API (codeherder.example.com), the web app (app.example.com)
# and the prototype preview (preview.example.net). The preview MUST be its own
# origin, never the app origin. The server serves the API only. Every header on
# the app and preview blocks is YOURS to install. The values equal the
# generated contract in app/src/config/csp.ts and app/src/config/previewCsp.ts.
# A test holds them equal (app/src/config/selfhostCaddyfile.test.ts).
#
# THE XFF CONTRACT: the server trusts X-Forwarded-For and reads the entry its
# trusted proxy wrote (CH_TRUSTED_PROXY_HOPS, default 1). The proxy must drop any client-supplied X-Forwarded-For and set the
# real client address. Caddy does this by default: it ignores inbound
# X-Forwarded-* headers. That holds only while no `trusted_proxies` is set.
# NEVER add `trusted_proxies` here. Doing so makes Caddy honour a forged
# X-Forwarded-For and defeats per-IP rate limits and audit IP attribution.

(ch_access_log) {
	output stdout
	format filter {
		wrap {args[0]}
		fields {
			request>headers>Authorization delete
			request>headers>Proxy-Authorization delete
			request>headers>Cookie delete
			request>headers>Sec-Websocket-Protocol delete
			request>headers>X-Ch-Edge-Secret delete
			request>headers>X-Api-Key delete
			request>headers>Referer regexp "\?.*" "?REDACTED"
			request>uri regexp "\?.*" "?REDACTED"
			resp_headers>Set-Cookie delete
			resp_headers>Sec-Websocket-Protocol delete
			resp_headers>Location regexp "\?.*" "?REDACTED"
		}
	}
}

codeherder.example.com {
	reverse_proxy 127.0.0.1:7878 {
		transport http {
			read_timeout 120s
		}
	}
	log {
		import ch_access_log console
	}
}

app.example.com {
	root * /srv/codeherder/app
	try_files {path} /index.html
	file_server
	header {
		Content-Security-Policy "default-src 'self'; script-src 'self'; style-src 'self' 'unsafe-inline'; img-src 'self' data: blob:; media-src 'self' blob:; frame-src 'self' blob: https://preview.example.net; font-src 'self' data:; connect-src 'self' https://codeherder.example.com wss://codeherder.example.com; object-src 'none'; base-uri 'self'; form-action 'self'; frame-ancestors 'none'"
		Strict-Transport-Security "max-age=31536000; includeSubDomains"
		X-Content-Type-Options "nosniff"
		Referrer-Policy "strict-origin-when-cross-origin"
		-Server
	}
	log {
		import ch_access_log console
	}
}

preview.example.net {
	root * /srv/codeherder/preview
	file_server
	header {
		Content-Security-Policy "default-src 'none'; script-src https://preview.example.net 'unsafe-inline' 'unsafe-eval'; style-src https://preview.example.net 'unsafe-inline'; img-src data: blob:; font-src https://preview.example.net data:; connect-src 'none'; form-action 'none'; frame-ancestors https://app.example.com; base-uri 'none'"
		Cross-Origin-Resource-Policy "cross-origin"
		# The sandboxed prototype frame has an opaque origin. It loads the kit
		# fonts in CORS mode, so the preview must allow any origin. It serves
		# public static files only.
		Access-Control-Allow-Origin "*"
		X-Robots-Tag "noindex, nofollow"
		Strict-Transport-Security "max-age=31536000; includeSubDomains"
		X-Content-Type-Options "nosniff"
		-Server
	}
	log {
		import ch_access_log console
	}
}

Check the result

Score the unit file. A score of 2.0 or lower is the target. The reference unit scores 1.6. Offline scoring needs systemd 250 or later (RHEL 8 ships 239).

systemd-analyze security --offline=true /etc/systemd/system/codeherder.service

On a running host, boot the server under the unit and check the health route:

sudo systemctl status codeherder
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:7878/v1/health

Health returns 200. Check the security headers on all three hosts. The server does not set the web app or preview headers, so a missing header means your proxy is wrong:

curl -sI https://codeherder.example.com/v1/health
curl -sI https://app.example.com/
curl -sI https://preview.example.net/

The web app response needs Content-Security-Policy, Strict-Transport-Security and X-Content-Type-Options. The preview response also needs Cross-Origin-Resource-Policy: cross-origin and Access-Control-Allow-Origin: *. The sandboxed prototype frame has no origin of its own, so it loads the kit fonts in CORS mode. Then read the journal. A line with SIGSYS, operation not permitted or permission denied means the sandbox blocked something the server needs. Report it through a support bundle.

Verify a self-hosted host — check the installed units, file modes, proxy, updates, core dumps, certificates and clocks
Self-hosted deployment — download, run and upgrade the server
Self-hosted logs — log formats and shipping logs off the host
Replacing a compromised self-hosted server — what to do when a host is suspect