Verify a self-hosted host

You run your own host. Use this page to prove it still matches the reference in Hardening a self-hosted server. Each section gives an expected value and a command. One script runs most checks at once.

Run the host check

Save this script on the app host as /usr/local/sbin/verify-host.sh. Run it as root. Pass your public host name for the certificate check.

sudo bash /usr/local/sbin/verify-host.sh --host codeherder.example.com

The script prints PASS, FAIL or SKIP for each check. It exits 1 when any check fails. A SKIP names the command to run by hand. Run the script after each host change and on a schedule. Alarm on exit code 1, as you do for synthetic checks.

#!/usr/bin/env bash
# verify-host.sh - check a self-hosted CodeHerder host against the reference.
# Prints PASS, FAIL or SKIP per check. Exits 1 on any FAIL. Run as root.
# Usage: verify-host.sh [--host <public host>] [--unit <name>]
set -u
unit=codeherder host="" root="" secret_owner=root state_owner=codeherder fails=0
while [ $# -gt 0 ]; do
  case $1 in
    --host) host=$2 ;; --unit) unit=$2 ;; --root) root=$2 ;;
    --secret-owner) secret_owner=$2 ;; --state-owner) state_owner=$2 ;;
    *) echo "unknown flag: $1" >&2; exit 2 ;;
  esac
  shift 2 || { echo "flag needs a value" >&2; exit 2; }
done
have() { command -v "$1" >/dev/null 2>&1; }
pass() { echo "PASS $1 $2"; }
skip() { echo "SKIP $1 $2"; }
fail() { echo "FAIL $1 $2"; fails=$((fails + 1)); }
quiet() { "$@" >/dev/null 2>&1; }
# verdict <id> <pass detail> <fail detail> <command...>: PASS when the command succeeds.
verdict() {
  local id=$1 p=$2 f=$3; shift 3
  if "$@"; then pass "$id" "$p"; else fail "$id" "$f"; fi
}
show() { systemctl show -p "$1" "$unit" 2>/dev/null; }
# prop_ok <id> <Prop=value>... : every pair must appear in systemctl show.
prop_ok() {
  local id=$1 bad="" p; shift
  for p in "$@"; do show "${p%%=*}" | grep -qxF "$p" || bad="$bad $p"; done
  verdict "$id" "$*" "missing:$bad" [ -z "$bad" ]
}
# file_ok <id> <path> "<modes>" <owner>
file_ok() {
  local out; out=$(stat -c '%a %U' "$root$2" 2>/dev/null) || { fail "$1" "$2 missing"; return; }
  case " $3 " in *" ${out% *} "*) ;; *) fail "$1" "$2 mode ${out% *}, want $3"; return ;; esac
  verdict "$1" "$2 $out" "$2 owner ${out#* }, want $4" [ "${out#* }" = "$4" ]
}

verdict unit.active "$unit" "$unit is not active" [ "$(systemctl is-active "$unit" 2>/dev/null)" = active ]
d=$(show DropInPaths); bad=""
for f in 10-production 15-origins 20-secrets 25-mail 30-identity 40-capacity; do
  case $d in *"$f.conf"*) ;; *) bad="$bad $f.conf" ;; esac
done
verdict unit.dropins "all six drop-ins loaded" "missing:$bad" [ -z "$bad" ]
prop_ok unit.sandbox ProtectSystem=strict ProtectHome=yes PrivateTmp=yes NoNewPrivileges=yes \
  PrivateDevices=yes CapabilityBoundingSet= User=codeherder UMask=0027
e=$(show Environment); bad=""
for p in CH_ENV=production CH_REQUIRE_STRONG_AT_REST_KEY=true; do
  case " ${e#Environment=} " in *" $p "*|*" \"$p\" "*) ;; *) bad="$bad $p" ;; esac
done
verdict unit.env "CH_ENV and strong-key rule set" "missing:$bad" [ -z "$bad" ]
if have systemd-analyze; then
  s=$(systemd-analyze security "$unit" --no-pager 2>/dev/null | sed -n 's/.*Overall exposure level for [^:]*: \([0-9.]*\).*/\1/p')
  if [ -z "$s" ]; then fail unit.score "no score"
  elif awk "BEGIN{exit !($s <= 2.0)}"; then pass unit.score "$s"; else fail unit.score "$s is above 2.0"; fi
else skip unit.score "systemd-analyze not found"; fi
prop_ok core.limit LimitCORE=0

file_ok files.secrets /etc/codeherder/secrets.env 600 "$secret_owner"
file_ok files.secrets /etc/codeherder/db.env 600 "$secret_owner"
file_ok files.secrets /etc/codeherder "700 750" "$secret_owner"
out=$(stat -c '%A %U' "$root/usr/local/bin/codeherder" 2>/dev/null) || out=""
case $out in
  "") fail files.binary "/usr/local/bin/codeherder missing" ;;
  ?????-??-*" $secret_owner") pass files.binary "$out" ;;
  *) fail files.binary "$out: want owner $secret_owner, no group or other write" ;;
esac
file_ok files.state /var/lib/codeherder "750 700" "$state_owner"

if [ -r "$root/etc/caddy/Caddyfile" ]; then
  if grep -v '^[[:space:]]*#' "$root/etc/caddy/Caddyfile" | grep -q trusted_proxies; then
    fail proxy.trusted_proxies "Caddyfile sets trusted_proxies"
  else pass proxy.trusted_proxies "not set"; fi
else skip proxy.trusted_proxies "no Caddyfile at /etc/caddy/Caddyfile"; fi
l=$(ss -ltnH 2>/dev/null | awk '$4 ~ /:7878$/ {print $4}')
if [ -z "$l" ]; then fail proxy.loopback "nothing listens on 7878"
elif echo "$l" | grep -qvE '^(127\.0\.0\.1|\[::1\]):7878$'; then fail proxy.loopback "7878 is not loopback only: $l"
else pass proxy.loopback "$l"; fi

ok=0; [ "$(timedatectl show -p NTPSynchronized --value 2>/dev/null)" = yes ] && ok=1
off=""
if have chronyc; then
  off=$(chronyc tracking 2>/dev/null | awk '/^System time/ {print $4}')
  awk "BEGIN{exit !(\"$off\" != \"\" && $off < 1)}" 2>/dev/null || ok=0
fi
verdict clock.ntp "synchronized${off:+, offset ${off}s}" "not synchronized or offset over 1 s${off:+ (${off}s)}" [ "$ok" = 1 ]

id="" ver=""
if [ -r "$root/etc/os-release" ]; then
  id=$(sed -n 's/^ID=//p' "$root/etc/os-release" | tr -d '"')
  ver=$(sed -n 's/^VERSION_ID=//p' "$root/etc/os-release" | tr -d '"')
fi
maj=${ver%%.*} sd=$(systemctl --version 2>/dev/null | awk 'NR==1 {print $2}')
case $id in
  amzn) min=2023 ;; debian) min=12 ;; ubuntu) min=24 ;; rhel|rocky|almalinux|centos) min=9 ;; *) min="" ;;
esac
if [ -n "$min" ] && [ "${maj:-0}" -ge "$min" ] 2>/dev/null && [ "${sd:-0}" -ge 252 ] 2>/dev/null; then
  pass os.release "$id $ver, systemd $sd"
else fail os.release "$id $ver, systemd ${sd:-?}: want AL2023, Debian 12, Ubuntu 24.04 or RHEL 9 and systemd 252 or later"; fi

if [ -e "$root/run/reboot-required" ]; then
  verdict os.reboot "reboot pending for under 7 days" "reboot pending for more than 7 days" \
    [ -z "$(find "$root/run/reboot-required" -mtime +7 2>/dev/null)" ]
elif have needs-restarting; then
  verdict os.reboot "no reboot pending" "reboot pending" quiet needs-restarting -r
elif [ "$id" = debian ] || [ "$id" = ubuntu ]; then pass os.reboot "no reboot pending"
else skip os.reboot "no reboot check for this OS"; fi

en() { [ "$(systemctl is-enabled "$1" 2>/dev/null)" = enabled ]; }
if en dnf-automatic.timer || en dnf-automatic-install.timer; then pass os.autoupdate "dnf-automatic enabled"
elif en apt-daily-upgrade.timer && have unattended-upgrade; then pass os.autoupdate "unattended-upgrades enabled"
else fail os.autoupdate "no automatic update timer enabled"; fi

if [ -z "$host" ]; then skip tls.expiry "pass --host <public host>"
elif echo | openssl s_client -connect "$host:443" -servername "$host" 2>/dev/null \
  | openssl x509 -noout -checkend 1814400 >/dev/null 2>&1; then pass tls.expiry "$host valid for 21 days or more"
else fail tls.expiry "$host: no certificate or it expires within 21 days"; fi

src=$(df --output=source "$root/var/lib/codeherder" 2>/dev/null | tail -n 1)
if [ -n "$src" ] && lsblk -s -no TYPE "$src" 2>/dev/null | grep -qx crypt; then pass disk.encryption "$src is on a crypt device"
else skip disk.encryption "not visible to the guest; on EC2 run: aws ec2 describe-volumes --query 'Volumes[].Encrypted'"; fi

[ $fails -eq 0 ]

Units and drop-ins

The reference unit and its six drop-ins must be the ones that run. Replace codeherder with your unit name if you changed it.

Check	Expected value	Command
Unit state	active	systemctl is-active codeherder
Drop-ins loaded	10-production.conf, 15-origins.conf, 20-secrets.conf, 25-mail.conf, 30-identity.conf, 40-capacity.conf	systemctl show -p DropInPaths codeherder
Sandbox	ProtectSystem=strict, ProtectHome=yes, NoNewPrivileges=yes, PrivateDevices=yes, User=codeherder, UMask=0027	systemctl show -p ProtectSystem -p ProtectHome -p NoNewPrivileges -p PrivateDevices -p User -p UMask codeherder
Production mode	CH_ENV=production and CH_REQUIRE_STRONG_AT_REST_KEY=true	systemctl show -p Environment codeherder
Exposure score	2.0 or lower. The reference unit scores 1.6.	systemd-analyze security codeherder

Compare each installed file with the copy in Hardening a self-hosted server. Save the guide’s copy as reference.service, then run diff. No output means no drift.

systemctl cat codeherder
diff /etc/systemd/system/codeherder.service reference.service
diff -r /etc/systemd/system/codeherder.service.d reference.service.d

A difference in a commented Cognito line is normal. Review any other difference.

File permissions

The unit runs as codeherder. Secrets stay readable by root only.

Path	Owner	Mode
/etc/codeherder	root	700 or 750
/etc/codeherder/secrets.env	root	600
/etc/codeherder/db.env	root	600
/usr/local/bin/codeherder	root	755, not writable by group or other
/var/lib/codeherder	codeherder	750 or 700

sudo stat -c '%a %U %n' /etc/codeherder /etc/codeherder/secrets.env /etc/codeherder/db.env /usr/local/bin/codeherder /var/lib/codeherder

Reverse proxy

The proxy must not set trusted_proxies. The server must listen on loopback only. See the contract at the top of the reference Caddyfile.

Check	Expected value	Command
Config is valid	exit code 0	caddy validate --config /etc/caddy/Caddyfile
No trusted_proxies	no output	grep -v '^[[:space:]]*#' /etc/caddy/Caddyfile | grep trusted_proxies
Loopback only	one line, 127.0.0.1:7878	ss -ltn | grep 7878

Check the forwarded address by hand. Send curl -H 'X-Forwarded-For: 198.51.100.1' https://<api>/v1/.... Read the access log or the audit IP. It must show your real address, not 198.51.100.1. If it shows 198.51.100.1, fix CH_TRUSTED_PROXY_HOPS.

Supported operating systems and updates

The reference unit needs systemd 252 or later. CodeHerder tested the reference on this floor. It is not a vendor certification.

Host	Supported versions
App host	x86_64 or arm64 Linux with systemd 252 or later: Amazon Linux 2023, Debian 12, Ubuntu 24.04, RHEL 9
Not supported	Amazon Linux 2, RHEL 8, Ubuntu 22.04 (systemd older than 252)
Device	The ch targets in the release, on macOS 12 or later or a Linux kernel your vendor supports. Docker mode needs Docker Engine.

Enable automatic OS updates on the app host. Use dnf-automatic with apply_updates = yes on Amazon Linux and RHEL. Use unattended-upgrades on Debian and Ubuntu. Reboot within 7 days of a kernel or systemd update.

cat /etc/os-release
systemctl --version | head -n 1
systemctl is-enabled dnf-automatic.timer || systemctl is-enabled apt-daily-upgrade.timer
needs-restarting -r; ls -l /run/reboot-required

After each reboot, run these checks. Every one must pass.

systemctl is-active codeherder
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:7878/v1/health
sudo bash /usr/local/sbin/verify-host.sh --host codeherder.example.com
ch instance check

Health returns 200. For a server upgrade, follow Updating.

After an OS update or a device replacement, check each device. ch device list must show the device online. In ch device show <ref>, the checks tunnel_connected, version_current and clock_skew must read ok. In Docker mode, docker_ready must read ok too. See Launch a device on AWS and Running the device server.

Disk encryption

The app host disk and the database must be encrypted at rest. A guest cannot see EBS encryption, so check it in AWS.

Check	Expected value	Command
EBS default	true	aws ec2 get-ebs-encryption-by-default --query EbsEncryptionByDefault
Host volumes	every value true	aws ec2 describe-volumes --filters Name=attachment.instance-id,Values=<instance id> --query 'Volumes[].Encrypted'
RDS	every value true	aws rds describe-db-instances --query 'DBInstances[].StorageEncrypted'
Other hosts	a crypt line under the data mount	lsblk -s -no TYPE $(df --output=source /var/lib/codeherder | tail -n 1)

Device disk encryption is your choice, recorded per device. See Self-hosted responsibilities.

Core dumps

A core dump holds the four at-rest keys and the database credentials. The reference unit sets LimitCORE=0, so the server writes none.

Check	Expected value	Command
Unit limit	LimitCORE=0	systemctl show -p LimitCORE codeherder
Running process	Max core file size 0 0	grep 'core file' /proc/$(systemctl show -p MainPID --value codeherder)/limits
No dumps stored	empty list	coredumpctl list codeherder

To stop core dumps for every service, create /etc/systemd/coredump.conf.d/10-none.conf with [Coredump] and Storage=none.

Certificates and domains

Check every public endpoint: the server host, the Cognito custom domain, and an OTLP or SIEM collector if it is public. Each certificate must have 21 days or more left. Caddy renews at 30 days left, so under 21 days means renewal failed.

echo | openssl s_client -connect codeherder.example.com:443 -servername codeherder.example.com 2>/dev/null \
  | openssl x509 -noout -enddate -checkend 1814400

-checkend prints Certificate will not expire and exits 0 when the certificate is good. Repeat the command for each endpoint.

Check each domain registration too. In Route 53, AutoRenew must be true. Elsewhere, read the expiry from whois.

aws route53domains get-domain-detail --region us-east-1 --domain-name example.com --query '[AutoRenew,ExpirationDate]'
whois example.com | grep -i expir

CodeHerder also emails workspace admins before an SSO credential expires. See Self-hosted deployment.

Clock sync

A skewed clock breaks token checks and orders audit events wrongly. Check the app host, each audit receiver and each device.

Host	Expected value	Command
API host	NTPSynchronized=yes	timedatectl show -p NTPSynchronized
API host and audit receivers	offset under 1 second	chronyc tracking (read System time)
Device	clock_skew reads ok (skew under 10 seconds)	ch device show <ref>

On AWS, use Amazon Time Sync. Amazon Linux 2023 runs it through chrony by default. Audit receivers are the OTLP collector and the SIEM forwarder. See Alarm on trace export failures.

Related guides

• Hardening a self-hosted server — the reference unit, drop-ins and Caddyfile
• Self-hosted synthetic checks — the scheduled ch instance check
• Self-hosted responsibilities and launch acceptance — who owns each check
• Self-hosted deployment — install and upgrade the server