# Verify a self-hosted host

Source: https://codeherder.com/docs/self-host-host-verification/

Check a self-hosted server host against the reference. Each check has an expected value and a command, plus one script that prints pass or fail.

You run your own host. Use this page to prove it still matches the reference in [Hardening a self-hosted server](https://codeherder.com/docs/self-host-hardening/). Each section gives an expected value and a command. One script runs most checks at once.

## Run the host check

Save this script on the app host as `/usr/local/sbin/verify-host.sh`. Run it as root. Pass your public host name for the certificate check.

```
sudo bash /usr/local/sbin/verify-host.sh --host codeherder.example.com
```

The script prints `PASS`, `FAIL` or `SKIP` for each check. It exits 1 when any check fails. A `SKIP` names the command to run by hand. Run the script after each host change and on a schedule. Alarm on exit code 1, as you do for [synthetic checks](https://codeherder.com/docs/self-host-checks/).

```
#!/usr/bin/env bash
# verify-host.sh - check a self-hosted CodeHerder host against the reference.
# Prints PASS, FAIL or SKIP per check. Exits 1 on any FAIL. Run as root.
# Usage: verify-host.sh [--host <public host>] [--unit <name>]
set -u
unit=codeherder host="" root="" secret_owner=root state_owner=codeherder fails=0
while [ $# -gt 0 ]; do
  case $1 in
    --host) host=$2 ;; --unit) unit=$2 ;; --root) root=$2 ;;
    --secret-owner) secret_owner=$2 ;; --state-owner) state_owner=$2 ;;
    *) echo "unknown flag: $1" >&2; exit 2 ;;
  esac
  shift 2 || { echo "flag needs a value" >&2; exit 2; }
done
have() { command -v "$1" >/dev/null 2>&1; }
pass() { echo "PASS $1 $2"; }
skip() { echo "SKIP $1 $2"; }
fail() { echo "FAIL $1 $2"; fails=$((fails + 1)); }
quiet() { "$@" >/dev/null 2>&1; }
# verdict <id> <pass detail> <fail detail> <command...>: PASS when the command succeeds.
verdict() {
  local id=$1 p=$2 f=$3; shift 3
  if "$@"; then pass "$id" "$p"; else fail "$id" "$f"; fi
}
show() { systemctl show -p "$1" "$unit" 2>/dev/null; }
# prop_ok <id> <Prop=value>... : every pair must appear in systemctl show.
prop_ok() {
  local id=$1 bad="" p; shift
  for p in "$@"; do show "${p%%=*}" | grep -qxF "$p" || bad="$bad $p"; done
  verdict "$id" "$*" "missing:$bad" [ -z "$bad" ]
}
# file_ok <id> <path> "<modes>" <owner>
file_ok() {
  local out; out=$(stat -c '%a %U' "$root$2" 2>/dev/null) || { fail "$1" "$2 missing"; return; }
  case " $3 " in *" ${out% *} "*) ;; *) fail "$1" "$2 mode ${out% *}, want $3"; return ;; esac
  verdict "$1" "$2 $out" "$2 owner ${out#* }, want $4" [ "${out#* }" = "$4" ]
}

verdict unit.active "$unit" "$unit is not active" [ "$(systemctl is-active "$unit" 2>/dev/null)" = active ]
d=$(show DropInPaths); bad=""
for f in 10-production 15-origins 20-secrets 25-mail 30-identity 40-capacity; do
  case $d in *"$f.conf"*) ;; *) bad="$bad $f.conf" ;; esac
done
verdict unit.dropins "all six drop-ins loaded" "missing:$bad" [ -z "$bad" ]
prop_ok unit.sandbox ProtectSystem=strict ProtectHome=yes PrivateTmp=yes NoNewPrivileges=yes \
  PrivateDevices=yes CapabilityBoundingSet= User=codeherder UMask=0027
e=$(show Environment); bad=""
for p in CH_ENV=production CH_REQUIRE_STRONG_AT_REST_KEY=true; do
  case " ${e#Environment=} " in *" $p "*|*" \"$p\" "*) ;; *) bad="$bad $p" ;; esac
done
verdict unit.env "CH_ENV and strong-key rule set" "missing:$bad" [ -z "$bad" ]
if have systemd-analyze; then
  s=$(systemd-analyze security "$unit" --no-pager 2>/dev/null | sed -n 's/.*Overall exposure level for [^:]*: \([0-9.]*\).*/\1/p')
  if [ -z "$s" ]; then fail unit.score "no score"
  elif awk "BEGIN{exit !($s <= 2.0)}"; then pass unit.score "$s"; else fail unit.score "$s is above 2.0"; fi
else skip unit.score "systemd-analyze not found"; fi
prop_ok core.limit LimitCORE=0

file_ok files.secrets /etc/codeherder/secrets.env 600 "$secret_owner"
file_ok files.secrets /etc/codeherder/db.env 600 "$secret_owner"
file_ok files.secrets /etc/codeherder "700 750" "$secret_owner"
out=$(stat -c '%A %U' "$root/usr/local/bin/codeherder" 2>/dev/null) || out=""
case $out in
  "") fail files.binary "/usr/local/bin/codeherder missing" ;;
  ?????-??-*" $secret_owner") pass files.binary "$out" ;;
  *) fail files.binary "$out: want owner $secret_owner, no group or other write" ;;
esac
file_ok files.state /var/lib/codeherder "750 700" "$state_owner"

if [ -r "$root/etc/caddy/Caddyfile" ]; then
  if grep -v '^[[:space:]]*#' "$root/etc/caddy/Caddyfile" | grep -q trusted_proxies; then
    fail proxy.trusted_proxies "Caddyfile sets trusted_proxies"
  else pass proxy.trusted_proxies "not set"; fi
else skip proxy.trusted_proxies "no Caddyfile at /etc/caddy/Caddyfile"; fi
l=$(ss -ltnH 2>/dev/null | awk '$4 ~ /:7878$/ {print $4}')
if [ -z "$l" ]; then fail proxy.loopback "nothing listens on 7878"
elif echo "$l" | grep -qvE '^(127\.0\.0\.1|\[::1\]):7878$'; then fail proxy.loopback "7878 is not loopback only: $l"
else pass proxy.loopback "$l"; fi

ok=0; [ "$(timedatectl show -p NTPSynchronized --value 2>/dev/null)" = yes ] && ok=1
off=""
if have chronyc; then
  off=$(chronyc tracking 2>/dev/null | awk '/^System time/ {print $4}')
  awk "BEGIN{exit !(\"$off\" != \"\" && $off < 1)}" 2>/dev/null || ok=0
fi
verdict clock.ntp "synchronized${off:+, offset ${off}s}" "not synchronized or offset over 1 s${off:+ (${off}s)}" [ "$ok" = 1 ]

id="" ver=""
if [ -r "$root/etc/os-release" ]; then
  id=$(sed -n 's/^ID=//p' "$root/etc/os-release" | tr -d '"')
  ver=$(sed -n 's/^VERSION_ID=//p' "$root/etc/os-release" | tr -d '"')
fi
maj=${ver%%.*} sd=$(systemctl --version 2>/dev/null | awk 'NR==1 {print $2}')
case $id in
  amzn) min=2023 ;; debian) min=12 ;; ubuntu) min=24 ;; rhel|rocky|almalinux|centos) min=9 ;; *) min="" ;;
esac
if [ -n "$min" ] && [ "${maj:-0}" -ge "$min" ] 2>/dev/null && [ "${sd:-0}" -ge 252 ] 2>/dev/null; then
  pass os.release "$id $ver, systemd $sd"
else fail os.release "$id $ver, systemd ${sd:-?}: want AL2023, Debian 12, Ubuntu 24.04 or RHEL 9 and systemd 252 or later"; fi

if [ -e "$root/run/reboot-required" ]; then
  verdict os.reboot "reboot pending for under 7 days" "reboot pending for more than 7 days" \
    [ -z "$(find "$root/run/reboot-required" -mtime +7 2>/dev/null)" ]
elif have needs-restarting; then
  verdict os.reboot "no reboot pending" "reboot pending" quiet needs-restarting -r
elif [ "$id" = debian ] || [ "$id" = ubuntu ]; then pass os.reboot "no reboot pending"
else skip os.reboot "no reboot check for this OS"; fi

en() { [ "$(systemctl is-enabled "$1" 2>/dev/null)" = enabled ]; }
if en dnf-automatic.timer || en dnf-automatic-install.timer; then pass os.autoupdate "dnf-automatic enabled"
elif en apt-daily-upgrade.timer && have unattended-upgrade; then pass os.autoupdate "unattended-upgrades enabled"
else fail os.autoupdate "no automatic update timer enabled"; fi

if [ -z "$host" ]; then skip tls.expiry "pass --host <public host>"
elif echo | openssl s_client -connect "$host:443" -servername "$host" 2>/dev/null \
  | openssl x509 -noout -checkend 1814400 >/dev/null 2>&1; then pass tls.expiry "$host valid for 21 days or more"
else fail tls.expiry "$host: no certificate or it expires within 21 days"; fi

src=$(df --output=source "$root/var/lib/codeherder" 2>/dev/null | tail -n 1)
if [ -n "$src" ] && lsblk -s -no TYPE "$src" 2>/dev/null | grep -qx crypt; then pass disk.encryption "$src is on a crypt device"
else skip disk.encryption "not visible to the guest; on EC2 run: aws ec2 describe-volumes --query 'Volumes[].Encrypted'"; fi

[ $fails -eq 0 ]
```

## Units and drop-ins

The reference unit and its six drop-ins must be the ones that run. Replace `codeherder` with your unit name if you changed it.

| Check | Expected value | Command |
| --- | --- | --- |
| Unit state | `active` | `systemctl is-active codeherder` |
| Drop-ins loaded | `10-production.conf`, `15-origins.conf`, `20-secrets.conf`, `25-mail.conf`, `30-identity.conf`, `40-capacity.conf` | `systemctl show -p DropInPaths codeherder` |
| Sandbox | `ProtectSystem=strict`, `ProtectHome=yes`, `NoNewPrivileges=yes`, `PrivateDevices=yes`, `User=codeherder`, `UMask=0027` | `systemctl show -p ProtectSystem -p ProtectHome -p NoNewPrivileges -p PrivateDevices -p User -p UMask codeherder` |
| Production mode | `CH_ENV=production` and `CH_REQUIRE_STRONG_AT_REST_KEY=true` | `systemctl show -p Environment codeherder` |
| Exposure score | 2.0 or lower. The reference unit scores 1.6. | `systemd-analyze security codeherder` |

Compare each installed file with the copy in [Hardening a self-hosted server](https://codeherder.com/docs/self-host-hardening/). Save the guide’s copy as `reference.service`, then run `diff`. No output means no drift.

```
systemctl cat codeherder
diff /etc/systemd/system/codeherder.service reference.service
diff -r /etc/systemd/system/codeherder.service.d reference.service.d
```

A difference in a commented Cognito line is normal. Review any other difference.

## File permissions

The unit runs as `codeherder`. Secrets stay readable by root only.

| Path | Owner | Mode |
| --- | --- | --- |
| `/etc/codeherder` | `root` | `700` or `750` |
| `/etc/codeherder/secrets.env` | `root` | `600` |
| `/etc/codeherder/db.env` | `root` | `600` |
| `/usr/local/bin/codeherder` | `root` | `755`, not writable by group or other |
| `/var/lib/codeherder` | `codeherder` | `750` or `700` |

```
sudo stat -c '%a %U %n' /etc/codeherder /etc/codeherder/secrets.env /etc/codeherder/db.env /usr/local/bin/codeherder /var/lib/codeherder
```

## Reverse proxy

The proxy must not set `trusted_proxies`. The server must listen on loopback only. See the contract at the top of the reference Caddyfile.

| Check | Expected value | Command |
| --- | --- | --- |
| Config is valid | exit code 0 | `caddy validate --config /etc/caddy/Caddyfile` |
| No `trusted_proxies` | no output | `grep -v '^[[:space:]]*#' /etc/caddy/Caddyfile \| grep trusted_proxies` |
| Loopback only | one line, `127.0.0.1:7878` | `ss -ltn \| grep 7878` |

Check the forwarded address by hand. Send `curl -H 'X-Forwarded-For: 198.51.100.1' https://<api>/v1/...`. Read the access log or the audit IP. It must show your real address, not `198.51.100.1`. If it shows `198.51.100.1`, fix `CH_TRUSTED_PROXY_HOPS`.

## Supported operating systems and updates

The reference unit needs systemd 252 or later. CodeHerder tested the reference on this floor. It is not a vendor certification.

| Host | Supported versions |
| --- | --- |
| App host | x86_64 or arm64 Linux with systemd 252 or later: Amazon Linux 2023, Debian 12, Ubuntu 24.04, RHEL 9 |
| Not supported | Amazon Linux 2, RHEL 8, Ubuntu 22.04 (systemd older than 252) |
| Device | The `ch` targets in the release, on macOS 12 or later or a Linux kernel your vendor supports. Docker mode needs Docker Engine. |

Enable automatic OS updates on the app host. Use `dnf-automatic` with `apply_updates = yes` on Amazon Linux and RHEL. Use `unattended-upgrades` on Debian and Ubuntu. Reboot within 7 days of a kernel or systemd update.

```
cat /etc/os-release
systemctl --version | head -n 1
systemctl is-enabled dnf-automatic.timer || systemctl is-enabled apt-daily-upgrade.timer
needs-restarting -r; ls -l /run/reboot-required
```

After each reboot, run these checks. Every one must pass.

```
systemctl is-active codeherder
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:7878/v1/health
sudo bash /usr/local/sbin/verify-host.sh --host codeherder.example.com
ch instance check
```

Health returns `200`. For a server upgrade, follow [Updating](https://codeherder.com/docs/updating/).

After an OS update or a device replacement, check each device. `ch device list` must show the device online. In `ch device show <ref>`, the checks `tunnel_connected`, `version_current` and `clock_skew` must read `ok`. In Docker mode, `docker_ready` must read `ok` too. See [Launch a device on AWS](https://codeherder.com/docs/launch-a-device-on-aws/) and [Running the device server](https://codeherder.com/docs/running-the-device-server/).

## Disk encryption

The app host disk and the database must be encrypted at rest. A guest cannot see EBS encryption, so check it in AWS.

| Check | Expected value | Command |
| --- | --- | --- |
| EBS default | `true` | `aws ec2 get-ebs-encryption-by-default --query EbsEncryptionByDefault` |
| Host volumes | every value `true` | `aws ec2 describe-volumes --filters Name=attachment.instance-id,Values=<instance id> --query 'Volumes[].Encrypted'` |
| RDS | every value `true` | `aws rds describe-db-instances --query 'DBInstances[].StorageEncrypted'` |
| Other hosts | a `crypt` line under the data mount | `lsblk -s -no TYPE $(df --output=source /var/lib/codeherder \| tail -n 1)` |

Device disk encryption is your choice, recorded per device. See [Self-hosted responsibilities](https://codeherder.com/docs/self-host-responsibilities/).

## Core dumps

A core dump holds the four at-rest keys and the database credentials. The reference unit sets `LimitCORE=0`, so the server writes none.

| Check | Expected value | Command |
| --- | --- | --- |
| Unit limit | `LimitCORE=0` | `systemctl show -p LimitCORE codeherder` |
| Running process | `Max core file size 0 0` | `grep 'core file' /proc/$(systemctl show -p MainPID --value codeherder)/limits` |
| No dumps stored | empty list | `coredumpctl list codeherder` |

To stop core dumps for every service, create `/etc/systemd/coredump.conf.d/10-none.conf` with `[Coredump]` and `Storage=none`.

## Certificates and domains

Check every public endpoint: the server host, the Cognito custom domain, and an OTLP or SIEM collector if it is public. Each certificate must have 21 days or more left. Caddy renews at 30 days left, so under 21 days means renewal failed.

```
echo | openssl s_client -connect codeherder.example.com:443 -servername codeherder.example.com 2>/dev/null \
  | openssl x509 -noout -enddate -checkend 1814400
```

`-checkend` prints `Certificate will not expire` and exits 0 when the certificate is good. Repeat the command for each endpoint.

Check each domain registration too. In Route 53, `AutoRenew` must be `true`. Elsewhere, read the expiry from `whois`.

```
aws route53domains get-domain-detail --region us-east-1 --domain-name example.com --query '[AutoRenew,ExpirationDate]'
whois example.com | grep -i expir
```

CodeHerder also emails workspace admins before an SSO credential expires. See [Self-hosted deployment](https://codeherder.com/docs/self-hosting/).

## Clock sync

A skewed clock breaks token checks and orders audit events wrongly. Check the app host, each audit receiver and each device.

| Host | Expected value | Command |
| --- | --- | --- |
| API host | `NTPSynchronized=yes` | `timedatectl show -p NTPSynchronized` |
| API host and audit receivers | offset under 1 second | `chronyc tracking` (read `System time`) |
| Device | `clock_skew` reads `ok` (skew under 10 seconds) | `ch device show <ref>` |

On AWS, use Amazon Time Sync. Amazon Linux 2023 runs it through chrony by default. Audit receivers are the OTLP collector and the SIEM forwarder. See [Alarm on trace export failures](https://codeherder.com/docs/self-host-trace-export/).

## Related guides

- [Hardening a self-hosted server](https://codeherder.com/docs/self-host-hardening/) — the reference unit, drop-ins and Caddyfile
- [Self-hosted synthetic checks](https://codeherder.com/docs/self-host-checks/) — the scheduled `ch instance check`
- [Self-hosted responsibilities and launch acceptance](https://codeherder.com/docs/self-host-responsibilities/) — who owns each check
- [Self-hosted deployment](https://codeherder.com/docs/self-hosting/) — install and upgrade the server