# Verify the audit export Source: https://codeherder.com/docs/self-host-audit-integrity/ Check that the audit webhook export you keep is complete and unaltered, even after the server pruned its own rows or a workspace was deleted. CodeHerder prunes old audit rows. A workspace owner can also delete a whole workspace. Your audit receiver keeps its own copy. This page shows how to prove that copy is complete and unaltered. The proof comes from digests. A digest is a signed webhook delivery that lists the hash of every other delivery. You need only the raw delivery bodies you already store. ## What a digest is Each webhook subscription gets a `webhook.digest` delivery about every 15 minutes, when new deliveries exist. Each digest covers the deliveries older than five minutes that no earlier digest covers. Each digest also links to the digest before it. The digest arrives like any other delivery. It carries the same `X-CodeHerder-Signature` header, so the subscription secret signs it. A digest lists every delivery of the subscription. That includes `webhook.test` deliveries, earlier digests, and deliveries that failed. A failed delivery is a gap you must be able to see. The digest body uses the subscription format: - Format `codeherder`: the digest is the `payload` object of the envelope. - Format `ocsf`: the digest is the `unmapped` object of the record. The digest object has these fields: | Field | Meaning | | --- | --- | | `version` | Always `1`. | | `subscriptionId` | The subscription the chain belongs to. | | `seq` | The digest number. The first digest is `1`. Each next digest adds `1`. | | `prevDigestHash` | The `digestHash` of the digest before it. Empty for `seq` 1. | | `digestHash` | The hash of this digest. See the next section. | | `final` | `true` only for the last digest of a deleted workspace. | | `records` | One entry per delivery: `deliveryId`, `eventId`, `eventType`, `sha256`. | `sha256` is the hex SHA-256 of the exact body bytes CodeHerder sent for that delivery. The `X-CodeHerder-Delivery-Id` header names the delivery. Store the raw bytes, not a parsed copy. ## How to compute `digestHash` Build this text, then take the hex SHA-256 of its UTF-8 bytes: ``` codeherder-webhook-digest-v1 ``` Then add one line per record, in the order of `records`: ``` ``` `` is a line feed. `` is one space. `seq` is a plain decimal number. Test vector. This input gives the hash below: ``` codeherder-webhook-digest-v1 0198a000-0000-7000-8000-000000000001 2 aa 0198a000-0000-7000-8000-000000000010 0198a000-0000-7000-8000-000000000011 task.done bb 0198a000-0000-7000-8000-000000000020 0198a000-0000-7000-8000-000000000021 webhook.test cc ``` ``` 6bc4082c736616511cee62ab3c4acde36e350803f7b57c4575460cd7bcfb3064 ``` ## The three checks Run these checks on your stored copy, subscription by subscription. 1. **Signature.** Check `X-CodeHerder-Signature` on the digest, as for any delivery. See [Webhooks](https://codeherder.com/docs/webhooks/) for the signature format. A digest that fails this check is forged. 2. **Every record is present and unaltered.** For each record in each digest, find the stored body with that `deliveryId`. No body means a missing record. A body whose SHA-256 differs from `sha256` means an altered record. 3. **The chain is unbroken.** The first digest must have `seq` 1. Each next digest must have `seq` one higher and a `prevDigestHash` equal to the earlier `digestHash`. Recompute `digestHash` from the fields and compare. A gap in `seq` means a dropped digest. A stored body that no digest lists is an extra or forged record. The newest deliveries are not listed until the next digest arrives. Allow for that delay. ## A short verifier This Python script applies checks 2 and 3. It reads a folder of raw bodies named `.json`, and a list of digest delivery ids in arrival order. ``` import hashlib, json, sys, pathlib def digest_object(body): doc = json.loads(body) return doc.get("payload") or doc["unmapped"] def digest_hash(d): text = "codeherder-webhook-digest-v1\n%s\n%d\n%s\n" % ( d["subscriptionId"], d["seq"], d["prevDigestHash"]) for r in d["records"]: text += "%s %s %s %s\n" % (r["deliveryId"], r["eventId"], r["eventType"], r["sha256"]) return hashlib.sha256(text.encode()).hexdigest() def verify(folder, digest_ids): problems, seq, prev = [], 0, "" for did in digest_ids: d = digest_object((folder / (did + ".json")).read_bytes()) if d["seq"] != seq + 1 or d["prevDigestHash"] != prev: problems.append("chain break at seq %d" % d["seq"]) if digest_hash(d) != d["digestHash"]: problems.append("digest %d does not match its hash" % d["seq"]) for r in d["records"]: path = folder / (r["deliveryId"] + ".json") if not path.exists(): problems.append("missing record " + r["deliveryId"]) elif hashlib.sha256(path.read_bytes()).hexdigest() != r["sha256"]: problems.append("altered record " + r["deliveryId"]) seq, prev = d["seq"], d["digestHash"] return problems if __name__ == "__main__": folder = pathlib.Path(sys.argv[1]) ids = pathlib.Path(sys.argv[2]).read_text().split() found = verify(folder, ids) print("\n".join(found) or "ok") sys.exit(1 if found else 0) ``` Run it as `python3 verify.py ./bodies digest-ids.txt`. Put the digest delivery ids in `digest-ids.txt`, one per line, in arrival order. The script exits non-zero when it finds a problem. ## When a workspace is deleted Deleting a workspace removes its subscriptions and deliveries. Before that, CodeHerder seals one `final` digest for each subscription. The final digest lists every delivery no earlier digest covers. It links to the last digest the subscription sent. CodeHerder does not send the final digest to your endpoint, because the subscription no longer exists. It stores the digest in a deletion record on the instance instead. An instance operator reads the records: ``` curl -H "Authorization: Bearer $CH_TOKEN" \ "$CH_ADDR/v1/instance/workspace-deletions?limit=50" ``` Each record holds the workspace id and name, the ids of the workspace and its child workspaces, who deleted it (`deletedByMemberId`, `deletedByEmail`, or the system), the time, the number of events removed, and `finalDigests`. The deletion record is not deleted with the workspace. Nothing in CodeHerder updates or removes it. Verify a final digest as you verify any digest. Its `prevDigestHash` must equal the `digestHash` of the last digest you received for that subscription. A digest delivery can still be waiting to send when the workspace is deleted. That digest never reaches you. The final digest then lists it as a missing record, and you see a gap in `seq`. This is expected at deletion. It is not a sign of tampering. ## Limits - The digests cover the webhook export. They do not cover the OpenTelemetry trace export. - A digest proves loss or change after CodeHerder queued the delivery. It does not detect a database administrator who edits an event before the webhook matcher reads it. - A delivery row that commits more than five minutes after its id was minted can miss its digest window. - The final digest of a deleted workspace is not signed. Trust it as far as you trust the instance operator’s read of the record. Its chain link to the last signed digest is the check. - The newest deliveries are unproven until the next digest arrives. To ship logs and audit records off the host, see [Self-hosted logs](https://codeherder.com/docs/self-host-logs/).