Self-hosted diagnosis exercise

A facilitator runs this exercise with your operator and a device owner. The facilitator is anyone other than the operator, for example a second administrator. If your support agreement includes this exercise, the support contact can be the facilitator. Use a test device and a test task. Never use a device that does live work.

The operator follows Diagnose device tunnel failures and Release a provisioning breaker. The facilitator watches the clock and does not tell the operator the fault in advance. The handoff goes to whoever answers support handoffs for your install: your support contact, or the facilitator in that role.

Before you start

• The test device is online and passes ch device show.
• The operator has a ch login as a workspace administrator and an instance operator login.
• The person who receives the handoff has the channel from Self-hosted support and incident response open.
• Everyone agrees: only the bundle and typed answers cross in the handoff. No token, service.env or raw log.

Fault A: tunnel drop

Time	Who	Action	Record
T+0	Device owner	Stop the device server on the test device. Or block its outbound traffic to the server.	Fault start time
T+3 min	Operator	Notice the device offline. Start step 1 of the tunnel runbook.	Detected at
T+8 min	Operator	Finish steps 2 and 3. Fill in the answer table.	Findings
T+10 min	Device owner	Restore the device server or the traffic.	Restored at
T+12 min	Operator	Run ch instance support-bundle. Read the bundle.	Bundle produced at
T+15 min	Operator	Send the bundle, device ID, time window and answer table to the handoff receiver.	Handoff sent at
Next	Handoff receiver	Send back the severity, a reading and the next step.	Answer at

Pass when the operator finds the drop in the server timeline, hands off by T+15 min, and sends no credential.

Fault B: provisioning breaker

Time	Who	Action	Record
T+0	Device owner	Stop other work on the test device. Put a plain file where the device keeps its repository copies (see below).	Fault start time
T+1 min	Facilitator	Create the test story with ch task create in the test workspace. The test device is the only device there, so it takes the task.	Task ID
T+3 min	Facilitator	Watch ch activity task <task> until task.device_provision_ineligible shows. Two failures cause it.	Event time
T+5 min	Operator	Notice the stuck task. Follow steps 1 and 2 of the breaker runbook.	Detected at
T+10 min	Device owner	Remove the file and put the directory back (see below).	Fixed at
T+12 min	Operator	Run ch device clear-provision-breaker <device>. Expect cleared provision breaker for 1 task(s) and a task.device_provision_breaker_cleared event.	Released at
T+15 min	Operator	Run ch instance support-bundle. Read it. Send the bundle, device ID, time window, task ID and answer table to the handoff receiver.	Handoff sent at
Next	Handoff receiver	Send back the severity, a reading and the next step.	Answer at

Pass when the release clears 1 task, the task runs again within 5 minutes of the release, the operator hands off by T+15 min, and sends no credential.

Stage the fault

Use a test workspace that has one repository and only the test device. Read the session root path from the session_root row of ch device show <device>. On the device, run:

mv <session-root>/.mirrors <session-root>/.mirrors.aside   # skip if .mirrors does not exist
touch <session-root>/.mirrors

The device server now cannot create its copy of the repository, so each start fails as provision_failed. The readiness checks still pass: session_root tests only the root, and git_credentials reads the remote directly. So the device stays online and healthy, and the server keeps sending the task to it. Do not break the git credential instead. A failed git_credentials check makes the device ineligible before any start, so no breaker event occurs.

To fix the fault, run:

rm <session-root>/.mirrors
mv <session-root>/.mirrors.aside <session-root>/.mirrors   # skip if you skipped the first mv

What to expect

• The second failure on the device emits task.device_provision_ineligible. The device then sits out this task for 10 minutes.
• The task does not reach the fifth provision_failed inside this script, so no person-only blocker note appears. If the fault stays past two cooldowns, the note appears. The release clears it.
• The device never fails a readiness check, so nothing releases the breaker for you. The manual release in step 3 of the runbook must clear 1 task. A count of 0 means the cooldown was already released; record it as a gap.

Evidence record

Fill in one record for each fault. Leave a row blank until the evidence exists.

Field	Fault A	Fault B
Date
People (operator, device owner, facilitator, handoff receiver)
Fault start
Detected at
Bundle produced at
Handoff sent at
Answer at
Total time
Bundle and typed answers only; no token, service.env or raw log sent (yes or no)
Fault B: failures before task.device_provision_ineligible; cleared count
Gaps found
Pass or fail
Signed by the facilitator

Related guides

• Diagnose device tunnel failures
• Release a provisioning breaker
• Self-hosted acceptance journey — the first-account journey