CodeHerderSearch⌘KRequest access →

Self-hosted diagnosis exercise

A facilitator stages a tunnel fault and a breaker fault on a test device. Your operator follows the runbooks, hands off and fills in the records.

A facilitator runs this exercise with your operator and a device owner. The facilitator is anyone other than the operator, for example a second administrator. If your support agreement includes this exercise, the support contact can be the facilitator. Use a test device and a test task. Never use a device that does live work.

The operator follows Diagnose device tunnel failures and Release a provisioning breaker. The facilitator watches the clock and does not tell the operator the fault in advance. The handoff goes to whoever answers support handoffs for your install: your support contact, or the facilitator in that role.

Before you start

  • The test device is online and passes ch device show.
  • The operator has a ch login as a workspace administrator and an instance operator login.
  • The person who receives the handoff has the channel from Self-hosted support and incident response open.
  • Everyone agrees: only the bundle and typed answers cross in the handoff. No token, service.env or raw log.

Fault A: tunnel drop

Time Who Action Record
T+0 Device owner Stop the device server on the test device. Or block its outbound traffic to the server. Fault start time
T+3 min Operator Notice the device offline. Start step 1 of the tunnel runbook. Detected at
T+8 min Operator Finish steps 2 and 3. Fill in the answer table. Findings
T+10 min Device owner Restore the device server or the traffic. Restored at
T+12 min Operator Run ch instance support-bundle. Read the bundle. Bundle produced at
T+15 min Operator Send the bundle, device ID, time window and answer table to the handoff receiver. Handoff sent at
Next Handoff receiver Send back the severity, a reading and the next step. Answer at

Pass when the operator finds the drop in the server timeline, hands off by T+15 min, and sends no credential.

Fault B: provisioning breaker

Time Who Action Record
T+0 Device owner Stop other work on the test device. Put a plain file where the device keeps its repository copies (see below). Fault start time
T+1 min Facilitator Create the test story with ch task create in the test workspace. The test device is the only device there, so it takes the task. Task ID
T+3 min Facilitator Watch ch activity task <task> until task.device_provision_ineligible shows. Two failures cause it. Event time
T+5 min Operator Notice the stuck task. Follow steps 1 and 2 of the breaker runbook. Detected at
T+10 min Device owner Remove the file and put the directory back (see below). Fixed at
T+12 min Operator Run ch device clear-provision-breaker <device>. Expect cleared provision breaker for 1 task(s) and a task.device_provision_breaker_cleared event. Released at
T+15 min Operator Run ch instance support-bundle. Read it. Send the bundle, device ID, time window, task ID and answer table to the handoff receiver. Handoff sent at
Next Handoff receiver Send back the severity, a reading and the next step. Answer at

Pass when the release clears 1 task, the task runs again within 5 minutes of the release, the operator hands off by T+15 min, and sends no credential.

Stage the fault

Use a test workspace that has one repository and only the test device. Read the session root path from the session_root row of ch device show <device>. On the device, run:

mv <session-root>/.mirrors <session-root>/.mirrors.aside   # skip if .mirrors does not exist
touch <session-root>/.mirrors

The device server now cannot create its copy of the repository, so each start fails as provision_failed. The readiness checks still pass: session_root tests only the root, and git_credentials reads the remote directly. So the device stays online and healthy, and the server keeps sending the task to it. Do not break the git credential instead. A failed git_credentials check makes the device ineligible before any start, so no breaker event occurs.

To fix the fault, run:

rm <session-root>/.mirrors
mv <session-root>/.mirrors.aside <session-root>/.mirrors   # skip if you skipped the first mv

What to expect

  • The second failure on the device emits task.device_provision_ineligible. The device then sits out this task for 10 minutes.
  • The task does not reach the fifth provision_failed inside this script, so no person-only blocker note appears. If the fault stays past two cooldowns, the note appears. The release clears it.
  • The device never fails a readiness check, so nothing releases the breaker for you. The manual release in step 3 of the runbook must clear 1 task. A count of 0 means the cooldown was already released; record it as a gap.

Evidence record

Fill in one record for each fault. Leave a row blank until the evidence exists.

Field Fault A Fault B
Date
People (operator, device owner, facilitator, handoff receiver)
Fault start
Detected at
Bundle produced at
Handoff sent at
Answer at
Total time
Bundle and typed answers only; no token, service.env or raw log sent (yes or no)
Fault B: failures before task.device_provision_ineligible; cleared count
Gaps found
Pass or fail
Signed by the facilitator

Last updated

Try it on one repository for two weeks.

File a task. An agent builds it. A person reads every request. You see the cost of each task.

Try "pricing", "connect a device", or "who reviews the code"

↑↓ move · ↵ open · esc close