# Self-hosted diagnosis exercise

Source: https://codeherder.com/docs/self-host-diagnosis-exercise/

A facilitator stages a tunnel fault and a breaker fault on a test device. Your operator follows the runbooks and hands off. Blank evidence records follow.

A facilitator runs this exercise with your operator and a device owner. The facilitator is anyone other than the operator, for example a second administrator. If your support agreement includes this exercise, the support contact can be the facilitator. Use a test device and a test task. Never use a device that does live work.

The operator follows [Diagnose device tunnel failures](https://codeherder.com/docs/self-host-tunnel-failures/) and [Release a provisioning breaker](https://codeherder.com/docs/self-host-provision-breaker/). The facilitator watches the clock and does not tell the operator the fault in advance. The handoff goes to whoever answers support handoffs for your install: your support contact, or the facilitator in that role.

## Before you start

- The test device is online and passes `ch device show`.
- The operator has a `ch` login as a workspace administrator and an instance operator login.
- The person who receives the handoff has the channel from [Self-hosted support and incident response](https://codeherder.com/docs/self-host-support/) open.
- Everyone agrees: only the bundle and typed answers cross in the handoff. No token, `service.env` or raw log.

## Fault A: tunnel drop

| Time | Who | Action | Record |
| --- | --- | --- | --- |
| T+0 | Device owner | Stop the device server on the test device. Or block its outbound traffic to the server. | Fault start time |
| T+3 min | Operator | Notice the device offline. Start step 1 of the tunnel runbook. | Detected at |
| T+8 min | Operator | Finish steps 2 and 3. Fill in the answer table. | Findings |
| T+10 min | Device owner | Restore the device server or the traffic. | Restored at |
| T+12 min | Operator | Run `ch instance support-bundle`. Read the bundle. | Bundle produced at |
| T+15 min | Operator | Send the bundle, device ID, time window and answer table to the handoff receiver. | Handoff sent at |
| Next | Handoff receiver | Send back the severity, a reading and the next step. | Answer at |

Pass when the operator finds the drop in the server timeline, hands off by T+15 min, and sends no credential.

## Fault B: provisioning breaker

| Time | Who | Action | Record |
| --- | --- | --- | --- |
| T+0 | Device owner | Stop other work on the test device. Put a plain file where the device keeps its repository copies (see below). | Fault start time |
| T+1 min | Facilitator | Create the test story with `ch task create` in the test workspace. The test device is the only device there, so it takes the task. | Task ID |
| T+3 min | Facilitator | Watch `ch activity task <task>` until `task.device_provision_ineligible` shows. Two failures cause it. | Event time |
| T+5 min | Operator | Notice the stuck task. Follow steps 1 and 2 of the breaker runbook. | Detected at |
| T+10 min | Device owner | Remove the file and put the directory back (see below). | Fixed at |
| T+12 min | Operator | Run `ch device clear-provision-breaker <device>`. Expect `cleared provision breaker for 1 task(s)` and a `task.device_provision_breaker_cleared` event. | Released at |
| T+15 min | Operator | Run `ch instance support-bundle`. Read it. Send the bundle, device ID, time window, task ID and answer table to the handoff receiver. | Handoff sent at |
| Next | Handoff receiver | Send back the severity, a reading and the next step. | Answer at |

Pass when the release clears 1 task, the task runs again within 5 minutes of the release, the operator hands off by T+15 min, and sends no credential.

### Stage the fault

Use a test workspace that has one repository and only the test device. Read the session root path from the `session_root` row of `ch device show <device>`. On the device, run:

```
mv <session-root>/.mirrors <session-root>/.mirrors.aside   # skip if .mirrors does not exist
touch <session-root>/.mirrors
```

The device server now cannot create its copy of the repository, so each start fails as `provision_failed`. The readiness checks still pass: `session_root` tests only the root, and `git_credentials` reads the remote directly. So the device stays online and healthy, and the server keeps sending the task to it. Do not break the git credential instead. A failed `git_credentials` check makes the device ineligible before any start, so no breaker event occurs.

To fix the fault, run:

```
rm <session-root>/.mirrors
mv <session-root>/.mirrors.aside <session-root>/.mirrors   # skip if you skipped the first mv
```

### What to expect

- The second failure on the device emits `task.device_provision_ineligible`. The device then sits out this task for 10 minutes.
- The task does not reach the fifth `provision_failed` inside this script, so no person-only blocker note appears. If the fault stays past two cooldowns, the note appears. The release clears it.
- The device never fails a readiness check, so nothing releases the breaker for you. The manual release in step 3 of the runbook must clear 1 task. A count of 0 means the cooldown was already released; record it as a gap.

## Evidence record

Fill in one record for each fault. Leave a row blank until the evidence exists.

| Field | Fault A | Fault B |
| --- | --- | --- |
| Date |  |  |
| People (operator, device owner, facilitator, handoff receiver) |  |  |
| Fault start |  |  |
| Detected at |  |  |
| Bundle produced at |  |  |
| Handoff sent at |  |  |
| Answer at |  |  |
| Total time |  |  |
| Bundle and typed answers only; no token, `service.env` or raw log sent (yes or no) |  |  |
| Fault B: failures before `task.device_provision_ineligible`; `cleared` count |  |  |
| Gaps found |  |  |
| Pass or fail |  |  |
| Signed by the facilitator |  |  |

## Related guides

- [Diagnose device tunnel failures](https://codeherder.com/docs/self-host-tunnel-failures/)
- [Release a provisioning breaker](https://codeherder.com/docs/self-host-provision-breaker/)
- [Self-hosted acceptance journey](https://codeherder.com/docs/self-host-acceptance-journey/) — the first-account journey