# How agent environments work

Source: https://codeherder.com/docs/agent-environments/

How a stage's container is built, mounted and fenced in, from the base image up to the agent, and where your code lives.

This page explains how the parts of an agent environment fit together. It covers the image, your settings, what a stage can reach at run time, and where your code lives. It describes the container option. For the how-to steps, follow the links in each section.

## The layers, in order

A stage in a container is built up in layers. Each layer comes from a different place.

```
harness            the coding agent CLI the stage runs
guardrails         rules the agent cannot rewrite
before_stage       your setup script, run before the agent starts
toolchain          your pinned tools, mounted read-only at /opt/ch-env
ch-env image       base image plus your system packages
base image         the CodeHerder stage image
```

| Layer | Where it comes from | When it changes |
| --- | --- | --- |
| Base image | The newest runtime image that CodeHerder promotes. It holds no `ch` or harness. | When the runtime image changes. A `ch` or harness release does not change it. |
| System packages | Built on the device, on top of the base image, as a `ch-env` image. | When you change packages or Dockerfile fragments, or when the runtime image changes. |
| Toolchain | Mounted from the device’s toolchain cache. | When you change the pinned tools. |
| `before_stage` | Runs at stage start, inside the container. | On every stage run. |
| Guardrails | Mounted into the stage and enforced there. | When the guardrail settings change. |
| `ch` | Mounted read-only at `/opt/ch/bin/ch` from the device’s binary cache. It is the `ch` of the device’s own version. | When the device updates. |
| Harness, `gh` and `glab` | Mounted read-only at `/opt/ch-harness/<name>/<version>/` from the device’s binary cache. The device downloads each release straight from the vendor, once per version and platform. An older image can hold its own copy. | When the server resolves a new release, or when you pin a version. |

## The base image

The base image holds Debian, git, the shell tools and the Docker CLI. It holds no `ch`, no harness CLI, no `gh` and no `glab`. The device mounts each of them into the stage from its own cache, so a `ch` or harness release never changes the image. CodeHerder pins the base image in four ways:

- CodeHerder pins the Debian base, `debian:trixie-slim`, by digest.
- Apt points at one dated `snapshot.debian.org` snapshot, so the package set is the same on every build.
- Each downloaded file is checked with `sha256sum -c` before the build uses it.
- A few packages are patched builds with a `+chN` version. See [Packages that cannot be installed on the base image](https://codeherder.com/docs/stage-environments/#packages-that-cannot-be-installed-on-the-base-image) for the list.

The device mounts its own `ch` into every stage. This holds for the base image and for a custom image. The device keeps the binaries in `~/.codeherder/bin`, one for each version and platform. It reads the platform from the stage image. An image that runs under emulation gets a matching binary.

The first stage on a new device downloads the binary and checks it. This happens before the stage starts. The device needs a route to the CLI release host for it. On Docker Desktop the device copies the binary into a Docker volume when it cannot bind-mount its own directory.

A device can run the current image only when it can supply every part: it runs the Docker executor, it has a binary cache, and it can supply a Linux `ch` of its own version. A device that cannot gets an older image that still carries its own copies of these tools. If no older image is left, the device gets no image. Update `ch` on the device to fix this.

If the current image needs a part the device cannot supply, the stage does not start. The message names the missing part. The part can be `ch`, the harness, `gh` or `glab`.

A workspace with system packages gets its own `ch-env` image. The device builds it on top of the newest runtime image. A `ch` or harness release rebuilds no `ch-env` image. A `ch-env` image built on the current base runs only on a device that can supply every part. For any other device, the server refuses the stage and asks you to update `ch` on the device. When only the runtime image changes, the device keeps serving the old `ch-env` image while it builds the new one. When your packages or fragments change, there is no old image that fits. The stage waits for the build. See [Giving stages the tools they need](https://codeherder.com/docs/stage-environments/) for how to fix a stage that waits.

## The harness version

The server picks the harness version for each spawn. It also picks the `gh` and `glab` versions. A device does not pick them.

- The default is the newest stable vendor release inside the harness’s version range. `gh` and `glab` have their own ranges.
- A prerelease, a draft or a tag with a suffix is never a candidate.
- Every 30 minutes, the server looks for a newer release and stores it. A spawn reads only the stored releases. It never calls the vendor.
- If the vendor is unreachable, the server keeps the newest stored release. A normal stage uses that one.
- The server scans each new release for known advisories in the background. It reports what it finds. A finding never blocks a version.

A launch config can pin one exact version. Set `harnessVersion` on the API, or run `ch agent config create` or `ch agent config edit` with `--harness-version <version>`. An empty value means the newest release in range.

- The server checks a pin when you write it. A version outside the range, or a version the vendor does not have, is refused with `invalid_input`. The message names the range.
- A harness with no vendor release list cannot take a pin.
- When you change the harness and do not give a new version, the pin is cleared. A copied agent keeps its pin.
- An evaluation pins a version through the same launch config field.

A pin never falls back to another version. If the pinned version has no stored release at spawn time, the spawn is refused with `harness_release_unresolved`.

The server tells the device which release of each binary to use. The device mounts exactly that set and keeps it until the stage ends. It downloads each build once per version and platform, and checks its checksum. It fetches a new latest release ahead of the next spawn, so a new Claude Code release inside the range reaches a stage with no image build.

- **Vendor outage.** If a download fails, a normal stage uses the newest version the device already cached, inside the harness’s range. The run records it as an outage fallback. With nothing cached, an older image runs its own harness. The current image has none, so the stage does not start.
- **Pinned version.** A pinned version that is not cached and cannot be downloaded refuses the spawn with `harness_release_unavailable`. It never falls back to another version.
- **No build for the image.** If the vendor has no build for the stage image’s platform or libc, the spawn is refused with `harness_platform_unavailable`. The message names both platforms. A custom image on Alpine (musl) has no Claude Code build and gets this refusal.
- **Gh and glab.** If one cannot be had, an older image runs its own copy. On the current image the stage does not start, and the message names the missing tool.

The device needs outbound access to the vendor hosts. See [Self-hosting](https://codeherder.com/docs/self-hosting/).

## What a session records

Each session records what its device ran. It never records what the server asked for.

- The version of `ch` that ran, and where it came from: the device cache, the image, or the host.
- The version, checksum and platform of each mounted binary: the harness, `gh` and `glab`.
- The digest of the runtime image.
- The hash of the `ch-env` image that served.

A binary from an outage fallback carries a mark. It shows the version that ran and the version the server asked for.

A `ch-env` image can serve from the fallback while the new image builds. The session then records the hash and digest of the fallback. It also names the hash it asked for.

These values are a record of what ran. They do not change the session’s setup, so a `ch` release does not either.

Run `ch session show <id>` to read them. The `launch.*` rows list each value. The API returns the same values. A session from an older device can show only some rows. A missing value prints as `unknown`. A device from before the binary cache sends no `launch` block.

A session on a microVM pool records different values. The microVM runs each stage in a container on its own Docker daemon. The session records:

- The `ch` the microVM runs, with the source `host`. A new `ch` release does not rebuild the pool image. A microVM that starts on an older `ch` hands over to the current release before it takes a session.
- The identifier of the pool image the microVM launched from. It shows in the runtime image row.

## The three settings

You control three settings. They map to three layers above.

| Setting | Where it lands | Use it for |
| --- | --- | --- |
| Tools | Mounted read-only at `/opt/ch-env` from the device toolchain cache. | A pinned version of a language tool, a CLI or a package. |
| System | Debian packages and Dockerfile fragments, built into the `ch-env` image. | A library or program that needs the operating system. |
| `before_stage` | A script that runs in the container before the agent starts. | Work that depends on the checkout, such as an install step. |

CodeHerder folds the settings from the widest layer to the narrowest. The order is the base, each ancestor group from the root, the workspace, the workflow and then the stage. The rules are:

- A tool merges by name. The later layer wins. A `remove` entry drops a tool a wider layer declared.
- Packages form one sorted union. A narrower layer cannot remove a package.
- Dockerfile fragments keep layer order.
- Setup scripts join in layer order, widest first.

For the commands and the web app pages, see [Giving stages the tools they need](https://codeherder.com/docs/stage-environments/). To name your own image, see [Running a stage in your own container image](https://codeherder.com/docs/stage-images/).

## A stage at run time

Each stage runs in its own container. Each run gets a private Docker network with no route out. A run proxy on that network is the only network route out of the stage itself.

The container mounts:

- The task’s worktree, read-write, at `/workspace`.
- The workspace mirror’s git directory, read-write, with its `config` and hooks read-only.
- The session’s transcript directory, read-write.
- The toolchain, the session bearer, the harness settings and the guardrails, read-only.
- A socket for git credentials.

The stage never sees the device’s credential pool or the device-local socket. It never sees other mirrors, other sandboxes or the operator’s home directory.

The run proxy offers these services:

| Service | What it does |
| --- | --- |
| Git relay | Sends git and git host API calls from the stage to the host. The push credential stays outside the stage. See [Git credentials on a device](https://codeherder.com/docs/git-tokens-on-a-device/). |
| Integration relay | Forwards a call to an integration. The server adds the vendor key, so the stage never holds it. |
| Pull cache | Serves package and browser downloads from a device cache. It also serves the `.deb` files of a `ch-env` image rebuild. See [Package and browser downloads in container stages](https://codeherder.com/docs/package-downloads/). |

Egress follows the stage’s **Writable** switch. A writable stage runs in strict mode. The proxy allows only a list of hosts. A read-only stage is open by default. The device operator can change this with `EGRESS_MODE`.

Both modes refuse private ranges, loopback and cloud metadata. When a stage needs a blocked host, the proxy files a request. A person approves it, and the stage retries. See [Network access approvals](https://codeherder.com/docs/egress-approvals/) and [Isolating agent runs on a device](https://codeherder.com/docs/agent-isolation/).

### Docker in a stage

A stage can use Docker through a per-run gateway. The gateway keeps the run’s containers, networks and volumes scoped to that run. The stage never sees the Docker socket.

Stage Docker is on by default. Set `CH_STAGE_DOCKER=false` on the device to turn it off. See [Self-hosted device isolation](https://codeherder.com/docs/self-host-device-isolation/#docker-inside-a-stage-testcontainers).

## Where code lives

The device keeps one mirror for each workspace and repository URL. The mirror is a bare clone that every sandbox of that workspace and repository shares.

Each task gets one sandbox. It is a git worktree on its own branch. The sandbox stays between stages, so the next stage starts from the same files.

When the device removes a sandbox, it protects the work first:

1. It commits uncommitted work as a WIP commit.
2. It pushes the branch.
3. If it cannot show that the work is safe in git, it keeps the whole folder. It does not delete it.

The device makes no bundle file. The server stores task records, the event timeline and settings. It stores no copy of your repository. It stores no chat transcript either, unless the workspace turns on [stored session transcripts](https://codeherder.com/docs/session-transcripts/). Worktrees and the device copy of each transcript stay on the device. Code also stays on your git host. See [Where your data lives](https://codeherder.com/docs/self-host-data-residency/) and [Following a live agent session](https://codeherder.com/docs/following-a-live-session/).

## EC2 and microVM side by side

Both device kinds run the same executor, the same stage image and the same proxy. They differ in the host and in how the image arrives.

| Property | EC2 device | MicroVM runner |
| --- | --- | --- |
| Docker daemon | The host’s daemon. | A daemon inside the microVM. |
| Image delivery | The device pulls the promoted image from the server. | CodeHerder builds a pool image with the stage and `ch-env` images inside it. |
| Update path | The device pulls a new image and refreshes `ch` at boot. | CodeHerder rebuilds the pool image only when the stage runtime image or a runner input changes. A microVM hands over to the current `ch` before it takes a session. |
| Lifetime | Long-lived. | Short-lived. Each microVM serves sessions, then ends. |

See [Launch a device on AWS](https://codeherder.com/docs/launch-a-device-on-aws/), [MicroVM runners](https://codeherder.com/docs/microvm-runners/) and [Devices that clean up after themselves](https://codeherder.com/docs/ephemeral-devices/).

## Known limits

- The mirror is writable. Every sandbox of the same workspace and repository shares it. A hostile stage can change it.
- A read-only stage has open egress by default. Set `EGRESS_MODE` to change it.
- A change to packages or fragments has no fallback image. The stage waits for the build.
- A device in host mode has none of the container controls on this page.
- The Docker daemon pulls images for a stage. The stage egress filter does not apply to that pull.
- The Docker socket proxy does not limit image registries. This matters only if the device server is compromised.
- A pinned version that the device has not cached needs the vendor hosts to be reachable the first time. Without them, the spawn is refused.

See [Known limits of the container option](https://codeherder.com/docs/agent-isolation/#known-limits-of-the-container-option) for the first reading of these limits.

## Related guides

- [Giving stages the tools they need](https://codeherder.com/docs/stage-environments/): the how-to for tools, packages and `before_stage`
- [Running a stage in your own container image](https://codeherder.com/docs/stage-images/): a custom image per stage
- [Isolating agent runs on a device](https://codeherder.com/docs/agent-isolation/): what a stage can reach
- [Network access approvals](https://codeherder.com/docs/egress-approvals/): approve a blocked host
- [Package and browser downloads in container stages](https://codeherder.com/docs/package-downloads/): the pull cache
- [Git credentials on a device](https://codeherder.com/docs/git-tokens-on-a-device/): how git access reaches a stage
- [MicroVM runners](https://codeherder.com/docs/microvm-runners/): short-lived AWS devices
- [Launch a device on AWS](https://codeherder.com/docs/launch-a-device-on-aws/): an EC2 device
- [Devices that clean up after themselves](https://codeherder.com/docs/ephemeral-devices/): throwaway devices
