How agent environments work
How a stage's container is built, mounted and fenced in, from the base image up to the agent, and where your code lives.
This page explains how the parts of an agent environment fit together. It covers the image, your settings, what a stage can reach at run time, and where your code lives. It describes the container option. For the how-to steps, follow the links in each section.
The layers, in order
A stage in a container is built up in layers. Each layer comes from a different place.
harness the coding agent CLI the stage runs
guardrails rules the agent cannot rewrite
before_stage your setup script, run before the agent starts
toolchain your pinned tools, mounted read-only at /opt/ch-env
ch-env image base image plus your system packages
base image the CodeHerder stage image
| Layer | Where it comes from | When it changes |
|---|---|---|
| Base image | The newest runtime image that CodeHerder promotes. It holds no ch or harness. |
When the runtime image changes. A ch or harness release does not change it. |
| System packages | Built on the device, on top of the base image, as a ch-env image. |
When you change packages or Dockerfile fragments, or when the runtime image changes. |
| Toolchain | Mounted from the device’s toolchain cache. | When you change the pinned tools. |
before_stage |
Runs at stage start, inside the container. | On every stage run. |
| Guardrails | Mounted into the stage and enforced there. | When the guardrail settings change. |
ch |
Mounted read-only at /opt/ch/bin/ch from the device’s binary cache. It is the ch of the device’s own version. |
When the device updates. |
Harness, gh and glab |
Mounted read-only at /opt/ch-harness/<name>/<version>/ from the device’s binary cache. The device downloads each release straight from the vendor, once per version and platform. An older image can hold its own copy. |
When the server resolves a new release, or when you pin a version. |
The base image
The base image holds Debian, git, the shell tools and the Docker CLI. It holds no ch, no harness CLI, no gh and no glab. The device mounts each of them into the stage from its own cache, so a ch or harness release never changes the image. CodeHerder pins the base image in four ways:
- CodeHerder pins the Debian base,
debian:trixie-slim, by digest. - Apt points at one dated
snapshot.debian.orgsnapshot, so the package set is the same on every build. - Each downloaded file is checked with
sha256sum -cbefore the build uses it. - A few packages are patched builds with a
+chNversion. See Packages that cannot be installed on the base image for the list.
The device mounts its own ch into every stage. This holds for the base image and for a custom image. The device keeps the binaries in ~/.codeherder/bin, one for each version and platform. It reads the platform from the stage image. An image that runs under emulation gets a matching binary.
The first stage on a new device downloads the binary and checks it. This happens before the stage starts. The device needs a route to the CLI release host for it. On Docker Desktop the device copies the binary into a Docker volume when it cannot bind-mount its own directory.
A device can run the current image only when it can supply every part: it runs the Docker executor, it has a binary cache, and it can supply a Linux ch of its own version. A device that cannot gets an older image that still carries its own copies of these tools. If no older image is left, the device gets no image. Update ch on the device to fix this.
If the current image needs a part the device cannot supply, the stage does not start. The message names the missing part. The part can be ch, the harness, gh or glab.
A workspace with system packages gets its own ch-env image. The device builds it on top of the newest runtime image. A ch or harness release rebuilds no ch-env image. A ch-env image built on the current base runs only on a device that can supply every part. For any other device, the server refuses the stage and asks you to update ch on the device. When only the runtime image changes, the device keeps serving the old ch-env image while it builds the new one. When your packages or fragments change, there is no old image that fits. The stage waits for the build. See Giving stages the tools they need for how to fix a stage that waits.
The harness version
The server picks the harness version for each spawn. It also picks the gh and glab versions. A device does not pick them.
- The default is the newest stable vendor release inside the harness’s version range.
ghandglabhave their own ranges. - A prerelease, a draft or a tag with a suffix is never a candidate.
- Every 30 minutes, the server looks for a newer release and stores it. A spawn reads only the stored releases. It never calls the vendor.
- If the vendor is unreachable, the server keeps the newest stored release. A normal stage uses that one.
- The server scans each new release for known advisories in the background. It reports what it finds. A finding never blocks a version.
A launch config can pin one exact version. Set harnessVersion on the API, or run ch agent config create or ch agent config edit with --harness-version <version>. An empty value means the newest release in range.
- The server checks a pin when you write it. A version outside the range, or a version the vendor does not have, is refused with
invalid_input. The message names the range. - A harness with no vendor release list cannot take a pin.
- When you change the harness and do not give a new version, the pin is cleared. A copied agent keeps its pin.
- An evaluation pins a version through the same launch config field.
A pin never falls back to another version. If the pinned version has no stored release at spawn time, the spawn is refused with harness_release_unresolved.
The server tells the device which release of each binary to use. The device mounts exactly that set and keeps it until the stage ends. It downloads each build once per version and platform, and checks its checksum. It fetches a new latest release ahead of the next spawn, so a new Claude Code release inside the range reaches a stage with no image build.
- Vendor outage. If a download fails, a normal stage uses the newest version the device already cached, inside the harness’s range. The run records it as an outage fallback. With nothing cached, an older image runs its own harness. The current image has none, so the stage does not start.
- Pinned version. A pinned version that is not cached and cannot be downloaded refuses the spawn with
harness_release_unavailable. It never falls back to another version. - No build for the image. If the vendor has no build for the stage image’s platform or libc, the spawn is refused with
harness_platform_unavailable. The message names both platforms. A custom image on Alpine (musl) has no Claude Code build and gets this refusal. - Gh and glab. If one cannot be had, an older image runs its own copy. On the current image the stage does not start, and the message names the missing tool.
The device needs outbound access to the vendor hosts. See Self-hosting.
What a session records
Each session records what its device ran. It never records what the server asked for.
- The version of
chthat ran, and where it came from: the device cache, the image, or the host. - The version, checksum and platform of each mounted binary: the harness,
ghandglab. - The digest of the runtime image.
- The hash of the
ch-envimage that served.
A binary from an outage fallback carries a mark. It shows the version that ran and the version the server asked for.
A ch-env image can serve from the fallback while the new image builds. The session then records the hash and digest of the fallback. It also names the hash it asked for.
These values are a record of what ran. They do not change the session’s setup, so a ch release does not either.
Run ch session show <id> to read them. The launch.* rows list each value. The API returns the same values. A session from an older device can show only some rows. A missing value prints as unknown. A device from before the binary cache sends no launch block.
A session on a microVM pool records different values. The microVM runs each stage in a container on its own Docker daemon. The session records:
- The
chthe microVM runs, with the sourcehost. A newchrelease does not rebuild the pool image. A microVM that starts on an olderchhands over to the current release before it takes a session. - The identifier of the pool image the microVM launched from. It shows in the runtime image row.
The three settings
You control three settings. They map to three layers above.
| Setting | Where it lands | Use it for |
|---|---|---|
| Tools | Mounted read-only at /opt/ch-env from the device toolchain cache. |
A pinned version of a language tool, a CLI or a package. |
| System | Debian packages and Dockerfile fragments, built into the ch-env image. |
A library or program that needs the operating system. |
before_stage |
A script that runs in the container before the agent starts. | Work that depends on the checkout, such as an install step. |
CodeHerder folds the settings from the widest layer to the narrowest. The order is the base, each ancestor group from the root, the workspace, the workflow and then the stage. The rules are:
- A tool merges by name. The later layer wins. A
removeentry drops a tool a wider layer declared. - Packages form one sorted union. A narrower layer cannot remove a package.
- Dockerfile fragments keep layer order.
- Setup scripts join in layer order, widest first.
For the commands and the web app pages, see Giving stages the tools they need. To name your own image, see Running a stage in your own container image.
A stage at run time
Each stage runs in its own container. Each run gets a private Docker network with no route out. A run proxy on that network is the only network route out of the stage itself.
The container mounts:
- The task’s worktree, read-write, at
/workspace. - The workspace mirror’s git directory, read-write, with its
configand hooks read-only. - The session’s transcript directory, read-write.
- The toolchain, the session bearer, the harness settings and the guardrails, read-only.
- A socket for git credentials.
The stage never sees the device’s credential pool or the device-local socket. It never sees other mirrors, other sandboxes or the operator’s home directory.
The run proxy offers these services:
| Service | What it does |
|---|---|
| Git relay | Sends git and git host API calls from the stage to the host. The push credential stays outside the stage. See Git credentials on a device. |
| Integration relay | Forwards a call to an integration. The server adds the vendor key, so the stage never holds it. |
| Pull cache | Serves package and browser downloads from a device cache. It also serves the .deb files of a ch-env image rebuild. See Package and browser downloads in container stages. |
Egress follows the stage’s Writable switch. A writable stage runs in strict mode. The proxy allows only a list of hosts. A read-only stage is open by default. The device operator can change this with EGRESS_MODE.
Both modes refuse private ranges, loopback and cloud metadata. When a stage needs a blocked host, the proxy files a request. A person approves it, and the stage retries. See Network access approvals and Isolating agent runs on a device.
Docker in a stage
A stage can use Docker through a per-run gateway. The gateway keeps the run’s containers, networks and volumes scoped to that run. The stage never sees the Docker socket.
Stage Docker is on by default. Set CH_STAGE_DOCKER=false on the device to turn it off. See Self-hosted device isolation.
Where code lives
The device keeps one mirror for each workspace and repository URL. The mirror is a bare clone that every sandbox of that workspace and repository shares.
Each task gets one sandbox. It is a git worktree on its own branch. The sandbox stays between stages, so the next stage starts from the same files.
When the device removes a sandbox, it protects the work first:
- It commits uncommitted work as a WIP commit.
- It pushes the branch.
- If it cannot show that the work is safe in git, it keeps the whole folder. It does not delete it.
The device makes no bundle file. The server stores task records, the event timeline and settings. It stores no copy of your repository. It stores no chat transcript either, unless the workspace turns on stored session transcripts. Worktrees and the device copy of each transcript stay on the device. Code also stays on your git host. See Where your data lives and Following a live agent session.
EC2 and microVM side by side
Both device kinds run the same executor, the same stage image and the same proxy. They differ in the host and in how the image arrives.
| Property | EC2 device | MicroVM runner |
|---|---|---|
| Docker daemon | The host’s daemon. | A daemon inside the microVM. |
| Image delivery | The device pulls the promoted image from the server. | CodeHerder builds a pool image with the stage and ch-env images inside it. |
| Update path | The device pulls a new image and refreshes ch at boot. |
CodeHerder rebuilds the pool image only when the stage runtime image or a runner input changes. A microVM hands over to the current ch before it takes a session. |
| Lifetime | Long-lived. | Short-lived. Each microVM serves sessions, then ends. |
See Launch a device on AWS, MicroVM runners and Devices that clean up after themselves.
Known limits
- The mirror is writable. Every sandbox of the same workspace and repository shares it. A hostile stage can change it.
- A read-only stage has open egress by default. Set
EGRESS_MODEto change it. - A change to packages or fragments has no fallback image. The stage waits for the build.
- A device in host mode has none of the container controls on this page.
- The Docker daemon pulls images for a stage. The stage egress filter does not apply to that pull.
- The Docker socket proxy does not limit image registries. This matters only if the device server is compromised.
- A pinned version that the device has not cached needs the vendor hosts to be reachable the first time. Without them, the spawn is refused.
See Known limits of the container option for the first reading of these limits.
Related guides
- Giving stages the tools they need: the how-to for tools, packages and
before_stage - Running a stage in your own container image: a custom image per stage
- Isolating agent runs on a device: what a stage can reach
- Network access approvals: approve a blocked host
- Package and browser downloads in container stages: the pull cache
- Git credentials on a device: how git access reaches a stage
- MicroVM runners: short-lived AWS devices
- Launch a device on AWS: an EC2 device
- Devices that clean up after themselves: throwaway devices
Last updated