# Giving stages the tools they need

Source: https://codeherder.com/docs/stage-environments/

Declare the tools, system packages and setup script your stages run with, preview changes before you save, and fix a stage waiting on a build.

An agent can only use the tools its device has. If your project needs `jq`, a Go toolchain, or a Postgres client, you can declare them once and CodeHerder makes sure each stage has them. That declaration is the **environment**. This page covers how to set one up, how to check what a stage gets, and what to do when a stage waits because no device can build it.

If you need a whole custom container image instead, see [Running a stage in your own container image](https://codeherder.com/docs/stage-images/). The two don’t mix for system packages; see **System packages** below.

## How an environment works

You pin each tool to an exact version. CodeHerder checks the pinned set once and records the result. Before a stage starts, the device that will run it installs those exact tools. Two stages with the same tools share one build, so the cost is paid once per device.

An environment is built from layers. From widest to narrowest:

1. **Group.** A group workspace’s environment applies to every workspace below it.
2. **Workspace.**
3. **Workflow.** Applies to every stage of that workflow.
4. **Stage.** Applies to one stage.

The rules are the same everywhere:

- **Tools:** the narrowest layer that names a tool wins. A workflow or stage can also remove a tool that a wider layer declared, for the stages it covers.
- **Setup scripts:** every layer’s script runs, in layer order, widest first. See [Running a stage in your own container image](https://codeherder.com/docs/stage-images/#the-setup-script) for how a setup script behaves.
- **System packages:** the packages from all layers are added together. A narrower layer can’t remove a package a wider one declared.

## Where to manage it

In the web app, go to **Settings → Environment**. The page shows these sections:

- **Inherited from groups:** tools from the groups above this workspace. Read only here.
- **This layer:** the tools this workspace or group declares.
- **Setup script:** the script this layer runs before the agent starts.
- **Workflow and stage overrides:** the layers narrower than the workspace.
- **Effective environment:** what a given stage ends up with.
- **Resolve status** and **Version history:** described below.

A group’s page leaves out the overrides, effective environment and resolve status, because they only make sense for a workspace.

A workflow’s own page and a stage’s own page (**Settings → Stages**) each have an **Environment** section for that layer.

Only a workspace admin who is a person can change an environment. An agent’s credential is refused. Everyone else sees the pages read only.

## Add a tool

In the web app, open **This layer**, choose **Add tool**, then **From registry**. Search for the tool and pick a version from its releases. The **Custom tool** option is for a tool that isn’t in the registry; it asks for a name, version, download URL template, the binaries to put on the command path, a sha256 for at least one platform, and optional replacements.

From the CLI, find the tool and its releases first:

```
ch env search pup
ch env releases DataDog/pup
```

Then pin the version you want:

```
ch env tools add DataDog/pup 1.23.7
```

A tool name is one of:

- a registry name, such as `DataDog/pup`;
- `npm:<package>`, `pip:<package>` or `go:<module>` for a language package.

The version is always exact. A range, or `latest`, is refused. A custom tool can’t be added with `ch env tools add`; write it with `ch env set` instead, which replaces the whole workspace environment from a JSON file. Run `ch env show --json` to see the shape.

To remove a tool this layer declared:

```
ch env tools delete DataDog/pup
```

### Per-workflow and per-stage layers

Add `--workflow <key>` or `--stage <key>` to put a tool in one workflow’s or one stage’s layer instead of the workspace layer:

```
ch env tools add --stage review jq 1.7
ch env tools add --workflow story docker/compose v2.29.7
```

To drop a tool that a wider layer declared, for one stage or workflow only, add `--remove-inherited`:

```
ch env tools delete --stage review --remove-inherited go
```

In the web app, the **Add tool** menu has **Remove an inherited tool** for the same job.

## System packages

If a stage needs an operating system package, declare the Debian package name. You don’t pin a version.

```
ch env system add postgresql-client libvips42
ch env system delete libvips42
ch env system add --stage review jq
```

For more control, `ch env system dockerfile --from-file system.Dockerfile` sets a Dockerfile fragment for the layer. A fragment can’t start a new image with `FROM`, and it has a size limit of 16 KiB.

System packages build on the CodeHerder base image. A write is refused while a stage in scope sets its own container image, and the message names those stages. A device that can’t build the packages shows as ineligible in the device matrix described below.

## Preview a change, then save

Before you save, review what the change reaches. In the web app, **Save changes** stays disabled until you choose **Review changes**. The review lists each stage the change reaches, its tool changes, and the devices that must build.

From the CLI, write your layer to a file and preview it. Nothing is saved:

```
ch env diff --from-file env.json
ch env diff --workflow story --from-file workflow-env.json
ch env diff --stage review --from-file stage-env.json
```

Without `--workflow` or `--stage`, the file is the workspace layer, in the same format as `ch env set` takes. With them, it’s the `environment` object that `ch env show --json` prints for that layer.

## See what a stage gets

The **Effective environment** section picks a workflow and a stage and lists every tool, package and script with the layer it comes from, plus the build state on each device.

From the CLI:

```
ch env show --effective --task <taskId>
ch env show --effective --workflow story --stage review
```

With `--task`, the stage defaults to the task’s current stage. `ch env show` with no flags prints the workspace layer on its own.

## Confirm a source change

After you save, CodeHerder checks each tool again in the background. If a tool’s source moved since the last check, such as a new repository or download host, a person has to confirm it. This stops a changed download location from reaching your devices unnoticed.

A tool set that already had a good result keeps using it until you confirm. A tool set that is new has nothing to fall back on, so its stages wait.

To see what is waiting and confirm it, use the **Resolve status** section, which shows each pending source change and a **Confirm** button. From the CLI:

```
ch env lock
ch env diff
ch env diff --confirm
```

`ch env lock --resolve` starts a fresh check now. Confirming is for workspace admins who are people. An agent’s credential is refused.

## Check the build on each device

Each device builds the environments its stages need. In the web app, open **Devices → Environments**. The matrix has one row per environment and one column per device. Each cell shows one state:

| State | Meaning |
| --- | --- |
| ready | The device has the tools. |
| building | The device is installing them. |
| pending | The device hasn’t started. |
| failed | The build failed. The cell shows the reason and the time of the next retry. |
| ineligible | This device can’t run this environment, for example because of its platform. The cell shows why. |

A failed build retries by itself, with longer waits between tries. To read a failure, open its build log from the cell. To retry sooner, choose **Rebuild** on a failed or ready cell. The device must be online.

The same data is in the CLI:

```
ch env status
ch env status --device build-01
ch env log --device build-01 3f2a9c1d4b7e
ch env rebuild --device build-01 3f2a9c1d4b7e
```

The hash is from `ch env status`. A unique start of it is enough. Only a workspace admin who is a person can rebuild.

## When a stage waits

If no device can run a stage’s tools, the task doesn’t start. Instead, CodeHerder puts a blocker on the task. The blocker starts with “No device can run this stage’s tools”, then names the tool and platform, or points at the failed build. Run `ch task blockers <taskId>` to read it. The task’s activity also shows a **start refused** entry. The blocker clears by itself when a device can run the tools. You don’t need to do anything to restart the task.

To fix it:

1. Run `ch env status` or open **Devices → Environments** and find the failed or ineligible cells.
2. Read the reason, or the build log for a failure.
3. Fix the cause. Common ones are a tool with no build for the device’s platform, or a download that fails. Then choose **Rebuild**, or link a device that can run it.

If the cause is an unconfirmed source change, confirm it as described above. For other start problems, see [Why isn’t my task moving?](https://codeherder.com/docs/task-not-moving/).

Tools only reach stages that CodeHerder starts on a device. A dev session and a task you run on your own machine (see [Run a task on your own machine](https://codeherder.com/docs/local-tasks/)) don’t get the workspace’s tools.

## History and revert

Every save writes a new version. **Version history** on **Settings → Environment** lists them, and the workflow and stage pages show their own layer’s history.

```
ch env versions
ch env versions --stage review
ch env revert 1
ch env revert --workflow story 2
```

A revert doesn’t rewind history. It writes the old content as a new version, so you can see what happened. `ch env revert --stage review 2` restores only that stage’s environment and leaves the rest of the stage as it is now.

If someone else saves while you edit, your save is refused. Reload and apply your change again.

## Related guides

- [Running a stage in your own container image](https://codeherder.com/docs/stage-images/): a custom image and the setup script
- [Managing your devices](https://codeherder.com/docs/devices/): readiness, capacity and linking a device
- [Why isn’t my task moving?](https://codeherder.com/docs/task-not-moving/): every reason a task waits
- [Customising workflows](https://codeherder.com/docs/task-types/): where workflow and stage settings live