Recover from a half-applied self-host upgrade

An upgrade has three parts: the server, the webapp, and the devices. Any part can stop before the others. This page tells you how to finish the upgrade from the state you are in. It never asks you to go back.

The rule is to move forward. Fix the cause, then start the same new release again. To go back to an older release, use the rollback steps in Upgrade to a new release.

The order to follow

Upgrade in this order:

1. The server.
2. The webapp.
3. The devices.

Take the server and the webapp from the same manifest. Take the devices after both.

Find your case

What you see	Go to
The installer printed an error and stopped	Case 1
The new binary is on disk, but the server does not start	Case 2
The log shows migrations were applied, then the start failed	Case 3
The server is new, but the dashboard is old, or the reverse	Case 4
Some devices run the new release and some do not	Case 5

Case 1: the download or the check failed

install-codeherder-server.sh checks the signature first. It then checks the size and the SHA-256 checksum. It writes to the destination only after every check passes. A failed check leaves the destination directory as it was.

1. Read the error message. It names the check that failed.
2. Run the installer again. A brief network fault clears on a second run.
3. If the signature check fails again, stop. Do not use the files. Contact support.

The running server is not touched. It keeps serving the old release.

Case 2: the new server does not start

The new binary is on disk, but the process exits or does not listen.

1. Read the server log. See Self-hosted logs for where it is.
2. Fix the cause. A missing or wrong key, an unreachable database, and a busy port are the common ones.
3. Start the same new binary again, with the same encryption configuration and the same --db value. Keep the existing keys; see Self-hosted key custody.

This is safe to repeat. The server applies each pending migration in its own transaction and records it in the migration ledger. A migration that failed leaves no trace. A migration that finished is skipped on the next start. Two servers that start together take turns on a database lock.

Case 3: the migrations ran, and then the start failed

The log shows postgres migration applied lines, and then the server stopped.

Do not start the old binary on this database. An old server does not refuse a database that has newer migrations. It starts. Its behavior on the newer schema is not defined.

Choose one of these:

• Roll forward. Fix the cause, or install the next release that fixes it, and start that binary on the same database. This is the default.
• Roll back. Follow the rollback steps in Upgrade to a new release. Do not start the old binary until you have done them.

Case 4: the server and the webapp are out of step

The webapp is a separate archive. The server does not check that the two match. You can end up with a new server and an old dashboard, or the reverse.

1. Read the version of the server you run. See Confirm which build is live.
2. Download manifest.json for that version and verify its signature.
3. Download the webapp archive that the same manifest names. Check its size and SHA-256 checksum against the manifest.
4. Deploy the archive. The codeherder server does not serve the webapp. Your web server or reverse proxy serves the files from a directory that you chose when you first installed the webapp. Extract the archive into a new, empty directory. Point the web server at that directory, or replace the old directory’s contents with the new files. Then reload the web server.
5. Reload the dashboard in your browser, so that it loads the new files.

Take the webapp from the same manifest as the server. A webapp from another release can call an API that the server does not have. The compatibility test checks API route availability across one release step. It does not render the dashboard or prove that every new feature works against an older server. Reload and check the features you use after deploying the matching files.

Case 5: some devices are behind

A device runs the release it was on when it last updated. Within the same MAJOR, the server accepts devices up to two minor releases behind, and devices ahead of it. A device that falls behind the patch deadline eventually stops receiving new work. See Supported versions. The compatibility rehearsal checks old and new devices across one release step and reconnection after the server is upgraded in place. Finish updating the fleet before relying on features added in the new release.

1. Finish the server and the webapp first.
2. Upgrade each device. See Updating the CLI.
3. Confirm that each device shows as online in the dashboard.

Run ch start check after updating. A CLI on an older minor line can fail the version check; see Updating the CLI.

Related guides

• Self-hosted deployment — download, verify, run, and upgrade a server
• Self-hosted logs — where to find the server log
• Updating the CLI — what the version notices mean