When a merge request's pipeline fails
Classify every failed CI job first. Then retry, wait, fix or ask, as the class says. Also covers a pipeline that never started.
At the merge stage, the agent checks the pipeline on the request’s current head commit. Most red pipelines have a failed job to trace. Some never create any jobs. This page covers both cases. The merge-stage agent reads it with ch support show ci-triage. You can run the same command.
Classify every failed job first
Do this before you retry, fix or block. A red pipeline is not always a defect in the change. Many failures are infrastructure faults. They need a retry, not a person.
- Get each failed required job from the host. Read its
failure_reasonand its trace. On GitLab, useGET projects/:id/jobs/:job_idandGET projects/:id/jobs/:job_id/trace. On GitHub, usegh run view <run> --log-failed. - List the changed files, for example with
git diff --name-only <base>...HEAD. - Run
ch task ci-classify --job-name <job> --sha <head sha> --failure-reason <reason> --trace-file <trace> --changed-files-file <files>. Add--job-urland--pipelineas evidence. The server records the class and evidence on the task.
| Class | Meaning | Action |
|---|---|---|
infrastructure |
A timeout, a runner fault, a git fetch stall, a rate limit or a failed downstream pipeline. | retry_job: retry only that failed job after the wait named. Keep the passing jobs. It uses no retry budget and needs no person. The cap is 3 per task, commit and job. |
persistent_infrastructure |
The same infrastructure failure after the automatic retries. | timed_wait: run ch task block --for 30m, retry when it releases, and report the pattern. Never an operator blocker. |
flaky |
A test outside the diff fails, and a flake bug is open or main fails the same way. | retry_job once. Link the flake bug. Then continue. |
outside_diff |
A test outside the diff fails and nothing proves it is flaky. | confirm_flake: search for the test name, check main, file or link a flake bug. Classify again with --flake-bug <taskId> or --fails-on-main <evidence>. |
change |
An assertion or a compile error in code the change touches. | fix_change: fix it, or send the task back to code. |
policy |
A quota, a protected environment or a permission. | ask_operator: only this class permits ch agent ask. |
unclassified |
No rule matched. | triage: read the trace and judge it. Use the deterministic-failure rules. Do not ask an operator. |
To retry one failed job on GitLab, use POST projects/:id/jobs/<job_id>/retry. On GitHub, use gh run rerun <run> --failed.
A slow pipeline is never a reason for an operator blocker. An operator blocker is for a real policy decision, such as a raised budget or a skipped required check.
A pipeline with zero jobs
A red or cancelled pipeline with no jobs means your code never ran. The git host refused to create the jobs. On GitLab, this can happen when the project reaches its active-job limit.
A refused pipeline has one of two shapes on GitLab:
- It shows as FAILED with zero jobs.
- It shows as SKIPPED with one job: an old
CodeHerdercommit status. A pipeline made only by commit statuses, with sourceexternal, is the normal shape of a repo with no CI. It is not a refusal.
The REST pipeline resource often shows no yaml_errors and no failure_reason for this case. Retrying the pipeline gives another empty pipeline. A retry does not create jobs.
Step 1: Find the real cause
If the REST resource shows zero jobs and no error fields, ask the GraphQL API. It reports failureReason:
query {
project(fullPath: "<namespace>/<project>") {
pipeline(iid: <pipelineIid>) {
id
status
yamlErrors
failureReason
}
}
}
<pipelineIid> is the pipeline’s own project-local number. It differs from the merge request’s number. It also differs from the pipeline’s global id, which the retry endpoint takes. Read it from the iid field of GET projects/:id/pipelines/:pipeline_id. Keep these numbers apart, and keep the commit SHA you track apart from all of them.
A capacity refusal often reads like this: “The pipeline job activity limit was exceeded.”
Step 2: Classify it
- Configuration defect.
yamlErrorsis not empty, orfailureReasondescribes the pipeline definition. This is a real source problem. Fix it and push. Or send the task back to code, and quote the exact text. - Admission or capacity refusal. The host refused to create jobs. This is not a code defect. Leave the diff alone. Do not send the task back to code. Go to step 3.
Step 3: Wait, then recreate
Use this for capacity refusals. The host does not enforce these bounds. The policy sets them so the wait ends. Stay in the same merge-stage session. Do not end it just to check again later.
- Wait. Wait at least 2 minutes before the first recheck. If the refusal continues, roughly double the wait each time. Never wait more than 10 minutes between checks. Stop waiting after 30 minutes in total. Rapid polling spends the capacity you are waiting for.
- Recreate. After the 30 minutes, if the refusal persists, create a new pipeline. On GitLab, use
POST /projects/:id/merge_requests/:iid/pipelines. Do not usePOST .../pipelines/:id/retry. It replays the same empty job set. Recreate at most twice. Space the two attempts by the same backoff, never back to back. - Cancel with care. Cancel only an older pipeline of the same request that a newer head has replaced, and only if you have the right to cancel it. Never cancel an unrelated validation pipeline. Never cancel a protected release or deploy pipeline. Never cancel a pipeline only because it looks old.
The steps name GitLab endpoints. On another git host, use that host’s equivalent call to create a pipeline.
Step 4: Escalate
After the full wait and both recreations, stop retrying. Raise it with ch agent ask. Name the current commit SHA, the pipeline id and iid, and the exact cause text. Do not loop the task back to code. The code is not the cause, so it fails the same way each time.
A red full, test or test_race is a trigger job
A merge request pipeline runs stage 2 in ONE child pipeline. Its full bridge starts that child
when every fast job has passed, and takes its status (strategy: depend). The child holds
test_plan, test, race_plan, test_race, test_spa, test_spa_browser, test_pg_runner and
build_microvm_runner_stage. The Go test and test_race jobs in the child do not run tests
either: each starts a grandchild pipeline of shards. When full is red, find the failed job:
- Call
GET projects/:id/pipelines/:pipeline_id/bridges. Find the bridge namedfull. - Read its
downstream_pipeline.id. CallGET projects/:id/pipelines/<child id>/jobsand trace the failed job there. - If
testortest_racefailed, callGET projects/:id/pipelines/<child id>/bridges, read that bridge’sdownstream_pipeline.id, and trace the failedtest N/Mjob in that pipeline.
GET pipelines/:id/jobs on a pipeline does not list its child’s jobs. The test_plan job in the
full child shows the selection (selected-packages.txt) and the shard count.
A bridge with failure_reason: downstream_pipeline_creation_failed can be an admission refusal. GitLab refused the child pipeline, so no test ran. It is not a code defect. The cause does
not show on the bridge. Check the bridge’s downstream_pipeline. A refused child is failed and has
zero jobs. Its GraphQL failureReason reads “The pipeline job activity limit was exceeded.”
- Wait as in step 3. Then retry the bridge whose child was refused (
full,testortest_race).
A bridge with no downstream_pipeline does not name a cause. A failed lint job next to a refused
child is a real failure. Trace it as usual.
What you can do
The agent’s question or note appears where you see agent input. See When an agent needs your input for where to find it and how to answer.
The fix sits on the git-host side. Free some CI capacity, or raise the project’s job limit in GitLab. Then answer the agent so it can recreate the pipeline. If a task stays at the merge stage, Why isn’t my task moving? lists other causes.
Related guides
- When an agent needs your input — how questions and notes reach you
- Why isn’t my task moving? — diagnose a stuck task
- How work flows — the stages, including merge
Last updated