CodeHerderSearch⌘KRequest access →

When a merge request's pipeline fails

Classify every failed CI job first. Then retry, wait, fix or ask, as the class says. Also covers a pipeline that never started.

At the merge stage, the agent checks the pipeline on the request’s current head commit. Most red pipelines have a failed job to trace. Some never create any jobs. This page covers both cases. The merge-stage agent reads it with ch support show ci-triage. You can run the same command.

Classify every failed job first

Do this before you retry, fix or block. A red pipeline is not always a defect in the change. Many failures are infrastructure faults. They need a retry, not a person.

  1. Get each failed required job from the host. Read its failure_reason and its trace. On GitLab, use GET projects/:id/jobs/:job_id and GET projects/:id/jobs/:job_id/trace. On GitHub, use gh run view <run> --log-failed.
  2. List the changed files, for example with git diff --name-only <base>...HEAD.
  3. Run ch task ci-classify --job-name <job> --sha <head sha> --failure-reason <reason> --trace-file <trace> --changed-files-file <files>. Add --job-url and --pipeline as evidence. The server records the class and evidence on the task.
Class Meaning Action
infrastructure A timeout, a runner fault, a git fetch stall, a rate limit or a failed downstream pipeline. retry_job: retry only that failed job after the wait named. Keep the passing jobs. It uses no retry budget and needs no person. The cap is 3 per task, commit and job.
persistent_infrastructure The same infrastructure failure after the automatic retries. timed_wait: run ch task block --for 30m, retry when it releases, and report the pattern. Never an operator blocker.
flaky A test outside the diff fails, and a flake bug is open or main fails the same way. retry_job once. Link the flake bug. Then continue.
outside_diff A test outside the diff fails and nothing proves it is flaky. confirm_flake: search for the test name, check main, file or link a flake bug. Classify again with --flake-bug <taskId> or --fails-on-main <evidence>.
change An assertion or a compile error in code the change touches. fix_change: fix it, or send the task back to code.
policy A quota, a protected environment or a permission. ask_operator: only this class permits ch agent ask.
unclassified No rule matched. triage: read the trace and judge it. Use the deterministic-failure rules. Do not ask an operator.

To retry one failed job on GitLab, use POST projects/:id/jobs/<job_id>/retry. On GitHub, use gh run rerun <run> --failed.

A slow pipeline is never a reason for an operator blocker. An operator blocker is for a real policy decision, such as a raised budget or a skipped required check.

A pipeline with zero jobs

A red or cancelled pipeline with no jobs means your code never ran. The git host refused to create the jobs. On GitLab, this can happen when the project reaches its active-job limit.

A refused pipeline has one of two shapes on GitLab:

  • It shows as FAILED with zero jobs.
  • It shows as SKIPPED with one job: an old CodeHerder commit status. A pipeline made only by commit statuses, with source external, is the normal shape of a repo with no CI. It is not a refusal.

The REST pipeline resource often shows no yaml_errors and no failure_reason for this case. Retrying the pipeline gives another empty pipeline. A retry does not create jobs.

Step 1: Find the real cause

If the REST resource shows zero jobs and no error fields, ask the GraphQL API. It reports failureReason:

query {
  project(fullPath: "<namespace>/<project>") {
    pipeline(iid: <pipelineIid>) {
      id
      status
      yamlErrors
      failureReason
    }
  }
}

<pipelineIid> is the pipeline’s own project-local number. It differs from the merge request’s number. It also differs from the pipeline’s global id, which the retry endpoint takes. Read it from the iid field of GET projects/:id/pipelines/:pipeline_id. Keep these numbers apart, and keep the commit SHA you track apart from all of them.

A capacity refusal often reads like this: “The pipeline job activity limit was exceeded.”

Step 2: Classify it

  • Configuration defect. yamlErrors is not empty, or failureReason describes the pipeline definition. This is a real source problem. Fix it and push. Or send the task back to code, and quote the exact text.
  • Admission or capacity refusal. The host refused to create jobs. This is not a code defect. Leave the diff alone. Do not send the task back to code. Go to step 3.

Step 3: Wait, then recreate

Use this for capacity refusals. The host does not enforce these bounds. The policy sets them so the wait ends. Stay in the same merge-stage session. Do not end it just to check again later.

  • Wait. Wait at least 2 minutes before the first recheck. If the refusal continues, roughly double the wait each time. Never wait more than 10 minutes between checks. Stop waiting after 30 minutes in total. Rapid polling spends the capacity you are waiting for.
  • Recreate. After the 30 minutes, if the refusal persists, create a new pipeline. On GitLab, use POST /projects/:id/merge_requests/:iid/pipelines. Do not use POST .../pipelines/:id/retry. It replays the same empty job set. Recreate at most twice. Space the two attempts by the same backoff, never back to back.
  • Cancel with care. Cancel only an older pipeline of the same request that a newer head has replaced, and only if you have the right to cancel it. Never cancel an unrelated validation pipeline. Never cancel a protected release or deploy pipeline. Never cancel a pipeline only because it looks old.

The steps name GitLab endpoints. On another git host, use that host’s equivalent call to create a pipeline.

Step 4: Escalate

After the full wait and both recreations, stop retrying. Raise it with ch agent ask. Name the current commit SHA, the pipeline id and iid, and the exact cause text. Do not loop the task back to code. The code is not the cause, so it fails the same way each time.

A red full, test or test_race is a trigger job

A merge request pipeline runs stage 2 in ONE child pipeline. Its full bridge starts that child when every fast job has passed, and takes its status (strategy: depend). The child holds test_plan, test, race_plan, test_race, test_spa, test_spa_browser, test_pg_runner and build_microvm_runner_stage. The Go test and test_race jobs in the child do not run tests either: each starts a grandchild pipeline of shards. When full is red, find the failed job:

  1. Call GET projects/:id/pipelines/:pipeline_id/bridges. Find the bridge named full.
  2. Read its downstream_pipeline.id. Call GET projects/:id/pipelines/<child id>/jobs and trace the failed job there.
  3. If test or test_race failed, call GET projects/:id/pipelines/<child id>/bridges, read that bridge’s downstream_pipeline.id, and trace the failed test N/M job in that pipeline.

GET pipelines/:id/jobs on a pipeline does not list its child’s jobs. The test_plan job in the full child shows the selection (selected-packages.txt) and the shard count.

A bridge with failure_reason: downstream_pipeline_creation_failed can be an admission refusal. GitLab refused the child pipeline, so no test ran. It is not a code defect. The cause does not show on the bridge. Check the bridge’s downstream_pipeline. A refused child is failed and has zero jobs. Its GraphQL failureReason reads “The pipeline job activity limit was exceeded.”

  • Wait as in step 3. Then retry the bridge whose child was refused (full, test or test_race).

A bridge with no downstream_pipeline does not name a cause. A failed lint job next to a refused child is a real failure. Trace it as usual.

What you can do

The agent’s question or note appears where you see agent input. See When an agent needs your input for where to find it and how to answer.

The fix sits on the git-host side. Free some CI capacity, or raise the project’s job limit in GitLab. Then answer the agent so it can recreate the pipeline. If a task stays at the merge stage, Why isn’t my task moving? lists other causes.

Last updated

Try it on one repository for two weeks.

File a task. An agent builds it. A person reads every request. You see the cost of each task.

Try "pricing", "connect a device", or "who reviews the code"

↑↓ move · ↵ open · esc close