Git host request budget
See how many requests your server and devices send to GitLab, why work pauses on a 429, and which server settings tune the pacing.
A shared GitLab token has a request limit. On gitlab.com, many agents and devices can reach it fast. CodeHerder counts every git-host request and paces them on gitlab.com so work slows down before GitLab refuses it. This page shows how the pacing works, where to read the numbers and what to change.
It applies to a self-hosted server. For a full git-host outage, see When GitHub or GitLab is down.
How requests are counted
The server and its devices count each request to the git host. They count it against two things:
- The identity that sent it. This is a connection, the GitHub App, a device’s own token or a short fingerprint of a credential. It is never the token itself.
- The top-level group of the repository. A group is the first part of the repository path.
How GitLab is paced
On gitlab.com the server spends a token from a bucket for each request. A bucket refills at a steady rate. These are the defaults:
| Bucket | Default |
|---|---|
| One identity | 80 requests a minute |
| One group | 350 requests a minute and 4,000 requests an hour |
GitLab allows 100 a minute per identity, 400 a minute per group and 5,000 an hour per group. The defaults leave some room.
When a bucket runs low, the work that matters most goes first:
- Interactive work never queues behind lower work. It waits up to 30 seconds for a token. This covers agent git, provisioning and actions you take.
- Event work waits up to 10 seconds. This covers webhooks, status checks and merge request notes.
- Background work never waits. The server skips it for now, and it runs again later. This covers merge polling, repository reconcile, mirror clean-up and credential probes.
Other git hosts are counted but not paced by these buckets.
What a 429 does
When the git host answers 429, CodeHerder pauses that identity. It does not send more requests with it until the pause ends.
- If the reply names a reset time, the pause lasts until then.
- If it names none, the pause is 30 seconds. Each further
429doubles it, up to 10 minutes. - Up to 20% extra time is added, so devices do not all resume together.
A device that gets a 429 also pauses the repository’s group. It tells the server, and the server tells the group’s other devices. Git, probes and proxied requests all wait.
Paused work parks. It does not count as a failure, and the task keeps its stage. It tries again after the reset. A start that the git host refuses this way shows the reason git_host_rate_limited. The device does not count against the provisioning breaker. See Release a provisioning breaker.
Read the budget
You need an instance operator login. Run:
ch instance githost
For the same data as JSON, add --json. The app shows it under Instance → Git host budget.
The numbers cover the last 60 minutes. Each API process keeps its own window. A restart clears them.
The command prints three tables.
GROUPS has one row for each host and top-level group.
| Column | Meaning |
|---|---|
LAST HOUR |
Requests in the last hour |
CAP/H and CAP/MIN |
The hourly and per-minute limits for the group. — means no limit applies |
PEAK MIN |
The busiest minute in the window |
429s |
Replies of 429 in the last hour |
PAUSED UNTIL (UTC) |
When the pause ends. — means the group is not paused |
TOP SOURCES |
The three biggest causes of requests, with counts |
IDENTITIES has one row for each host and identity. It has the same columns, without the hourly cap. It adds REMAINING/LIMIT, the quota the host last reported for that identity.
BUCKETS has one row for each bucket window.
| Column | Meaning |
|---|---|
SCOPE and KEY |
group or identity, and which one |
WINDOW |
minute or hour |
TOKENS/BURST |
Tokens left and the most the bucket holds |
LIMIT |
The refill limit for the window |
SHED |
Requests dropped in the last hour, by class |
WAITED |
Requests that waited in the last hour, by class |
Hosts the server does not pace have no BUCKETS rows.
Change the pacing
Set these in the server environment. Restart the server to apply them. Self-hosted servers keep the defaults unless you change them. See Self-hosted deployment.
| Setting | Default | What it does |
|---|---|---|
CH_GITLAB_IDENTITY_PER_MIN |
80 |
Most gitlab.com requests a minute for one identity |
CH_GITLAB_GROUP_PER_MIN |
350 |
Most gitlab.com requests a minute for one group |
CH_GITLAB_GROUP_PER_HOUR |
4000 |
Most gitlab.com requests an hour for one group |
CH_GITHOST_PACE_BACKOFF_MIN |
30s |
First pause after a 429 with no reset time |
CH_GITHOST_PACE_BACKOFF_MAX |
10m |
Longest pause after a 429 with no reset time |
CH_GITLAB_MINTS_PER_MIN |
12 |
Most sandbox token requests a minute for one GitLab connection. 0 turns the limit off |
CH_GITHOST_BUDGET_LOG_INTERVAL |
1h |
How often the server logs one line of request totals for each group |
Do not set a limit above what GitLab allows for your plan. The server then hits 429 instead of pacing itself. A self-managed GitLab sets its own limits, and these buckets do not apply to it.
When work is paused
- Run
ch instance githost. Look for a value inPAUSED UNTIL (UTC). - Wait for that time. Parked tasks resume by themselves. You do not need to unblock them.
- If
429skeeps rising, look atTOP SOURCES. One source that dominates tells you which work uses the budget. - If
SHEDorWAITEDlistsinteractiveoreventin the last hour, the limits may be too low for your load. Raise them, within what GitLab allows.
Limits
- The token buckets cover gitlab.com only.
- Pauses and counts live in memory. A restart clears them. A new
429sets a fresh pause. - With more than one API process, each keeps its own window and pauses.
- The page and the command show counts. They never show a token or a repository path.
Last updated