The pipeline had been failing on image pulls, intermittently, for two months. The cause was a rate limit on anonymous pulls from the public registry, shared across every runner on the same egress address, and the fix we tried first was the most expensive of the three available.
The symptom
$ gh run list --workflow=ci --limit=200 --json conclusion,startedAt
| jq -r '.[] | select(.conclusion=="failure") | .startedAt[11:13]'
| sort | uniq -c | sort -rn | head -4
18 09
6 14
4 10
2 11
# and the message
toomanyrequests: You have reached your pull rate limit.Eighteen of thirty failures in the nine o’clock hour, which is when everybody pushes what they wrote the previous evening. The limit is per address and the runners share one, so the effective quota is divided by however many people are working.
Why it happens
Anonymous pulls are limited and a CI runner is anonymous by default. Nothing about the setup is wrong; the quota simply was not sized for a team pushing in bursts, and the failure is intermittent enough to look like flakiness rather than like a limit.
The fix
The three options
authenticate
a token in the pipeline. raises the quota
substantially. costs: a credential to rotate, and
it does not help the local development pulls.
mirror (pull-through cache)
one container. transparent — no image references
change. costs: a service on the critical path.
host a full registry
push our own copies of everything. total control.
costs: storage, garbage collection, authentication,
and being the reason a build fails.
we did the third one.The third was chosen because it sounded like the thorough answer and because “we should not depend on an external registry” is an easy sentence to say in a meeting. It is the right answer for an organisation with a compliance requirement about image provenance, and it was not the problem we had.
What three weeks of running one taught us
services:
registry:
image: registry:2
environment:
REGISTRY_STORAGE_DELETE_ENABLED: 'true'
REGISTRY_AUTH: htpasswd
REGISTRY_AUTH_HTPASSWD_REALM: turkerdev
REGISTRY_AUTH_HTPASSWD_PATH: /auth/htpasswd
volumes:
- registry-data:/var/lib/registry
- ./auth:/auth:roweek 1 set up, mirrored 14 images by hand, pointed
the pipeline at it. worked.
week 2 storage at 40 GB and growing. discovered that
deleting a tag does not free anything without
a garbage collection run, which requires the
registry to be read-only while it runs.
week 3 a Saturday: the disk filled. every build
failed, including the one that would have
deployed the fix.
and the finding that ended it: REGISTRY_AUTH
had been configured for push and the read path
was open. anybody who could reach the host
could pull our images.The open read path is the one that mattered. It was on an internal network and it was not what anybody had intended, and the reason nobody noticed is that pulling worked — which is exactly what it would do if the authentication were correct. A registry is a piece of infrastructure with a security posture, and we had taken it on as a caching problem.
The cache, which took an afternoon
services:
registry-cache:
image: registry:2
environment:
REGISTRY_PROXY_REMOTEURL: https://registry-1.docker.io
REGISTRY_PROXY_USERNAME: ${DOCKERHUB_USER}
REGISTRY_PROXY_PASSWORD: ${DOCKERHUB_TOKEN}
REGISTRY_STORAGE_DELETE_ENABLED: 'true'
volumes: [registry-cache:/var/lib/registry]// /etc/docker/daemon.json, on every runner
{
"registry-mirrors": ["http://registry-cache.internal:5000"]
}
// and nothing else changes. every FROM, every pull,
// every compose file is untouched — the daemon
// substitutes the mirror transparently and falls back
// to the upstream if the mirror is unreachable.The transparency is the whole argument. No image reference changes, no pipeline is edited, and a mirror that goes down produces a slower build rather than a failed one — the daemon falls back, which is behaviour the self-hosted registry could not have had.
The credentials on the mirror
the proxy credentials are the interesting part: the
cache authenticates upstream, so every pull through it
counts against an authenticated quota rather than the
anonymous one.
which means one token, in one place, raises the limit
for every runner and every developer — rather than a
token per machine.
and the failure mode: a rotated token silently drops
everything back to anonymous, and the rate limit
returns two months later.
→ a check that pulls an image and asserts the response
header showing the authenticated quota.Storage, which still needs managing
# monthly, and the registry must be read-only during it
docker compose stop registry-cache
docker compose run --rm registry-cache
registry garbage-collect /etc/docker/registry/config.yml
docker compose start registry-cache
# 41 GB → 12 GB, and about four minutes
# scheduled at 04:00 on the first SundayA cache still grows without bound, so the garbage collection problem does not go away — what goes away is the authentication surface and the manual mirroring. Four minutes of read-only once a month is acceptable for something on the critical path, and it is scheduled when nothing is building.
Verifying it worked
$ gh run list --workflow=ci --limit=200 --json conclusion
| jq -r '.[].conclusion' | sort | uniq -c
198 success
2 failure # both genuine test failures
$ ./bin/pipeline-duration --median --since=30d
4m 10s # was 6m 40s — the cache is
# faster than the upstream
# the fallback, tested deliberately
$ docker compose stop registry-cache
$ docker pull php:8.3-fpm
# succeeds, from upstream, 40s instead of 4s
# and the quota check
$ ./bin/check-registry-quota
ratelimit-limit: 5000;w=21600 # authenticatedStopping the cache and confirming a pull still succeeds is the test that distinguishes this arrangement from the one it replaced. The pipeline getting faster was not a goal — a local cache serving a layer over a gigabit link beats a public registry every time — and it is the reason nobody has asked to remove it.
What this costs
A cache on the critical path for every build, with a documented fallback that is one line in a daemon configuration and that nobody will remember exists. The health check on the mirror is now as important as the mirror, and a mirror that is up but returning stale manifests is a failure mode we have not tested.
The three weeks are the more useful part of this to record. Running a registry properly is a real piece of work with a security surface, storage management and an availability requirement, and we took it on to solve a rate limit — the option that was transparent, reversible and an afternoon was available from the start and looked insufficiently serious.