A registry cache that filled a disk on a Sunday

A pull-through cache with no garbage collection schedule, and a disk that filled at two in the morning.

# the job that should have existed from the start
docker compose stop registry-cache
docker compose run --rm registry-cache 
  registry garbage-collect /etc/docker/registry/config.yml
docker compose start registry-cache

# 78 GB → 14 GB, four minutes, first Sunday of the month

# and the alert that should also have existed
- alert: RegistryCacheDiskHigh
  expr: node_filesystem_avail_bytes{mountpoint="/srv/registry"} < 20e9

A cache grows without bound by definition and the only surprise is which Sunday. The failure mode is worse than it sounds — a full disk on the mirror means every build fails, including the one that would deploy the fix, which is the same shape as the self-hosted registry incident from last year and was not learned from.