The CI runner mounted the Docker socket so that jobs could build images, which is what every guide recommends and is equivalent to giving every job root on the host. Nobody had said that out loud, and once it was said the conversation became short. Docker 19.03 in July is the release where there are answers.
The symptom
# what "the runner can build images" actually permits
$ docker run -v /:/host -it alpine chroot /host sh
# you are root on the host. no exploit, no escape — this is the API.
$ cat .gitlab-ci.yml | grep -A2 volumes
volumes = ["/var/run/docker.sock:/var/run/docker.sock"]
$ git log --format='%an' --all | sort -u | wc -l
14 # any of whom can open a merge request that runs on that runnerThere is no permission model on the socket. It is all or nothing, and every socket-mounting container is fully privileged whether or not anyone has decided that. On a repository that accepts merge requests from forks it is a straightforward remote code execution path with the machine’s own credentials attached.
Why it happens
The daemon runs as root because it needs to create namespaces, configure networking and mount filesystems, and the socket is its complete API. Building an image requires talking to the daemon, so building an image requires the socket, so a CI job that builds requires root — the chain is short and each link is reasonable.
Docker-in-Docker replaces one problem with another: the inner daemon needs --privileged, which is at least as broad. The real fix is a builder that is not a root daemon, and that is what BuildKit as a standalone component makes possible.
The fix
BuildKit, which is the default now and faster for a reason
$ DOCKER_BUILDKIT=1 docker build -t app .
[+] Building 18.4s (14/14) FINISHED
=> [builder 3/5] RUN composer install --no-dev 11.2s
=> [assets 3/4] RUN npm ci 9.8s ← concurrent
=> [stage-2 4/6] COPY --from=assets /app/public/build 0.3s
# same Dockerfile, old builder: 31.6s. the two stages ran in series.
# and permanently, in /etc/docker/daemon.json:
# { "features": { "buildkit": true } }The old builder executes instructions in order on one goroutine; BuildKit builds a dependency graph, runs independent branches concurrently and skips stages nothing needs. On any multi-stage build with independent stages that concurrency is free, and a PHP stage and an asset stage are almost always independent.
Cache mounts and build secrets, which need the new frontend
# syntax=docker/dockerfile:1.1-experimental
FROM php:7.3-fpm-alpine
COPY composer.json composer.lock ./
# a cache that lives outside the image and survives between builds
RUN --mount=type=cache,target=/root/.composer/cache
composer install --no-dev --no-scripts --no-autoloader
# a secret mounted for one instruction, never written to a layer
RUN --mount=type=secret,id=composer_auth
COMPOSER_AUTH="$(cat /run/secrets/composer_auth)"
composer install --no-devThe syntax directive at the top selects a frontend that understands these flags, and forgetting it produces a parse error rather than a helpful message. The cache mount is the more immediately valuable of the two: a lock file change re-resolves without re-downloading eighty packages, and the cache never appears in the image.
The secret mount solves a problem that is usually solved badly. Passing a private repository token as a build argument puts it in the image history where docker history will read it back out, and deleting the file in a later instruction does not help because layers are additive. This is the only mechanism that actually works.
$ docker build --secret id=composer_auth,src=./auth.json .
$ docker history app --no-trunc | grep -c COMPOSER_AUTH
0
# whereas, with a build arg:
$ docker history app-old --no-trunc | grep COMPOSER_AUTH
|1 COMPOSER_AUTH={"github-oauth":{"github.com":"ghp_..."}}Rootless, and the four things that stop working
$ dockerd-rootless-setuptool.sh install
$ export DOCKER_HOST=unix:///run/user/1000/docker.sock
$ docker info | grep -i rootless
rootless
# what stops working:
# ports below 1024 setcap on rootlesskit, or a proxy in front
# --net=host a separate network namespace
# cgroup resource limits needs cgroup v2 and systemd
# overlayfs on some kernels → fuse-overlayfs, measurably slowerThe security gain is specific and real: a container escape lands the attacker as an unprivileged user rather than as root on the host. For a CI runner building untrusted branches that is a meaningful difference and probably worth the limitations.
For a single-purpose production host running only your own application, the port and cgroup restrictions usually outweigh it in 2019. The honest position is that this is early — it works, the caveats are real, and trialling it on build machines before anything else is the sensible sequence.
The other answer: a builder that is not a daemon
# the runner no longer needs the socket at all
build:
image:
name: moby/buildkit:rootless
entrypoint: ["sh", "-c"]
variables:
BUILDKITD_FLAGS: --oci-worker-no-process-sandbox
script:
- buildctl-daemonless.sh build
--frontend dockerfile.v0
--local context=.
--local dockerfile=.
--output type=image,name=/app:,push=trueA build that produces and pushes an image with no daemon and no privileged container is the arrangement that removes the original problem rather than mitigating it. The command line is uglier than docker build and the flags are the sort of thing that gets copied without understanding, which is a fair criticism.
This is also where the ecosystem is going, and knowing that in 2019 is worth more than adopting it immediately — the migration is easier when the Dockerfiles are already written for BuildKit, which is a reason to turn the feature flag on now regardless of what builds it.
Verifying it worked
$ docker history app --no-trunc | grep -ciE 'token|secret|auth'
0
$ time docker build -t app .
real 0m18.412s # was 0m31.604s
$ ps -o user= -p $(pgrep -f buildkitd)
gitlab-runner # not root
$ docker run --rm -v /:/host alpine chroot /host sh
docker: Error response from daemon: ... permission deniedThe last command failing is the assertion the whole exercise was for, and it is worth running deliberately rather than assuming — a rootless setup where the old socket is still mounted somewhere has none of the benefit and all of the complexity. Grepping the image history for anything credential-shaped is the other check and it belongs in CI rather than in a person’s memory.
What this costs
Experimental flags in a production pipeline. The syntax directive names an experimental frontend, rootless mode is new, and buildctl is a lower-level tool than docker build with correspondingly less documentation. Each of those is a thing that can change under you, and a build that stops working because an upstream image moved is a bad morning. Pinning the frontend version rather than tracking latest is the mitigation and it is easy to forget.
The larger cost is that the pipeline is now understood by fewer people. docker build is universally known; a daemonless BuildKit invocation with three flags is not, and the person who set it up becomes the person who fixes it. Writing down why each flag is there, in the file, is the cheapest defence — and the security reasoning is the part worth writing down most, because without it somebody will simplify this back to a mounted socket during a deadline.