Docker for Mac replaced the virtual machine

The attempt at this last year nearly failed on one number: 2.9 seconds to render a page that took 120 milliseconds natively, because the project directory reached the container through a VirtualBox shared folder. Docker for Mac went generally available this month and that number is now 380 milliseconds without any of the workarounds.

The symptom

The workarounds were the problem. NFS instead of vboxsf, a container-only volume masking vendor, and a documented setup procedure with six steps — all of which worked and none of which a new developer got right first time.

# 2015: Toolbox, VirtualBox, vboxsf
native                            120 ms
docker, vboxsf                   2900 ms
docker, vboxsf + NFS + no vendor  380 ms

# and the setup for that last line was six manual steps

Why it happens

Toolbox ran a full Linux guest under VirtualBox and bridged the filesystem with vboxsf, a protocol that makes every stat a round trip through the hypervisor. PHP performs thousands of those per request — the autoloader alone checks for a file per class — so the cost multiplies by exactly the wrong factor.

Docker for Mac replaces VirtualBox with the native Hypervisor.framework and vboxsf with osxfs, a filesystem built for this purpose rather than for sharing a folder with a desktop VM. It is still a bridge and it is a much shorter one.

The fix

Almost nothing in the compose file

That is the headline. The migration is uninstalling one thing and installing another.

version: '2'

services:
  php:
    build: ./docker/php
    volumes:
      - .:/var/www
      # - /var/www/vendor      <- the 2015 workaround, no longer needed
  nginx:
    build: ./docker/nginx
    ports:
      - "8080:80"

The container-only vendor volume can go, which removes the most confusing part of the old setup — a directory that existed in the container and not on the host, so an IDE could not see the dependencies it was meant to index.

The other change is that localhost means localhost again. Under Toolbox the stack was at 192.168.99.100, so every developer had a hosts file entry and every README had a paragraph about docker-machine ip.

What the new file sharing costs

osxfs is much faster and it is not free, and the difference is still visible on the operations PHP performs most.

# 40,000 stat calls against the shared mount

linux, native bind mount     0.21s
docker for mac, osxfs        2.94s
toolbox, vboxsf             28.60s

# the same application page
native php-fpm    120 ms
docker for mac    380 ms
linux docker      140 ms

Fourteen times better than vboxsf and still fourteen times slower than a native bind mount on Linux. For a development loop that is comfortably acceptable; for anything running a large test suite against the mounted code it is the dominant cost, and moving the suite into the image rather than the mount is worth measuring.

The beta caveats that are still true

It went GA this month and the edges are visible. File change notification through the mount is unreliable, so anything watching the filesystem — a Sass watcher, a test runner in watch mode — misses events and has to fall back to polling.

// gulpfile.js — polling, because inotify does not cross osxfs reliably
gulp.watch('src/**/*.scss', { usePolling: true, interval: 500 }, ['styles']);

Polling costs CPU and works. It is the single most common “Docker for Mac is broken” report and it is not a Docker bug so much as a boundary that cannot carry inotify events.

Verifying it worked

# clean machine, no prior Docker install
$ git clone git@git:shop.git && cd shop
$ docker-compose up -d
$ docker-compose run --rm php composer install
$ open http://localhost:8080

real    9m41s

# versus 14m22s last year, and six fewer manual steps

The setup steps are what actually improved. Last year’s procedure had a VirtualBox install, a docker-machine creation, an NFS export configuration, a hosts file edit and an environment variable to source in every shell. This year it is install the application and run compose.

What to move off the mount

osxfs is fast enough for the edit-and-reload loop and it is still the dominant cost for anything touching many files at once. The test suite is the obvious case: 1,200 tests reading fixtures and autoloading classes across the bridge takes four times what it takes natively.

# through the shared mount
$ docker-compose run --rm php vendor/bin/phpunit
Time: 6:14

# code baked into the image, no bind mount
$ docker-compose -f docker-compose.ci.yml run --rm php vendor/bin/phpunit
Time: 1:38

# docker-compose.ci.yml
version: '2'

services:
  php:
    build:
      context: .
      dockerfile: docker/php/Dockerfile.ci
    # no volumes: the image contains the application

That is also what CI should be running, so the second compose file earns its keep twice — faster for a full local run, and the same environment the pipeline uses. The development file keeps the bind mount, because a suite that is fast but needs a rebuild to see a change is not a development loop.

The other candidate is node_modules, for the same reason and more so: an install through the mount writes tens of thousands of small files and is measured in minutes. A named volume costs nothing and is invisible to the host, which is right for a directory nobody edits.

What this costs

Linux and Mac have diverged again, in the opposite direction from last year. A Linux developer gets native speed and no polling; a Mac developer gets neither. That is a smaller gap than before and it still means a performance problem reported by one half of the team may not reproduce for the other.

It is also a very new product with a beta channel that people will be tempted onto for a fix, and the two channels have different bugs. Pinning the team to the stable channel and writing the version down is worth doing before the first “works for me” that turns out to be a channel difference.

Windows remains on Toolbox unless the machine has Hyper-V, which rules out Home editions entirely. A mixed team is not finished with the VM yet, and the documentation has to keep both paths until it is.