Docker 20.10 and the cgroup transition

The PHP container had a 512 megabyte limit and PHP believed it had 64 gigabytes, because /proc/meminfo inside a container is the host’s. That has always been true and became worth fixing when the hosts started moving to cgroup v2, where the numbers live somewhere different again.

The symptom

$ docker run --rm -m 512m app:latest free -m
              total        used        free
Mem:          64267        8102       41220

$ docker run --rm -m 512m app:latest nproc
32

# so a worker pool sized from nproc starts 32 processes
# in a container that can hold about four.

$ docker run --rm -m 512m app:latest php -r 'echo PHP_INT_MAX;'
# fine. but every library that reads meminfo is wrong.

Anything sizing itself from the machine — a worker count, a connection pool, a cache size — reads the host and configures for the host. The container then hits its limit and is killed by the OOM killer with exit code 137 and no message.

Why it happens

A container is a set of namespaces plus a cgroup, and /proc is not namespaced for these values. The limit is enforced by the cgroup and reported nowhere that a normal program looks, which is a design consequence rather than a bug.

cgroup v2 changes where the limit is readable, so code that learned to check /sys/fs/cgroup/memory/memory.limit_in_bytes finds nothing on a v2 host and falls back to the host value — which is the same bug with an extra step.

The fix

Reading the limit from where it actually is

function turkerdev_container_memory_limit(): ?int
{
    // cgroup v2
    $v2 = '/sys/fs/cgroup/memory.max';

    if (is_readable($v2)) {
        $raw = trim((string) file_get_contents($v2));

        return $raw === 'max' ? null : (int) $raw;
    }

    // cgroup v1
    $v1 = '/sys/fs/cgroup/memory/memory.limit_in_bytes';

    if (is_readable($v1)) {
        $raw = (int) trim((string) file_get_contents($v1));

        // v1 reports a huge sentinel when unlimited
        return $raw > (1 << 62) ? null : $raw;
    }

    return null;
}

The two sentinel values are the part that catches people: v2 writes the literal string max and v1 writes a number close to the maximum signed 64-bit value. Treating either as a real limit produces a pool sized for nine exabytes.

Returning null rather than a default lets the caller decide, which matters because the correct fallback differs — a worker pool should refuse to start without a known limit, and a cache should pick something conservative.

# and the CPU quota, which is what nproc should be reading
quota=$(cat /sys/fs/cgroup/cpu.max 2>/dev/null | cut -d' ' -f1)
period=$(cat /sys/fs/cgroup/cpu.max 2>/dev/null | cut -d' ' -f2)

if [ "" != "max" ] && [ -n "" ]; then
  echo $(( quota / period ))
else
  nproc
fi

Or letting the runtime do it

some runtimes already know, and it is worth checking
before writing the cgroup reader:

  the JVM        -XX:+UseContainerSupport, on by default
                 since 10. reads the cgroup.

  Node           does NOT. os.totalmem() is the host.
                 --max-old-space-size must be set explicitly.

  PHP            memory_limit is per-process and unrelated
                 to the container limit. sizing the pool is
                 the application's job.

  PHP-FPM        pm.max_children from the limit, by hand.
                 there is no dynamic mode that knows.

PHP-FPM is the one that matters most in this stack, because pm.max_children multiplied by the per-process memory is what determines whether the container is killed. Computing it at startup from the cgroup limit rather than hardcoding it is twenty lines in an entrypoint and removes a whole category of production surprise.

# docker/php/entrypoint.sh
limit=$(cat /sys/fs/cgroup/memory.max 2>/dev/null 
     || cat /sys/fs/cgroup/memory/memory.limit_in_bytes)

if [ "" != "max" ]; then
  per_child=$(( 48 * 1024 * 1024 ))
  children=$(( (limit * 80 / 100) / per_child ))
  sed -i "s/^pm.max_children = .*/pm.max_children = ${children}/" 
    /usr/local/etc/php-fpm.d/www.conf
fi

exec "$@"

The eighty per cent headroom is the part that is a judgement rather than arithmetic: the container also holds nginx, the opcache and whatever the operating system buffers, and sizing children to the full limit means the OOM killer arrives at peak rather than never.

What else 20.10 brought

cgroup v2 support   the reason this matters now. fedora 31+
                    and ubuntu 21.10 default to v2, and a
                    v1-only assumption breaks silently there.

rootless mode       the daemon as a normal user. genuinely
                    useful on shared build machines, and it
                    costs: no privileged ports, slower
                    networking, overlayfs needs a recent
                    kernel.

the compose plugin  a subcommand rather than a separate
                    binary. still beta; the python
                    docker-compose is what runs in anger.

CONTAINER_HOST      remote contexts over SSH, properly.

Rootless mode is the headline and is the one to be least hasty about. On a build machine shared by six people it is a real security improvement; on a production host it changes the networking path and the storage driver, and both differences show up as performance rather than as errors.

The cgroup v2 support is the one that forces the upgrade eventually, because a host distribution defaulting to v2 with an older Docker gets no resource limits at all — the flags are accepted and enforce nothing, which is worse than failing.

Verifying it worked

$ docker run --rm -m 512m app:latest 
    php -r 'echo turkerdev_container_memory_limit();'
536870912

$ docker run --rm -m 512m app:latest 
    grep max_children /usr/local/etc/php-fpm.d/www.conf
pm.max_children = 8

# and on a v1 host, which is most of them in 2020
$ docker run --rm -m 512m app:latest sh -c 
    'test -f /sys/fs/cgroup/memory.max && echo v2 || echo v1'
v1

# the load test that used to end in 137
$ hey -n 5000 -c 50 https://staging/checkout
  Status code distribution: [200] 5000 responses

Testing on both cgroup versions is the part that needs deliberate effort, because the development machines and the production hosts were both v1 and the failure only appears on a v2 host. A CI job on a newer base image is the cheapest way to cover it.

The load test ending in five thousand successful responses rather than a container restart is the outcome. Exit code 137 with no log line is the specific failure this removes, and it is one of the least informative failures in the whole stack.

What this costs

An entrypoint script doing arithmetic that used to be a constant in a configuration file, which is harder to read and easier to get subtly wrong. The forty-eight megabytes per child is an estimate that will drift as the application grows, and nothing recomputes it — so the sizing is correct at the moment it was measured and decays from there.

The deeper issue is that this is a workaround for containers not virtualising /proc, and every language runtime and library has to solve it independently. lxcfs exists to do it properly and is another component to run; the honest position is that reading the cgroup is the pragmatic answer for one application and does nothing for the third-party library that reads /proc/meminfo and sizes its own buffer.