The audit asked a simple question — where is the production database password — and the answer took forty minutes to assemble. It was in a .env.production committed in 2017 and since deleted but still in the history, baked into a Docker image from 2019, in the CI settings, in a shared password manager entry, and in a Slack message from the person who set it up.
The symptom
$ git log --all --oneline -S 'DB_PASSWORD=' -- .env.production
a41f2b8 remove committed env file
8c9e114 add production env
$ git show 8c9e114:.env.production | grep DB_PASSWORD
DB_PASSWORD=hunter2-real-one
# deleted three years ago. still in every clone.
$ docker history app:2019-11 --no-trunc | grep -c 'ENV DB_PASSWORD'
1
$ docker exec app-01 env | grep -c PASSWORD
4A deleted file is not a removed secret — it is a secret with an extra step. Every clone of the repository, every CI cache and every fork has it, and the only thing that removes it is rotating the credential.
Why it happens
Every one of those locations was the reasonable answer to a question at the time it was chosen. The committed file made deployment work when deployment was a git pull. The image build argument made it work when deployment became a container. The environment variable is what twelve-factor recommends and is a genuine improvement on both.
The accumulation is the problem rather than any individual choice. Nothing removed the previous location when a new one was added, so the secret exists in five places and rotating it means finding all five.
The fix
Where a secret can live, ranked
in the repository anybody with a clone. forever.
in the image anybody who can pull it. docker history
shows build args.
in environment: visible in `docker inspect`, in `ps auxe`,
and in every crash dump
in an env_file file permissions apply. better.
in a docker secret tmpfs, 0400, per-service scope
in a secret manager audited, rotatable, asked for at bootThe jump that matters most is out of the image, because an image is copied to registries and laptops and cannot be un-copied. The jump from environment variable to file is smaller and still worth making: docker inspect prints environment variables to anybody on the host, and a crash handler that dumps the environment sends them to an error tracker.
The build, which must not receive the secret at all
# wrong, and the most common mistake — ARG values are
# recorded in the image history whether or not they are used
ARG COMPOSER_AUTH
RUN composer install
# right: the secret is mounted for one RUN, in no layer
RUN --mount=type=secret,id=composer_auth
COMPOSER_AUTH="$(cat /run/secrets/composer_auth)"
composer install --no-dev --no-progress
# DOCKER_BUILDKIT=1 docker build
# --secret id=composer_auth,src=./composer-auth.json -t app .BuildKit secret mounts are the piece that makes a private Composer repository possible without a token in the image, and they need DOCKER_BUILDKIT=1 in 2020 because BuildKit is not yet the default. Forgetting the variable makes the --mount line a syntax error rather than a silent fallback, which is the good failure mode.
Runtime, without an environment variable
// config/database.php
'password' => turkerdev_secret('db_password'),
// the helper prefers a file and falls back to the variable
function turkerdev_secret(string $name): string
{
$key = strtoupper($name);
$file = $_ENV[$key . '_FILE'] ?? "/run/secrets/{$name}";
if (is_readable($file)) {
return rtrim((string) file_get_contents($file), "n");
}
return $_ENV[$key]
?? throw_secret_missing($name); // never returns null
}The _FILE convention is what the official MySQL, Postgres and Redis images use, so following it means the application configures the same way as everything around it. Trimming the trailing newline is not optional — an editor that adds one produces an authentication failure with a message that says nothing about whitespace.
Throwing rather than defaulting to empty is the part worth being firm about. A missing secret that becomes an empty password produces a connection attempt as an anonymous user, and the resulting error names the wrong problem.
services:
php:
image: app:build
secrets: [db_password]
environment: { DB_PASSWORD_FILE: /run/secrets/db_password }
secrets:
db_password:
file: ./secrets/db_password # swarm: external: trueOn plain compose this is a bind mount with a nicer name and it still gets the file semantics — permissions apply, docker inspect does not print it, and a crash dump does not include it. On Swarm it is a real secret distributed over the encrypted control plane and never written to disk on the node.
The rotation that has to be possible
a secret that cannot be rotated in an hour will not be
rotated when it needs to be. so the test is not "where is
it stored" but:
how many places must change?
does changing it require a deploy? downtime?
who is allowed to, and is that recorded?
before: 5 places, yes, yes, anybody, no.
after: 1 place, no (SIGHUP), no, two people, yes.Reframing the question from storage to rotation is what makes the decision tractable, because storage has no obviously right answer and rotation time does. A secret in five places with a deploy required is a secret that stays unchanged for four years, which is what had happened.
The reload without a deploy is what removes the downtime objection, and it is the part that needs application support: re-reading the file on SIGHUP rather than at boot, and a connection pool that can be drained. That is more work than the storage change and it is what makes the storage change worth anything.
And the history, which is not cleaned
# the tempting answer
$ git filter-repo --path .env.production --invert-paths
# what it achieves: every commit hash after the deletion
# changes, every branch and PR needs rebasing, every clone
# needs re-cloning — and forks, CI caches and backups keep
# the old objects. the credential is still valid.
# instead: rotate it today, then decide about the history.History rewriting is the response people reach for and it does not solve the problem — the secret was exposed and the only thing that unexposes it is a new secret. Rewriting is worth doing afterwards for a public repository and is close to pointless for a private one where the credential has already been rotated.
Verifying it worked
$ docker exec app-01 env | grep -c PASSWORD
0
$ docker inspect app-01 | jq '.[0].Config.Env' | grep -ci password
0
$ docker history --no-trunc app:2020-09 | grep -ci 'password|token'
0
$ docker exec app-01 ls -l /run/secrets/
-r-------- 1 www-data www-data 32 Sep 22 09:14 db_password
# and the hook, which prevents the next one
$ git commit -m 'wip'
detect-secrets............................................Failed
Potential secret: config/services.php:14The scanning hook is the only one of these checks that changes the future. Everything else is a cleanup, and a cleanup without a hook is a cleanup that gets repeated in eighteen months.
The baseline file that detect-secrets keeps is important to review rather than to generate and forget: it records the findings deliberately marked as false positives, and a baseline regenerated wholesale to make the hook pass silently accepts every real finding in it.
What this costs
A helper function between the application and its configuration, which is one more thing to understand, and a secrets directory that must exist on every host before the application starts. The second one is the operational cost: a new host without the files produces an application that throws at boot, which is correct and is also a deployment step somebody has to know about.
Docker secrets on plain compose are file mounts with better ergonomics rather than a real secrets system — there is no audit log, no rotation mechanism and no access control beyond file permissions. Calling it a solution overstates it; it is a substantial improvement over the environment variable and the honest description is that the next step is a secret manager, which is a service to run and a dependency at boot.