Six services shared one Redis password. Any of them could read the session store, any of them could flush the cache, and one of them was a reporting tool written by a contractor in 2018. Nobody had chosen that arrangement — it was what Redis offered, because AUTH took a password and there was nothing else.
The symptom
$ grep -rn 'REDIS_PASSWORD' --include='*.env*' /srv/*/ | wc -l
6
# and what that password permits, which is everything
> AUTH thepassword
OK
> FLUSHALL
OK
> CONFIG SET maxmemory 1mb
OK
> KEYS *
(... 1.4 million keys, on a single-threaded server)
> SHUTDOWN NOSAVEThe reporting tool had run KEYS * in production twice, which each time blocked the server for about four seconds. That was the incident that started this, and the honest observation is that it was not the tool’s fault — nothing had ever said it should not.
Why it happens
Redis was designed as a trusted-network component, and the security model was that it should not be reachable from anywhere untrusted. That is a coherent position and it collapses the moment six services on the same trusted network have different reasons to be trusted.
The rename-command mechanism was the workaround, and it is a blunt one: renaming FLUSHALL to something unguessable removes it from everybody including the operator. 6.0 replaces it with something that can distinguish between callers.
The fix
A user per service
> ACL SETUSER cache on '>c4ch3s3cr3t'
'~cache:*'
'+@read' '+@write' '+@keyspace' '-@dangerous'
> ACL SETUSER worker on '>w0rk3rs3cr3t'
'~queue:*' '~locks:worker:*'
'+@list' '+@stream' '+@transaction' '-@dangerous'
> ACL SETUSER reporting on '>r3p0rt1ng'
'~stats:*'
'+@read' '-@dangerous'
> ACL SAVEACL SAVE is not optional and is easy to forget: users created at runtime live in memory, and a restart without saving loses every one of them — which is an outage where six services simultaneously fail to authenticate. The ACL file is configured separately from redis.conf with aclfile, and a server without one cannot save at all.
Command categories rather than command lists is the choice that survives an upgrade. An enumerated list is wrong the moment Redis adds a command, and the additions are exactly the commands nobody has considered.
What -@dangerous actually removes
> ACL CAT dangerous
flushdb flushall keys client acl swapdb config debug slaveof
cluster info module shutdown monitor replicaof failover latency
... 31 in total
> AUTH cache c4ch3s3cr3t
> FLUSHALL
(error) NOPERM this user has no permissions to run 'flushall'One clause removes thirty-one commands and keeps removing new ones as they are added, which is why it is worth more than any enumerated list. Reading the category once is educational — several of those are commands an application has no business calling and every one of the six credentials could.
info being in the dangerous category catches people, because a monitoring agent legitimately needs it. Adding it back explicitly — -@dangerous +info — is the correct shape, and it is the mechanism that makes categories usable rather than all-or-nothing.
The keyspace pattern, which is where the mistake is
# looks scoped. is not.
> ACL SETUSER worker on '>pw' '~queue:*' '+@all'
# there is no path semantics. keys are a FLAT namespace and
# the colon is a convention. ~queue:* is a glob, not a prefix
# with meaning.
# several patterns are allowed, and that is the correct shape
> ACL SETUSER worker '~queue:*' '~locks:worker:*' '%R~config:*'
# %R = read only for this pattern. %W = write only. 6.2 —
# in 6.0 a pattern grants both.The command permissions get the attention and the key pattern is where the isolation lives: a service with write access to every key is not isolated by having fewer verbs. Getting the patterns wrong is the most likely way to produce an ACL that looks careful and prevents nothing.
Read-only patterns arrive in 6.2, so in 6.0 a service that needs to read a shared configuration key gets write access to it as well. That is worth knowing rather than assuming, and it is a reason to keep shared keys in a separate prefix that only one service can reach.
Rolling it out without an outage
The default user still exists and still has everything, so creating the new users changes nothing at all until it is disabled — and disabling it before the services have their own credentials is a total outage.
1. upgrade to 6.0. nothing changes; the default user is intact.
2. create the users. still nothing changes.
3. switch ONE service to its own credential. verify.
4. repeat, one at a time, over a week.
5. confirm nothing is still authenticating as default:
ACL LOG
and INFO clients, comparing counts
6. THEN: ACL SETUSER default off
7. ACL SAVE
step 5 is the one that cannot be skipped.> ACL LOG
count 4 reason command object keys
username reporting age-seconds 18.482
# every denied attempt, with the user and what it tried.
# this is the rollout's feedback loop.ACL LOG is what turns the rollout from a leap into an iteration: switch a service, watch the log, add the permission it turns out to need. On this migration it found three things nobody had documented — the worker calling INFO for a health check, the cache service using SCAN for an eviction job, and the reporting tool reading a key outside its prefix.
TLS, which arrived in the same release and is a separate decision
tls-port 6379
port 0 # plaintext off entirely
tls-cert-file /etc/redis/redis.crt
tls-key-file /etc/redis/redis.key
tls-ca-cert-file /etc/redis/ca.crt
tls-auth-clients yes # mutual — this is the useful part
# measured: +0.3ms per command on a persistent connectionMutual authentication is what earns its place: a client certificate is a credential that cannot be read out of a log or an environment variable, which is a genuine improvement over a password in a file. The handshake cost is negligible with persistent connections and is paid on every invocation by short-lived CLI processes, which is worth measuring rather than assuming.
On a private network with strict firewall rules the plaintext arrangement remains defensible, and saying so is more useful than a blanket recommendation. The two features are independent: ACLs without TLS is a large improvement, and TLS without ACLs is one shared identity over an encrypted channel.
Verifying it worked
# the assertion the whole exercise was for
$ redis-cli --user reporting --pass r3p0rt1ng FLUSHALL
(error) NOPERM this user has no permissions to run 'flushall'
$ redis-cli --user reporting --pass r3p0rt1ng KEYS '*'
(error) NOPERM this user has no permissions to run 'keys'
$ redis-cli --user reporting --pass r3p0rt1ng GET cache:homepage
(error) NOPERM no permissions to access one of the keys
$ redis-cli --user cache --pass c4ch3s3cr3t GET cache:homepage
"..."
> ACL GETUSER default
... "flags" 1) "off"Proving a service cannot do something by trying it is the only verification worth having, and it is the step that gets replaced by reading the configuration and being satisfied. The default user showing off is the other half — without it every one of the six old credentials still works and the whole exercise is decoration.
What this costs
Six credentials to rotate instead of one, and an ACL file that has to be kept in step with the services. Adding a service now means adding a user, which is a step somebody will forget — and the failure is a service that cannot authenticate at all, which is at least loud. The file itself is a piece of infrastructure state that lives outside the application repositories, which is exactly the kind of thing that drifts.
The subtler cost is that the permissions encode assumptions about what each service does, and those assumptions go stale. A service that grows a feature needing a new command fails in production with a NOPERM that nobody expected, and the fix requires access to the Redis server rather than to the application. Putting the ACL definitions in version control alongside the infrastructure code — and applying them from there rather than by hand — is what makes that a pull request rather than an incident.