Redis functions replace the Lua scripts nobody could version

A rate limiter, a distributed lock and a leaderboard update were three Lua scripts loaded at boot and invoked by SHA. After a Redis restart the scripts were gone, the application cached the SHAs, and every call returned NOSCRIPT until something noticed and reloaded them. Redis 7.0 in April replaced the whole arrangement with a registry that persists.

The symptom

$ redis-cli SCRIPT EXISTS 9c1f4a7e3b2d4f81a6e011d0c8b3f204
1) (integer) 0

$ tail -3 /var/log/app/error.log
NOSCRIPT No matching script. Please use EVAL.
NOSCRIPT No matching script. Please use EVAL.
NOSCRIPT No matching script. Please use EVAL.

# and the deeper problem, which is not the restart:
$ redis-cli SCRIPT LOAD "$(cat scripts/ratelimit.lua)"
"9c1f4a7e3b2d4f81a6e011d0c8b3f204"

# nothing on the server knows what that SHA is, what
# version it is, or where the source lives.

The NOSCRIPT failures are recoverable and are not the interesting part. The interesting part is that a running Redis instance had four scripts loaded, identified by hashes, with no way to answer which release each came from — a deployment artefact with no name and no version.

Why it happens

SCRIPT LOAD was never a deployment mechanism. It was a caching optimisation so that a client could send a hash rather than a script body, and it grew into the way people ship server-side logic because there was nothing else.

The consequence is that scripts live in the client’s memory, are reloaded on demand by whichever process notices first, and are not replicated to replicas in any dependable way. Nothing about that is a registry.

The fix

A library, with a name

#!lua name=turkerdev

local function rate_limit(keys, args)
  local n = redis.call('INCR', keys[1])

  if n == 1 then redis.call('EXPIRE', keys[1], args[1]) end

  return n
end

local function claim_lock(keys, args)
  if redis.call('SET', keys[1], args[1], 'NX', 'PX', args[2]) then
    return 1
  end

  return redis.call('GET', keys[1]) == args[1] and 1 or 0
end

redis.register_function('td_rate_limit', rate_limit)
redis.register_function('td_claim_lock', claim_lock)

The shebang line naming the library is mandatory and is what makes the whole file a unit — one file, one library, several functions, loaded and replaced atomically. The functions are called by name rather than by hash, which is the entire improvement.

$ redis-cli -x FUNCTION LOAD REPLACE < functions/turkerdev.lua
"turkerdev"

$ redis-cli FUNCTION LIST
1) "library_name"  2) "turkerdev"
3) "engine"        4) "LUA"
5) "functions"     6) 1) 1) "name" 2) "td_rate_limit" ...

$ redis-cli FCALL td_rate_limit 1 rl:user:4471 60
(integer) 1

What persistence actually means here

EVAL / SCRIPT LOAD   in memory only. gone on restart. not
                     in the RDB or the AOF. replicated only
                     as a side effect of the command being
                     propagated, which is subtle.

FUNCTION             serialised into the RDB, written to
                     the AOF, replicated as data. survives
                     a restart, a failover and a promotion.

which means it is now a thing that must be DEPLOYED rather
than a thing that is loaded on demand.

Being part of the dataset is the substantive change and it cuts both ways: the library survives a restart and it is also now in the backup, which means restoring an old RDB restores an old version of the library. That is a new failure mode and it is the reason the deploy step reloads unconditionally rather than checking whether the library exists.

The deploy step

set -euo pipefail

# REPLACE is not optional: without it, loading a library
# that already exists is an error.
for host in redis-01 redis-02 redis-03; do
  redis-cli -h "$host" -x FUNCTION LOAD REPLACE 
    < functions/turkerdev.lua
done

# and the assertion, per host, because a partial deploy
# leaves the cluster with two versions
for host in redis-01 redis-02 redis-03; do
  redis-cli -h "$host" FCALL td_version 0
done | sort -u | wc -l | grep -qx 1

A version function returning a constant is a small trick that makes the deploy verifiable — without it, “is the new library live everywhere” has no answer short of reading the source of each function. Loading to every node rather than relying on replication is deliberate: replication carries it, and a replica promoted mid-deploy may not have received it yet.

The constraint that has not changed

-- still single-threaded. a function blocks every other
-- client for its entire duration.

-- fine: a handful of commands
local n = redis.call('INCR', keys[1])

-- an outage: an unbounded loop
local ks = redis.call('KEYS', 'session:*')
for i, k in ipairs(ks) do redis.call('DEL', k) end

-- and lua-time-limit does NOT kill it. it starts
-- returning BUSY to other clients, which is worse.
-- only FUNCTION KILL stops it, and only before a write.

The blocking behaviour is what makes atomicity possible and is unchanged, so the discipline about keeping functions to a handful of commands is exactly as important as it was. The new error is FUNCTION KILL rather than SCRIPT KILL, which is a detail worth putting in the runbook before it is needed at three in the morning.

Migrating the four scripts

  rate limiter     → td_rate_limit,  direct translation
  distributed lock → td_claim_lock,  direct translation
  leaderboard      → td_leaderboard_bump, direct
  cache warmer     → deleted

the fourth had been iterating a key pattern with SCAN
inside a script, blocking the server for up to 400ms on
every invocation. nobody had noticed: it ran at 03:00.

Finding that during a mechanical migration is the usual outcome and is worth more than the migration itself. A script that blocks for four hundred milliseconds is a four-hundred-millisecond outage for every other client, and it had been running nightly since 2019.

Verifying it worked

$ redis-cli DEBUG RELOAD
OK
$ redis-cli FUNCTION LIST | grep -c library_name
1        # survived. SCRIPT LOAD would not have.

$ redis-cli -h redis-replica-01 FCALL td_version 0
"2022.4.2"        # replicated

$ redis-cli --latency-history -i 10 | head -3
min: 0, max: 1, avg: 0.19 (2402 samples)
min: 0, max: 1, avg: 0.18 (2398 samples)

# the nightly 400ms spike is gone, because the script that
# caused it is gone.

DEBUG RELOAD forces a save and reload in place, which is the fastest way to prove persistence without restarting anything. The latency history is the check that the removed script was the cause of the spike, and it needed a full day to be convincing.

What this costs

A Lua dependency that is now a deployable artefact with its own release step, its own version and its own place in the rollback plan. That is more process than SCRIPT LOAD and is the correct amount for something that executes on the database — the previous arrangement was cheap because it was not really deployment.

The library also being in the RDB is a genuine new hazard. Restoring a backup from last month restores last month’s functions, and if the application has since started calling a function that did not exist then, every call fails until somebody reloads. Adding the function load to the restore runbook is the mitigation and it is exactly the sort of step that gets omitted from a runbook written before the feature existed.