Redis beyond caching: locks, counters and rate limits

Redis had been in the stack for a year as a cache and nothing else. Meanwhile there were two cron servers running the same nightly export, a rate limiter implemented with a database table, and a counter that was three queries. All three are single commands.

The symptom

[2016-10-04 02:00:01] export.INFO: starting nightly export {"host":"cron-01"}
[2016-10-04 02:00:01] export.INFO: starting nightly export {"host":"cron-02"}
[2016-10-04 02:41:18] export.INFO: wrote 41208 rows {"host":"cron-01"}
[2016-10-04 02:41:52] export.INFO: wrote 41208 rows {"host":"cron-02"}

Two machines, the same crontab, the same job. The supplier received the file twice and had been silently de-duplicating it for months.

Why it happens

Cron has no notion of a cluster. Two servers configured identically — which is the goal — run everything twice, and the usual fixes make it worse: a lock file is per machine, a “run only on cron-01” flag reintroduces the single point of failure that the second machine was added to remove.

What is needed is somewhere both machines can agree, atomically. That is exactly what a single-threaded key-value server is good at, and there was already one.

The fix

A lock that expires

// SET key value NX EX seconds — atomic, one round trip
$token = bin2hex(random_bytes(16));

$acquired = $redis->set(
    'lock:nightly-export',
    $token,
    array('nx', 'ex' => 3600)
);

if (! $acquired) {
    return;   // someone else has it
}

The expiry is not optional and it is not a nicety. A process holding a lock without one and then being killed holds it forever, and the job never runs again until somebody notices — which is a worse failure than running twice.

The TTL has to exceed the longest plausible run. Forty minutes of work under a one-hour lock is comfortable; under a thirty-minute lock the second machine acquires it while the first is still writing, and now there is a duplicate and a lock nobody owns.

Releasing a lock you still own

The naive release is DEL, and it is wrong in exactly the case that matters. If the job overran and the lock expired, another machine now holds it — and the DEL deletes theirs.

$release = <<<'LUA'
if redis.call("get", KEYS[1]) == ARGV[1] then
    return redis.call("del", KEYS[1])
else
    return 0
end
LUA;

$redis->eval($release, array('lock:nightly-export', $token), 1);

The token is the proof of ownership, and the check and the delete have to be atomic — doing them as two commands reintroduces the race in a smaller window. Lua scripts run atomically on the server, which is what makes this correct rather than merely unlikely to fail.

Counters and a sliding window

The rate limiter had been a table with a row per request and a COUNT per check, which is three queries and a cleanup job. A fixed-window version is one command.

$key = 'rate:' . $ip . ':' . floor(time() / 60);

$hits = $redis->incr($key);

if ($hits === 1) {
    $redis->expire($key, 120);
}

if ($hits > 60) {
    abort(429);
}

The window boundary is the flaw: sixty requests at 11:59:59 and sixty more at 12:00:01 is a hundred and twenty in two seconds, all within the limit. A sorted set gives a true sliding window at the cost of more memory.

$now = microtime(true);
$key = 'rate:' . $ip;

$redis->zRemRangeByScore($key, 0, $now - 60);
$redis->zAdd($key, $now, $now . ':' . random_int(0, 9999));
$redis->expire($key, 60);

if ($redis->zCard($key) > 60) {
    abort(429);
}

Four commands rather than one, and each entry costs memory for the whole window — so this is the right choice for an API where the boundary matters and the wrong one for a login form where it does not. Pipelining the four removes three round trips.

Verifying it worked

The lock needs the race forced rather than waited for.

$ for h in cron-01 cron-02 cron-03; do
>   ssh $h 'php /var/www/shop/artisan export:nightly' &
> done; wait

cron-01  acquired lock, exporting
cron-02  lock held elsewhere, exiting
cron-03  lock held elsewhere, exiting

$ redis-cli TTL lock:nightly-export
(integer) 3574

Where the lock is on the critical path

Using Redis for correctness rather than speed changes what its failure means, and the two jobs now behave differently when it is unavailable — deliberately.

// the export: fail closed. running twice is worse than not running.
try {
    $lock = $this->locks->acquire('nightly-export', 3600);
} catch (RedisException $e) {
    Ops::alert('export skipped: lock unavailable', array('e' => $e->getMessage()));
    return;
}

// the rate limiter: fail open. a rate limit is not worth an outage.
try {
    $hits = $this->redis->incr($key);
} catch (RedisException $e) {
    $this->metrics->increment('ratelimit.bypassed');
    return $next($request);
}

Opposite decisions from the same outage, and both are right for what they protect. The rule of thumb: fail closed when the failure mode is duplicated work or corrupted data, fail open when it is a policy the site can survive not enforcing for ten minutes.

What neither may do is silently swallow the exception without a signal. The bypassed-rate-limit counter is the only thing that will tell you the limiter has been off since Tuesday.

What this costs

Redis is now on the critical path for correctness rather than only for speed. A cache that is down makes the site slow; a lock server that is down makes the nightly export either run three times or not at all, depending on which way the code fails. That decision has to be made deliberately — for an export, failing closed and alerting is right.

It also needs persistence configured, which a pure cache did not. A Redis restart with no AOF loses every lock instantly, and the jobs holding them do not find out.