The latency graph had a cliff at noon every day. Not a spike — a cliff, straight up to eleven seconds and back down over about ninety seconds. Nothing ran at noon. The cache had been warmed at midnight with a twelve-hour TTL, and twelve hours later every entry expired in the same second.
The symptom
$ curl -s 'localhost:9090/api/v1/query?query=histogram_quantile(0.99,...)'
| jq -r '.data.result[0].values[] | @tsv'
11:58:00 0.214
11:59:00 0.208
12:00:00 4.881
12:00:30 11.204 ← every worker on the same query
12:01:00 9.118
12:02:00 0.302
mysql> SHOW PROCESSLIST;
# 48 identical SELECTs, all 'Sending data', all started within 400msForty-eight workers running the same four-second query simultaneously, because all forty-eight missed the cache in the same instant. The database was not slow; it was doing forty-eight times the necessary work and each copy was therefore twelve times slower than usual.
Why it happens
A shared expiry means a shared miss. The naive get-or-compute pattern has no coordination at all: every request that finds the key absent proceeds to compute it, and under load “every request” is however many workers exist.
// the pattern in every codebase, and it is a stampede by design
$value = Cache::get($key);
if ($value === null) {
$value = $this->expensive(); // 48 workers, all here
Cache::put($key, $value, 43200);
}
return $value;It is worse than it looks, because the expensive computation is slower when forty-eight copies are running — so the window during which everyone misses is longer than the normal execution time, which lets even more requests in. That feedback is why it is a cliff rather than a bump.
The fix
A lock around the rebuild
public function remember(string $key, int $ttl, callable $build)
{
$value = $this->cache->get($key);
if ($value !== null) {
return $value;
}
$lock = $this->cache->lock($key . ':build', 30);
if ($lock->get()) {
try {
$value = $build();
$this->cache->put($key, $value, $ttl);
} finally {
$lock->release();
}
return $value;
}
return $this->whatLosersDo($key, $build);
}One worker rebuilds and forty-seven do something else, which removes the stampede outright. The finally is not optional: a build that throws while holding the lock blocks every other worker until the lock expires, which turns an error into an outage.
The lock TTL has to exceed the worst-case build time or a second worker starts rebuilding while the first is still going, and the whole mechanism quietly stops working. Thirty seconds for a four-second query is deliberate slack.
What the losers should do, which is the design decision
// option 1: serve stale. best where staleness is acceptable.
private function whatLosersDo(string $key, callable $build)
{
$stale = $this->cache->get($key . ':stale');
if ($stale !== null) {
return $stale;
}
// option 2: wait briefly for the winner
for ($i = 0; $i < 20; $i++) {
usleep(100000);
$value = $this->cache->get($key);
if ($value !== null) {
return $value;
}
}
// option 3: build it anyway. correct, and the thing we were avoiding.
return $build();
}All three are legitimate and the choice is per cache entry rather than global. A dashboard aggregate can be four minutes stale and nobody notices; a stock level cannot. Falling through to building it after a wait is the safety valve that keeps a lock failure from becoming a total failure, and it needs to exist even when it should never fire.
Two-tier expiry, which is what makes stale possible
public function put(string $key, $value, int $logicalTtl): void
{
// the entry everyone reads, and which expires when we want a refresh
$this->cache->put($key, $value, $logicalTtl);
// the same value, kept far longer, for the losers to serve
$this->cache->put($key . ':stale', $value, $logicalTtl * 6);
}Two entries rather than one, with the second acting as a floor. This is the part that is usually missing from implementations that still stampede occasionally — without a stale copy, the losers have nothing to serve and have to wait or build.
The memory cost is exactly double, which is worth stating plainly. For a cache of a few thousand computed aggregates that is nothing; for a cache of every product page it is a sizing decision. Applying it selectively to the expensive entries rather than universally is the sensible default.
Jittered TTLs, so nothing expires together again
// every entry written during a warm expires in the same second
$this->cache->put($key, $value, 43200);
// spread over twenty minutes, deterministically per key
$ttl = 43200 + (crc32($key) % 1200);
$this->cache->put($key, $value, $ttl);Deriving the jitter from the key rather than from randomness means the same key gets the same offset on every write, so behaviour stays reproducible and a hot key does not drift. Somewhere around two to five percent of the TTL is enough to flatten the cliff without making the cache lifetime unpredictable.
This is the cheapest of the four mitigations and the one most often skipped in favour of the lock. It is worth having both: the jitter prevents the synchronised miss and the lock handles the unsynchronised one, and each covers a case the other does not.
Invalidation by version rather than by deletion
private const SCHEMA = 'v3';
private function key(int $id): string
{
return sprintf('report:%s:%d', self::SCHEMA, $id);
}
// changing the shape of the cached value: bump the constant.
// every old entry is orphaned and expires on its own.
// no flush, no coordination, no stale reads during a rolling deploy.A full flush is the alternative and it is a stampede by another name — every key gone at once on a busy system, which is precisely the incident this article started with. Versioning also makes a rolling release safe: old and new instances run simultaneously and each reads its own entries rather than fighting over one.
The orphaned entries occupy memory until they expire, which is the reason to have maxmemory-policy allkeys-lru configured rather than leaving Redis to fill and start refusing writes.
Caching the computation, not the query result
Before any of the above is worth building, it is worth checking what is actually being cached. Storing the rows a query returned saves the round trip and leaves the hydration, the mapping and the aggregation to happen on every request anyway.
// saves 40ms of query, keeps 380ms of work
$rows = $this->remember("lines:{$id}", 3600, function () use ($id) {
return DB::table('order_lines')->where('order_id', $id)->get();
});
$total = $this->applyDiscountRules($rows); // every request
// saves both, and serialises to an integer rather than a collection
$total = $this->remember("total:{$id}", 3600, function () use ($id) {
return $this->applyDiscountRules($this->lines($id));
});The profile on that endpoint showed the query at forty milliseconds and the discount rules at three hundred and eighty, so caching the rows produced a ten percent improvement and a conclusion that the cache was not helping. Caching the wrong layer is the most common reason a cache disappoints, and it is invisible without a profile.
The second version also serialises to a single integer rather than a collection of model objects, which is a considerable difference once the cache itself is under memory pressure — and it removes the deserialisation cost that was quietly part of the ten percent.
The trade is invalidation. A cached total has to be cleared when a discount rule changes, which is a wider and less obvious set of events than “the rows changed” — and that is the real reason people cache the query result instead. Naming the events that invalidate each cached value, in a comment next to it, is the cheapest defence against getting it wrong later.
Warming as a deploy step, not a cron job
#!/usr/bin/env bash
set -euo pipefail
# after the release is on disk, before the node is marked ready
for path in / /shop /reports/summary; do
curl -fsS -o /dev/null "http://127.0.0.1${path}"
done
rm /var/www/app/shared/.drainingWarming from cron reintroduces the synchronised expiry the jitter was added to remove — everything written at 00:00 expires together, which is how this started. Warming as part of the deploy spreads writes across whatever the deploy takes and, more importantly, means the first real user is not the one paying for a cold cache.
Only the entries that are both expensive and certain to be needed are worth warming. Warming everything is a way of making deploys slow in exchange for a hit rate that a few minutes of traffic would have produced anyway.
Verifying it worked
# cause it deliberately, on staging
$ redis-cli --scan --pattern 'report:v3:*' | xargs redis-cli del
$ ab -n 2000 -c 50 https://staging/reports/summary
# before: 48 concurrent builds, p99 11.2s
# after: 1 concurrent build, p99 0.31s
mysql> SHOW PROCESSLIST;
# one SELECT, not forty-eight
$ redis-cli info stats | grep keyspace
keyspace_hits:1988
keyspace_misses:12Deleting the keys and immediately applying load is the only honest test, because a stampede cannot be observed by waiting for one. The process list showing a single query is a better assertion than the latency number, since latency can improve for reasons unrelated to the fix.
The production confirmation is the absence of the noon cliff over a fortnight, and it is worth leaving a panel on the dashboard specifically for it — a graph of p99 by hour makes a returning periodicity obvious in a way that an alert threshold does not.
What this costs
Stale data is now a product decision with a number attached, and somebody outside engineering has to own that number. “The dashboard may be up to four minutes behind” is a sentence that has to be said out loud and agreed rather than discovered, and it will be discovered by whoever compares two screens during a meeting. Writing the staleness budget next to each cached thing, in the code, is the cheapest way to keep it honest.
The implementation cost is a cache layer that is no longer a one-line call, and that is a real loss of clarity. Confining all of it — lock, stale tier, jitter, versioning — to one class with a remember() method keeps the call sites unchanged, and it is worth resisting the temptation to expose the knobs. Every option offered at the call site is an option somebody will set wrongly, and the defaults are right for almost everything.