A support ticket said a product price had reverted to its old value after being changed. It had been investigated twice and closed twice as unreproducible, because it is unreproducible — it happens when a read and a write interleave in a specific order, which on this system was about one write in four thousand.
The symptom
the sequence, which takes about 40 microseconds:
T1 request A reads the cache → miss
T2 request A queries the database → 4900
T3 request B updates the price → 4400
T4 request B invalidates the cache → deletes nothing
T5 request A writes the cache → 4900
the cache now holds 4900 and the database holds 4400,
and nothing will correct it until the TTL expires.
on this system: a 1-hour TTL, and a price that was wrong
for 58 minutes.Request A’s write is the last one and it writes a value it read before request B’s update. The invalidation at T4 is correct and does nothing, because there is nothing in the cache to invalidate yet.
Why it happens
The read-through pattern has a window between reading the source and writing the cache, and any write during that window is lost. The window is microseconds, which is why it is invisible in testing and inevitable at volume.
It is a genuine race rather than a bug in the invalidation, and no amount of correctness in the invalidation code closes it — the delete happens before the write it needs to prevent.
The fix
Deleting rather than updating, which narrows it
// write-through: update the cache with the new value
DB::transaction(fn () => $product->update(['price_cents' => 4400]));
Cache::put("product:{$id}", $product, 3600);
// which has a worse race: two concurrent writes can
// apply to the database in one order and to the cache in
// the other.
// delete-on-write: narrower, and still open
DB::transaction(fn () => $product->update(['price_cents' => 4400]));
Cache::forget("product:{$id}");Deleting rather than updating removes the two-writer reordering and leaves the read-then-write window, which is the one in the timeline above. It is the standard advice and it narrows the problem without closing it, which is worth knowing before treating it as a fix.
Versioned keys, which close it
public function find(int $id): Product
{
$version = Cache::get("product:{$id}:v", 0);
return Cache::remember(
"product:{$id}:{$version}",
3600,
fn () => Product::findOrFail($id),
);
}
public function update(Product $p, array $attributes): void
{
DB::transaction(fn () => $p->update($attributes));
Cache::increment("product:{$p->id}:v");
}the same interleaving, with versions:
T1 A reads version → 7, reads product:41:7 → miss
T2 A queries the database → 4900
T3 B updates, and INCREMENTS the version → 8
T4 A writes product:41:7 = 4900
T5 the next read gets version 8, reads product:41:8
→ miss, and queries the database → 4400
A's stale write lands in a key nobody will read again.The stale value is still written and is written to a key that is now unreachable, which is the whole trick. The version counter is a single integer that is incremented rather than compared, so it has no race of its own — INCR is atomic and the order of two concurrent increments does not matter.
The cost is one extra cache read per lookup, which on Redis is about a hundred microseconds and is the price of correctness. The orphaned keys expire on their own TTL and are not cleaned up, which means the cache holds some dead entries — bounded by the TTL and by the write rate.
Reproducing it, which is the hard part
// a load test with an ASSERTION, not a benchmark
public function testConcurrentReadAndWriteDoNotDiverge(): void
{
$product = Product::factory()->create(['price_cents' => 4900]);
$this->runConcurrently(
readers: 40,
writers: 4,
seconds: 30,
write: fn (int $n) => $this->service->update(
$product, ['price_cents' => 4000 + $n]
),
);
// the assertion: after everything settles, the cache
// and the database agree
$this->assertSame(
Product::find($product->id)->price_cents,
$this->service->find($product->id)->price_cents,
);
}Asserting on convergence after the load stops is what makes this a test rather than a benchmark, and it is the only way to catch a race that is invisible in a sequential test. The original implementation failed it within four seconds; the versioned one has run for thirty minutes without diverging.
$ ./bin/race-test --readers=40 --writers=4 --seconds=30
before:
divergences: 41 of 4,102 writes (1.0%)
first at: 3.9s
after:
divergences: 0 of 4,118 writes
max latency: +0.14ms per read (the version lookup)The variant that does not need a version
// for a value derived from a single row, the row's own
// updated_at is a version that already exists
$product = Product::select('id', 'updated_at')->findOrFail($id);
return Cache::remember(
"product:{$id}:{$product->updated_at->getTimestamp()}",
3600,
fn () => $this->expand($product),
);
// which costs a small query instead of a cache read, and
// is only correct if updated_at is reliably touched —
// a bulk UPDATE that does not set it breaks this silently.Using updated_at avoids the extra counter and depends on every write touching it, which a bulk update or a raw statement does not. That is a dependency on a convention rather than on a mechanism, and it broke once during a data fix — which is the argument for the explicit counter on anything that matters.
Verifying it worked
$ ./bin/race-test --readers=40 --writers=4 --seconds=300
divergences: 0 of 41,208 writes
# and the standing check, hourly, in production:
$ php artisan cache:reconcile products --sample=500
500 sampled, 0 divergent
# the support ticket, closed properly this time, with the
# timeline and the reproduction attached.A sampled reconciliation in production is the standing assertion and it is cheap — five hundred comparisons an hour. It is also what would have found the original bug, since a one-in-four-thousand divergence appears in a sample of five hundred within a few hours.
What this costs
A key scheme that is now load-bearing, and one extra read per lookup. Anything reading the cache directly — a debugging command, a warming script, a dashboard — has to know about the version indirection, and there were three such places that broke on the first deploy.
The orphaned entries are the other cost and they are bounded rather than absent: a product updated fifty times an hour with a one-hour TTL leaves fifty dead keys. On a large catalogue that is measurable memory, and shortening the TTL to bound it trades cache hit rate for memory — which is a decision that has to be made with numbers rather than by instinct.