A product price was changed on the second of March and the category page still showed the old one on the thirteenth. The product page was correct. The API was correct. The search results were correct. One page, wrong, for eleven days, and the invalidation code was demonstrably running.
The symptom
$ curl -s https://shop.example/api/products/8814 | jq .price_cents
4900
$ curl -s https://shop.example/products/laptop-stand | grep -o '£[0-9.]*'
£49.00
$ curl -s https://shop.example/category/desk | grep -o '£[0-9.]*' | head -1
£62.00 # the old price
$ curl -sI https://shop.example/category/desk | grep -i 'age|cache'
age: 918402
x-cache: HITAn age of 918,402 seconds is ten and a half days, which is the answer — the CDN had cached the page and nothing had told it not to. The application cache was being cleared correctly the whole time.
Why it happens
Caches accumulate one at a time, each added by somebody solving a specific problem, and no single person has ever enumerated them. The invalidation code is written against the cache that existed when the code was written.
Nothing in the system knows that four layers exist. There is no list, no diagram, and no test that asserts a change propagates from end to end — the only mechanism connecting them is somebody remembering.
The fix
Enumerating the layers, which nobody had done
1 opcache bytecode. cleared on deploy. not a
correctness risk for data.
2 object cache Redis. get_transient / Cache::remember.
cleared by the price observer. ✓
3 full page cache nginx fastcgi_cache, 1h TTL, keyed on
the URL. NOT cleared. ✗
4 CDN 24h TTL, keyed on the URL, with
stale-while-revalidate. NOT cleared. ✗
5 the browser Cache-Control on the HTML. 0. fine.Writing this list took forty minutes and was the entire diagnosis. It should have existed as a document from the moment the second layer was added, and the reason it did not is that each layer was added by a different person for a different reason.
A key that names its dependencies
// opaque: nothing can tell what invalidates it
Cache::remember("category:{$id}:page:{$page}", 3600, $fn);
// the dependencies are in the key, so the invalidation
// is derivable rather than remembered
Cache::remember(
$this->key('category', $id, [
'page' => $page,
'catalog' => $this->catalogVersion(), // bumped on any price change
'layout' => $this->layoutVersion(),
]),
3600,
$fn
);Including a version counter that any relevant write increments turns invalidation into a write to one integer, and the old entries expire on their own TTL. That is far more robust than enumerating keys to delete, because it cannot miss one.
The cost is that a version bump invalidates everything sharing it, so the granularity has to be chosen deliberately: one catalogue version invalidates every category page on any price change, which is correct and possibly more than intended. Per-category versions are the refinement and are another key to manage.
Purging the layers that are not yours
final class CategoryCacheInvalidator
{
public function invalidate(Category $category): void
{
$this->bumpVersion('catalog'); // layer 2
$urls = $this->urlsFor($category);
$this->nginx->purge($urls); // layer 3
$this->cdn->purge($urls); // layer 4
$this->logger->info('cache.invalidated', [
'category_id' => $category->id,
'urls' => count($urls),
]);
}
}Putting all three in one class is what makes the set enumerable — the next person adding a layer has one file to change, and the file is named after the job rather than after a technology. Logging the invalidation is what makes it possible to answer “did the purge run” without guessing.
Generating the URL list is the awkward part, because a category appears on its own page, on paginated variants, on the parent category and in a sitemap. Getting it wrong means a purge that misses a page, which is the original bug with more code.
Tags, and where they stop working
// tag-based invalidation, which reads beautifully
Cache::tags(['catalog', "category:{$id}"])->remember($key, 3600, $fn);
Cache::tags(["category:{$id}"])->flush();
// and the constraint nobody mentions:
// Redis tag flush is implemented as a set of keys per
// tag, and flushing walks it. a tag with 400,000 keys
// is a long blocking operation.
// the file and database drivers do not support tags
// at all.Tags are the right abstraction and they scale badly at exactly the point where caching matters most. A coarse tag on a large catalogue turns an invalidation into a multi-second blocking operation on a single-threaded server, which is an outage caused by a cache clear.
Version counters have none of that problem and are less expressive. The arrangement that worked was versions for the high-cardinality cases and tags for the small ones, with the choice documented per cache rather than applied uniformly.
The CDN header that does most of the work
add_header Cache-Control "public, max-age=0, s-maxage=3600,
stale-while-revalidate=86400, stale-if-error=604800";
# max-age=0 the browser always revalidates
# s-maxage the CDN caches for an hour
# swr serve stale for a day while refreshing
# → the purge matters much less
# stale-if-error serve stale for a week if the origin is downstale-while-revalidate is what makes a short shared TTL affordable: the CDN serves the stale copy immediately and refreshes in the background, so the origin sees one request per hour per page rather than a stampede at expiry. Shortening the TTL from twenty-four hours to one was possible only because of it.
stale-if-error is the underrated one and is a genuine availability feature — a page that has been cached in the last week keeps being served through an origin outage. It also means an incident can be invisible on the cached pages, which is worth knowing before it happens.
Verifying it worked
# the end-to-end assertion, as a scheduled job
$ ./bin/cache-propagation-check
setting price on product 8814 to a sentinel value...
api 0.4s ✓
product page 0.9s ✓
category page 1.8s ✓
cdn 2.4s ✓
restoring...
$ curl -sI https://shop.example/category/desk | grep -i age
age: 41A propagation check that writes a sentinel value and polls every layer until it appears is the only test that covers the actual failure, and it has to run against production because the layers do not all exist elsewhere. Running it nightly and alerting on a layer that does not converge is what turns eleven days into ten minutes.
Writing to production from a scheduled job needs care — a sentinel price on a real product is visible to customers for two seconds. Using a dedicated hidden product removes that risk and slightly weakens the test, which is the right trade.
What this costs
An invalidator class that has to know about every layer, including two that belong to other systems and can fail independently. A CDN purge that returns an error is now a thing the application has to handle, and the honest answer is to retry it and then alert, because there is no way to guarantee it from inside the request.
The deeper cost is that the cache key scheme is now a design artefact requiring maintenance — a new dependency means a new component in the key, and forgetting one produces exactly this bug again. The propagation check is the guard against that, and it is the piece most likely to be quietly disabled when it becomes flaky.