A cache that was a second database

An unplanned restart of the cache server lost four hours of user preferences, which is a sentence that should not be possible. A cache had become the only home for three kinds of data, gradually, over three years, and nothing had ever tested what happened when it emptied.

The symptom

the incident, in full:

  14:02  the cache host is restarted for a kernel update
  14:03  the application is fine. everything regenerates.
  14:09  the first support ticket: "my dashboard layout
         has reset"
  14:40  eleven tickets. all the same.
  15:20  the cause: dashboard layouts are written to the
         cache by the browser, read from the cache, and
         written nowhere else.

four hours of layout changes, gone. no backup, because
a cache is not backed up.

The write path was a REST endpoint that put the layout into the cache and returned 200, added in 2021 as a “temporary” store while the schema was being decided. The schema was never decided and the endpoint kept working.

Why it happens

A write-through cache with a database behind it becomes a write-only store the moment somebody removes the database half, and nothing about the code changes shape when that happens. The failure is invisible until the cache empties, which may be years.

The fix

Auditing every key

# every key pattern, normalised, with writes and reads
# listed separately — a pattern that appears in one list
# and not the other is the whole finding
$ ./bin/cache-keys --writes | sort -u > /tmp/w
$ ./bin/cache-keys --reads  | sort -u > /tmp/r
$ comm -23 /tmp/w /tmp/r      # written, never read here

41 key patterns, one question each: if this vanished,
could it be regenerated?

  34  derived from the database. safe.
   4  expensive but derivable — a report that takes 40
      seconds to rebuild. safe, slowly.
   3  NOT DERIVABLE:
        dashboard:layout:{user}
        onboarding:progress:{user}
        search:recent:{user}

all three written by the browser. all three read by
nothing else. all three since 2021.

The audit is a grep and an afternoon, and the useful output is not the list — it is the question, asked once per key, that nobody had ever asked. Thirty-four of forty-one were exactly what a cache is for, which is why the three were invisible.

Persistence as the wrong fix

the first proposal: enable persistence so a restart
preserves the data. it fixes a planned restart, and not:

  eviction under memory pressure — the cache has a
    maxmemory policy and will discard keys
  an explicit flush, which the deploy used to run
  a failover to a replica that is behind
  the fact that three keys have no other home

and it costs fsync on the write path, a file that grows,
and a rewrite competing for memory with what it protects.
rejected.

Persistence is right when a cache is expensive to rebuild and wrong when it is being used to make a cache durable. The question that settled it was whether the data would survive an eviction — no persistence setting affects that — and the answer meant the data had to move regardless.

Moving the data, keeping the cache

// before
Cache::forever("dashboard:layout:{$user->id}", $layout);

// after
$this->preferences->setLayout($user, $layout);

// and the repository writes the row, then the cache.
// the cache is in front of the database, which is where
// a cache goes.

CREATE TABLE user_preferences (
  user_id     BIGINT UNSIGNED NOT NULL,
  key_name    VARCHAR(64) NOT NULL,
  value       JSON NOT NULL,
  updated_at  DATETIME(6) NOT NULL,
  PRIMARY KEY (user_id, key_name)
) ENGINE=InnoDB;

Recovering what was left

# the four hours were gone. what remained in the cache
# was migrated rather than discarded:
$ redis-cli --scan --pattern 'dashboard:layout:*' | wc -l
8104

$ php artisan preferences:import-from-cache --dry-run
  keys found:     8,104
  users matched:  8,098
  orphaned:           6   (deleted users)
  would write:    8,098 rows

$ php artisan preferences:import-from-cache
  8,098 rows written in 41s

Migrating what was still in the cache rather than starting empty is the difference between eleven support tickets and eight thousand. It is also a one-off script that had to be written under time pressure, which is the ordinary cost of discovering this during an incident rather than during an audit.

A test that flushes

public function testTheApplicationSurvivesACacheFlush(): void
{
    $this->actingAs($user = User::factory()->create())
        ->put('/api/preferences/dashboard', ['layout' => ['a', 'b']])
        ->assertOk();

    Cache::flush();

    $this->actingAs($user)
        ->get('/api/preferences/dashboard')
        ->assertJsonPath('layout', ['a', 'b']);
}

Four lines, and it is the only test in the suite that would ever have caught this. It generalises to any store with a lifetime shorter than the data it holds, and the pattern — write, clear the volatile layer, read — belongs in the feature test for anything that touches a cache.

The rule, and how it is enforced

a cache key is registered with:
  what writes it
  what regenerates it
  what happens if it is missing

and there is no tool that enforces this. it is a review
question and a table in the repository, which means it
will drift.

the backstop that does not drift: the flush test, run
against the full feature suite once a week with the
cache cleared between every request.

that found one more, in July.

Verifying it worked

$ php artisan cache:clear && vendor/bin/phpunit --group=feature
  Tests: 302 passed

$ ./bin/cache-audit
  41 patterns, 41 with a documented regeneration path

# the drill, on staging, deliberately
$ redis-cli FLUSHALL && ./bin/smoke-test
  all 22 checks passed
  slowest first request: 1.8s (the report, expected)

# and the weekly job, since May
  runs: 8
  failures: 1  (July — a new key with no home)

The July failure is the argument for the weekly job. A new key with no database behind it was added in June by somebody who had not read any of this, which is what happens with a rule that is enforced by review — and the job caught it in a week rather than in three years.

What this costs

A rule enforced by review rather than by a tool, which means it will be broken. The weekly flush test is the backstop and it is a job that takes twenty minutes of runner time to protect against a mistake that has happened twice — which is a ratio somebody will question during a cost review.

The four hours of lost preferences are also not recoverable and were nobody’s fault in particular. A temporary store from 2021 that worked is the most durable kind of temporary, and the only thing that would have surfaced it earlier is the question the audit asks — which took an afternoon and could have been done at any point in three years.