system-design

  • Backpressure is a design decision, not an accident

    A producer faster than its consumer fills a queue, and what happens when the queue is full is a decision somebody makes — or a decision the infrastructure…

  • Idempotency for an endpoint that takes money

    The client timed out, retried, and the customer was charged twice. A timeout tells the client nothing about what the server did.

  • The dual write problem, and why the outbox exists

    Writing a row and publishing an event are two systems, and there is no ordering of the two that is safe. Neither ordering is correct and both are…

  • A read replica that was silently stale

    The replica was up, accepting connections and serving data from an hour ago. Seconds_Behind_Master reported zero, honestly, and was measuring the wrong thing.

  • An SLO is a number somebody agreed to

    Everyone wanted the site to be fast and nobody could say what fast meant. An indicator, an objective and a window — and the error budget is what…

  • Read-your-own-writes is a product requirement, not a bug

    A user who submits a form and immediately sees stale data reports it as a bug, and it is a consequence of a decision somebody made about replication.…

  • The error budget is what the objective is for

    An objective of 99.5% is also a statement that 0.5% of requests may fail, and that allowance is a resource with a use. The budget turns reliability from…

  • A stampede lock, and what the losers should do

    An expensive cached value expires under load and every concurrent request misses simultaneously, so forty workers run the same four-second query at once. What the losers do is…

  • Timeouts bound one request; only a cap bounds concurrency

    Setting a thirty-second timeout feels responsible and does nothing about the number of workers waiting simultaneously, which is what actually takes a site down. Dropping the timeout from…

  • Redis 6 and the shared password that was everywhere

    One password, six services, and FLUSHALL available to all of them. 6.0 landed in April with users — and the keyspace pattern is where the isolation actually is.