monitoring

  • Building an MCP server for a database

    turkerdev/mysql-mcp, and what a protocol for tool access actually requires. The authorisation model is the host’s problem, which means it is all yours.

  • An outage resolved before anybody woke up

    Four mechanisms built between 2023 and 2025, and the first incident where all of them were exercised at once. The absence alert is the one that did the…

  • A report that got slower because the data grew

    A query that was fine at four hundred thousand rows, acceptable at four million, and unusable at twelve. A query whose cost grows with the data is not…

  • A pattern that had been detached on nine pages

    A synced pattern on forty-one pages, nine of which had been detached and were now divergent copies. Detaching is one click with no confirmation and no subsequent indication,…

  • A nightly rebuild that failed silently for two days

    A reporting database rebuilt nightly, and a failure that produced a stale copy rather than an error. A half-completed rebuild is worse than a failed one because the…

  • The second host, and the failover we finally built

    Seven years on one host, and a business requirement that finally arrived. A recovery time that assumes the host comes back.

  • Alerting on absence, for every integration

    An alert per integration that fires when it stops doing anything, which is the category nobody adds. Every alert we had was for a value being too high…

  • A consumer that appeared in the usage log and nowhere else

    An API key making four hundred requests a day, belonging to nobody we had a record of. A key labelled “test” making four hundred production requests a day…

  • A dead letter queue that stayed at zero

    A queue that reached zero in 2025, and twelve months of staying there. An empty dead letter queue is not the goal and is a signal — a…

  • The integration that outlived the integrator

    A supplier acquired and shut down, and an endpoint returning 200 with an empty body for eleven weeks.