Notes from the backend

Architecture, code, data stores and the infrastructure underneath — written down while the details were still fresh.

Deep dives

  • An incident with no engineer

    An outage at 04:00 in August, resolved before anybody woke up. A worker crash loop that would have been a two-hour outage in 2023.

  • The command bus we removed

    A command bus introduced in 2021, and 88 commands with one handler each. A stack trace with four frames of dispatch before the work.

  • The test that had been quarantined since 2023

    A quarantine list with a deadline, three extensions, and one test covering a payment path that had been skipped for three years.

  • Reusable skills and agent definitions, and what they are actually for

    turkerdev/ai-skills and turkerdev/ai-agents, and eighteen months of retyping the same instructions. Context is per-session and knowledge is not.

  • Deleting a module

    A module built in 2022 for a product line discontinued in 2024. Eight thousand lines, a database schema and two scheduled jobs, all still running.

  • A pipeline described in one file, after four years of five

    Five workflow files, two composite actions, and nobody able to say what runs when. A change that needed editing three files, one of them wrong.


Recent notes


Latest snippets