Türker Yıldırım

backup

  • A drill that induced the same failure deliberately

    Note2 min read

    The April incident reproduced on purpose in May, to check whether the fix worked. Reproducing an incident deliberately is the only way to know a fix works, and…

  • The reporting database that is not a replica

    Deep dive10 min read

    A read replica serving reports, and reports that want a different schema. Eleven indexes on production tables that exist only for reporting.

  • A nightly rebuild that failed silently for two days

    Note1 min read

    A reporting database rebuilt nightly, and a failure that produced a stale copy rather than an error. A half-completed rebuild is worse than a failed one because the…

  • The second host, and the failover we finally built

    Deep dive14 min read

    Seven years on one host, and a business requirement that finally arrived. A recovery time that assumes the host comes back.

  • An ssh certificate authority, and the key’s own backup

    Note2 min read

    A certificate authority that removes eleven authorized_keys files and creates one key that must not be lost. Concentrating eleven files into one key is the point of a…

  • A schema change on a table nobody can lock

    Deep dive10 min read

    A column type change on a table taking four hundred writes a second, and an ALTER that estimates four hours.

  • A restore from the second copy, timed

    Note1 min read

    A drill against the offsite copy rather than the primary, which took considerably longer for a reason nobody had costed. A recovery objective quoted from the fast path…

  • A second backup provider with write-only credentials

    Note2 min read

    Every backup in one provider, and a plan that assumed the disaster would be local rather than at the provider. Ranking the failure modes is what makes this…

  • Object lock, and a deletion that could not happen

    Note1 min read

    A second backup copy that survives a compromised credential, which requires the storage to refuse deletion rather than the credential to lack permission. A write-only credential protects against…

  • The backup that had to leave the building

    Deep dive9 min read

    Every backup in one provider’s object storage, and a disaster recovery plan that assumes the disaster is local.