A reporting database rebuilt nightly, and a failure that produced a stale copy rather than an error.
the job: truncate, repopulate, swap. and the failure:
the source query timed out at step two, so the job
exited non-zero with the tables half-populated.
what nothing checked: whether the rebuild had
completed, as opposed to whether the process had run.
the reports were served from a partially populated
database for two days and looked plausible — revenue
down 40%, which finance noticed and assumed was real.A half-completed rebuild is worse than a failed one because the output is plausible, and the guard is a row count assertion after the swap rather than an exit code. Finance believing a forty per cent drop for two days is the part that made this an incident rather than a bug.