A backup job that exits zero every night proves that the job ran, not that the output can be restored. Truncated dumps, a mysqldump that hit a lock and gave up half way, an archive of an empty directory — all exit zero.
# nightly, after the dump: restore into a scratch database and count
mysql -e 'DROP DATABASE IF EXISTS restore_check; CREATE DATABASE restore_check;'
gunzip -c /backups/shop-latest.sql.gz | mysql restore_check
mysql -N -e "SELECT COUNT(*) FROM restore_check.orders;" |
awk '{ if ($1 < 1000) { print "RESTORE CHECK FAILED: " $1; exit 1 } }'The restore is the only test that matters, and automating it turns a hope into a check. Assert something about the content, not just that the import succeeded — a row count with a floor catches the truncated dump that imports cleanly and contains a third of the data. Record how long the restore took, too: that number is the recovery time, and it is usually longer than anyone assumed.