Most backup jobs report success once a file has been written somewhere. That tells you a file exists. It doesn't tell you the file restores, that it's complete, or how long a restore takes on the day you need it.
I'm Tyler, a software engineer at Kratos Defense & Security. At home I run a Linux server that holds things I can't lose: a few Postgres databases, a self-hosted secret store, and the home directories of the Claude Code agents that run on it. On that box a backup only counts once something has restored it. The databases and the secret store get a full restore every night, the home directories get a sampled one, and a timed drill covers what the nightly checks can't. This post covers each one.
Postgres: the dump has to match the database
Every night each database is dumped inside one REPEATABLE READ snapshot, and in that same snapshot the job takes an exact count(*) of every table. Then it restores the dump into a scratch Postgres 17 on a local socket and counts again. A dump only becomes the new latest copy when every table's count matches. Dumps go off the box to object storage, and each one is kept for 35 days.
The shared snapshot matters. pg_dump is consistent on its own, but counts taken in a separate transaction would include rows written after the dump started, and would never match it.
The secret store: restore it every night
The secret store is exported and dumped, encrypted with age, and uploaded. A separate timer downloads it every night, decrypts it, restores it, and checks that every live key name is there. An encrypted backup you've never decrypted is still a guess.
Home directories: restic, then a sample
Every home directory goes to a restic repository. After each nightly backup the job runs restic check over a slice of the stored data, then restores a random sample of files from three homes into a scratch directory and compares each one with the live file by sha256.
Don't trust the job, check where it wrote
A backup job can exit 0 and still leave nothing useful behind. So a separate monitor never reads the job's exit code. It reads the bucket itself and fails unless the newest dump is under 26 hours old and its verify.txt starts with restore: match. When it fails, its systemd unit fails, and every failed unit on this box opens a GitHub issue automatically. A silent backup becomes an open issue the next morning.
The drill
The nightly checks catch a backup that won't restore. They don't say how long a full recovery takes. So on October 30 I ran a timed drill: the secret store and one entire home directory, restored into scratch space, with recovery time measured from the first download to a verified restore.
| backup | recovery time | what was proven |
|---|---|---|
| backupsecret store | recovery time28 s | what was provenevery live key name present after the restore |
| backupone home directory | recovery time1 m 25 s | what was proven111,555 files and 7.21 GB, restic restore --verify clean, 186 of 186 sampled files byte-identical by sha256 |
A second run gave 21 s and 1 m 36 s. Every backup but one runs nightly, so for those the most data I can lose is about a day.
The one that runs weekly
One mostly idle database lives on a hosted free tier with a 5 GB monthly transfer cap. A nightly dump would have spent that allowance in about ten days, so the backup would have been the thing that broke it. That one runs weekly on purpose.