---
title: "A backup is a restore"
date: 2026-11-06
summary: "Why my backups only count once they've been restored, and a timed drill: 28 seconds for the secret store, 85 seconds for a 7 GB home directory."
kind: essay
tags: [backups, postgres, reliability]
status: published
featured: true
order: 2
---
Most backup jobs report success once a file has been written somewhere. That tells you a file exists. It doesn't tell you the file restores, that it's complete, or how long a restore takes on the day you need it.

I'm Tyler, a software engineer at Kratos Defense & Security. At home I run a Linux server that holds things I can't lose: a few Postgres databases, a self-hosted secret store, and the home directories of the Claude Code agents that run on it. On that box a backup only counts once something has restored it. The databases and the secret store get a full restore every night, the home directories get a sampled one, and a timed drill covers what the nightly checks can't. This post covers each one.

## Postgres: the dump has to match the database

Every night each database is dumped inside one `REPEATABLE READ` snapshot, and in that same snapshot the job takes an exact `count(*)` of every table. Then it restores the dump into a scratch Postgres 17 on a local socket and counts again. A dump only becomes the new latest copy when every table's count matches. Dumps go off the box to object storage, and each one is kept for 35 days.

The shared snapshot matters. pg_dump is consistent on its own, but counts taken in a separate transaction would include rows written after the dump started, and would never match it.

## The secret store: restore it every night

The secret store is exported and dumped, encrypted with age, and uploaded. A separate timer downloads it every night, decrypts it, restores it, and checks that every live key name is there. An encrypted backup you've never decrypted is still a guess.

## Home directories: restic, then a sample

Every home directory goes to a restic repository. After each nightly backup the job runs `restic check` over a slice of the stored data, then restores a random sample of files from three homes into a scratch directory and compares each one with the live file by sha256.

## Don't trust the job, check where it wrote

A backup job can exit 0 and still leave nothing useful behind. So a separate monitor never reads the job's exit code. It reads the bucket itself and fails unless the newest dump is under 26 hours old and its `verify.txt` starts with `restore: match`. When it fails, its systemd unit fails, and every failed unit on this box opens a GitHub issue automatically. A silent backup becomes an open issue the next morning.

## The drill

The nightly checks catch a backup that won't restore. They don't say how long a full recovery takes. So on October 30 I ran a timed drill: the secret store and one entire home directory, restored into scratch space, with recovery time measured from the first download to a verified restore.

| backup | recovery time | what was proven |
|---|---|---|
| secret store | **28 s** | every live key name present after the restore |
| one home directory | **1 m 25 s** | 111,555 files and 7.21 GB, `restic restore --verify` clean, 186 of 186 sampled files byte-identical by sha256 |

A second run gave 21 s and 1 m 36 s. Every backup but one runs nightly, so for those the most data I can lose is about a day.

## The one that runs weekly

One mostly idle database lives on a hosted free tier with a 5 GB monthly transfer cap. A nightly dump would have spent that allowance in about ten days, so the backup would have been the thing that broke it. That one runs weekly on purpose.