Backups you can restore

A backup you've never restored is a hypothesis, not a backup. This guide builds a small, real backup system: a script that runs on a schedule, keeps old versions, tells you when it fails — and a tested restore you run yourself. That's step 4 of the roadmap, and it's the step that makes everything before it matter.

What we're backing up

Two kinds of data, two kinds of care:

• App data (volumes) — e.g. Uptime Kuma's uptime-kuma-data volume. Plain files once the container is stopped, so a tar archive is enough.
• Databases — running databases must be dumped through the database's own tool (pg_dump, mysqldump) while they run. Copying a live database's files usually produces a corrupt backup. App-consistent means dump-first.

Config files (/opt/*/compose.yml, .env) ride along too — without them you'd have to remember how everything was configured.

1. The backup script

Create /opt/backup/backup.sh:

#!/usr/bin/env bash
# backup.sh — archive app volumes, DB dumps, and configs; keep N days; report failures.
set -euo pipefail

# ---- credentials live in a root-only env file, NOT in this script ----
set -a; . /opt/backup/backup.env; set +a

STAMP="$(date +%Y%m%d-%H%M%S)"
DEST="/var/backups/selfhost/$STAMP"
mkdir -p "$DEST"

log() { echo "[backup $STAMP] $*"; }

# 1. App-consistent DB dump (example: Immich on Postgres)
log "dumping databases"
mkdir -p "$DEST/db"
docker compose -f /opt/immich/compose.yml exec -T database \
  pg_dump -U postgres immich > "$DEST/db/immich.sql"

# 2. Stop app stacks so volume archives are consistent
log "stopping app stacks"
docker compose -f /opt/uptime-kuma/compose.yml stop

# 3. Archive data volumes via their host paths
log "archiving volumes"
mkdir -p "$DEST/volumes"
docker run --rm -v uptime-kuma-data:/data:ro -v "$DEST/volumes":/out alpine \
  tar czf /out/uptime-kuma-data.tar.gz -C /data .

# 4. Archive compose files and env files (the "how it was configured" part)
log "archiving configs"
mkdir -p "$DEST/configs"
cp /opt/uptime-kuma/compose.yml "$DEST/configs/"

# 5. Restart everything
log "restarting app stacks"
docker compose -f /opt/uptime-kuma/compose.yml start

# 6. Retention: keep 14 days
find /var/backups/selfhost -maxdepth 1 -mindepth 1 -mtime +14 -exec rm -rf {} +

log "backup complete: $DEST"

Two deliberate design choices to notice: the script fails loudly (set -euo pipefail — any error stops it with a non-zero exit), and credentials come from a separate env file.

2. Credentials, handled properly

The script needs secrets (database passwords for dumps). Storing them in the script itself means anyone who can read the script has them, and scripts have a way of getting pasted into forums. Instead:

sudo mkdir -p /opt/backup && sudo chmod 700 /opt/backup
sudo nano /opt/backup/backup.env

# /opt/backup/backup.env — root-only
PGPASSWORD=your-long-database-password
# export whatever else your dumps need, e.g. MYSQL_PWD

sudo chmod 600 /opt/backup/backup.env
sudo chmod 700 /opt/backup/backup.sh

Root-only permissions (700/600) plus the environment variable convention (PGPASSWORD is read by pg_dump automatically) keep secrets out of the process list and out of casual view. Nothing here protects against someone who already has root — but root can read everything anyway; the point is to not leak secrets to lesser accounts and not to scatter them across files.

3. Schedule it, and make failures find you

sudo crontab -e

# Nightly at 03:30; output (including errors) mailed to root
30 3 * * * /opt/backup/backup.sh >> /var/log/selfhost-backup.log 2>&1

A backup job that fails silently is worse than no job — you trust it. Pick your failure channel and wire the exit code to it. A simple pattern that works anywhere:

30 3 * * * /opt/backup/backup.sh >> /var/log/selfhost-backup.log 2>&1 \
  || echo "selfhost backup FAILED $(date)" | mail -s "backup FAILED" you@example.com

That requires a working mail setup (an SMTP relay like your provider's, or a small relay container). If you'd rather not run mail, send the failure line to a webhook — Uptime Kuma can receive push alerts, or use a free ntfy.sh topic:

|| curl -fsS -d "selfhost backup FAILED $(date)" ntfy.sh/your-secret-topic

Whichever you choose: a failure must become a message you actually see. Then check the log weekly, or have your monitoring watch /var/log/selfhost-backup.log for "FAILED".

4. The restore — into a test area, then verify

Restoring over your only copy is how backups become losses. Always restore beside production first. For a volume restore:

# 1. Fresh volume to restore INTO
docker volume create kuma-restore-test

# 2. Extract the archive into it
docker run --rm -v kuma-restore-test:/data -v /var/backups/selfhost/20261001-033000/volumes:/in alpine \
  sh -c "rm -rf /data/* && tar xzf /in/uptime-kuma-data.tar.gz -C /data"

# 3. Start a throwaway copy of the app against the test volume
mkdir -p /tmp/restore-test && cd /tmp/restore-test
sed 's/uptime-kuma-data/kuma-restore-test/; s/3001:3001/3002:3002/' \
  /opt/uptime-kuma/compose.yml > compose.yml
docker compose -f compose.yml up -d

# 4. Verify through the tunnel — you should see your real monitors
ssh -L 3002:localhost:3002 deploy@your-server-ip
# open http://localhost:3002 — data there? restore works.

# 5. Clean up the test
docker compose -f compose.yml down
docker volume rm kuma-restore-test

For a database restore, the dump makes it app-consistent by construction:

docker compose -f /opt/immich/compose.yml exec -T database \
  psql -U postgres -c "DROP DATABASE immich_restore_test;" || true
docker compose -f /opt/immich/compose.yml exec -T database \
  psql -U postgres -c "CREATE DATABASE immich_restore_test;"
docker compose -f /opt/immich/compose.yml exec -T database \
  psql -U postgres -d immich_restore_test < /var/backups/selfhost/20261001-033000/db/immich.sql
# then point a test Immich at immich_restore_test and open a few photos

Verification means opening the app and seeing your data — not "the command exited zero". A zero exit with an empty archive restores nothing and tells you nothing.

5. The off-server copy (brief, but not optional)

Everything above lives on the same disk as the data. One disk failure takes both. The fix is one more line in the script — sync the finished backup folder off-machine. restic to any S3-compatible storage, rsync to another machine you control, even rclone to a cloud drive — pick one, encrypt at rest (restic does by default), and include it in your failure reporting. A starter restic block:

restic -r s3:https://s3.example.com/selfhost-backups backup /var/backups/selfhost \
  --password-file /opt/backup/restic-password
restic -r s3:https://s3.example.com/selfhost-backups forget --keep-within 30d --prune

The restic password file gets the same 600 treatment. Losing your restic password means losing the off-site copies — store it in your password manager, not only on the server.

Troubleshooting

• pg_dump: "password authentication failed": PGPASSWORD isn't reaching the exec'd process — confirm backup.env is readable by the cron user (root) and sourced before the dump.
• Archive is empty: usually the volume name doesn't match (docker volume ls). The alpine helper silently archives nothing if the source mount is missing — that's why the script logs sizes; check the log.
• Backup takes longer than apps tolerate being down: move the volume archives to a copy-on-write filesystem (ZFS/btrfs snapshots) or accept crash-consistency for low-stakes apps and dump-first consistency for databases only.
• Disk filling with old backups: retention runs after a successful backup, by design — but if the script dies before step 6, nothing prunes. Let your monitoring alert on disk usage too.