infra: add optional data volume bind-mount phase

[boidushya]

Why

Root disk on prod filled up about a month back, which quietly killed daily B2 backups. cache.db is 12GB now, and the backup endpoint writes a full snapshot to local disk before rclone ships it. There wasn't space for that snapshot, so the file never landed, and the upload script kept finding an empty backups dir every morning.

We just moved /var/lib/lyrics-api onto a 50GB Hetzner Cloud Volume and bind-mounted it in place by hand on the box. This commits that change as an idempotent phase so a fresh bootstrap reproduces it.

What

infra/phases/11-data-volume.sh. Skipped entirely if LYRICS_API_VOLUME_ID is unset. If set, it:

• Confirms the volume is attached and Hetzner has auto-mounted it at /mnt/HC_Volume_<id>.
• rsyncs any existing /var/lib/lyrics-api contents onto the volume's lyrics-api/ subdir. The original is renamed to /var/lib/lyrics-api.pre-volume.<timestamp> rather than deleted, so the operator can verify the migration before reclaiming the space.
• Adds the bind mount to /etc/fstab with x-systemd.requires-mounts-for= so boot order is explicit (volume mount, then bind).
• Drops a systemd override on lyrics-api.service adding RequiresMountsFor=/var/lib/lyrics-api. Defense in depth: if the bind ever fails to mount, the service refuses to start instead of silently writing a fresh empty cache.db to the root disk.

Plus the supporting tweaks: secrets.env.example gets the new LYRICS_API_VOLUME_ID knob (blank by default), and the README's Manual Steps section calls out the volume as an optional extra at provisioning time.

Test plan

• Fresh box, no volume configured: phase 11 logs the skip line and exits 0, nothing else changes.
• Fresh box with volume attached and LYRICS_API_VOLUME_ID set: phase 11 sets up the bind mount, lyrics-api starts cleanly.
• Existing box with data already in /var/lib/lyrics-api (the prod case): phase 11 migrates the data, preserves the original at .pre-volume.<timestamp>, restarts the service. Verified manually on prod before this PR.
• Reboot: bind mount returns via fstab, service waits for it before starting.

[better-lyrics-harmonizer]

Preview torn down

This PR's ephemeral environment was cleaned up.

Detail	Value
Status	Torn down
Branch	infra/volume-mount-phase
Last commit	a89ba80
Closed	2026-06-09 09:28:14 UTC
Workflow	run #27196896250

_{Reopen the PR if you need to spin it back up.}