Skip to content

fix(backup): isolate component failures; disable apport fleet-wide#293

Merged
swares merged 1 commit into
mainfrom
fix/backup-resilience-and-apport
Jul 25, 2026
Merged

fix(backup): isolate component failures; disable apport fleet-wide#293
swares merged 1 commit into
mainfrom
fix/backup-resilience-and-apport

Conversation

@swares

@swares swares commented Jul 25, 2026

Copy link
Copy Markdown
Owner

backup-cloud ran the kubectl-dependent dumps first under 'set -e', so on
2026-07-25 a single unreachable pod aborted the whole run — and the etcd
snapshots and k3s server token, which are plain file copies with no cluster
dependency and are the most critical items in the backup, never reached R2.
One flaky node cost a day of offsite cluster state for a reason unrelated
to cluster state.

Now: dependency-free items are staged first, each cluster-dependent dump is
individually non-fatal, the restic backup always runs, and the script exits
non-zero at the end if anything was missed. A bad node costs one component
and an alert instead of everything.

Postgres dumps are validated on DECOMPRESSED size. An empty pg_dump gzips to
a ~20-byte archive that passes both a size test and 'gzip -t' — the same
well-formed-but-empty shape as the 4 KB etcd snapshots that started this.

Also:

  • backup-verify no longer uses restic's directory-filter form of 'ls'.
    Current restic documents it; 0.12.1 does not behave the same way, and it
    reported the server token missing from a snapshot that contained it.
  • Vault preflight in backup-cloud. The reads carry no_log, which hides the
    error as well as the secret, so an expired token surfaced only as
    'censored due to no_log'.
  • check_mode: false on the read-only Vault probes. Without it --check
    skipped them and then died parsing nothing behind no_log.
  • TMPDIR renamed to STAGE; it is rm -rf'd and should not shadow the
    conventional temp variable.
  • New apport.yml. A wedged apport had burned a full core on n150-2 for
    3 weeks, taking systemd with it: k3s down, etcd quorum to 2/3, lldap
    stranded on its local-path PVC, SSO down, recovery via SysRq. n150-1 was
    found in the same state, one crash from the same outage.

Refs: docs/REVIEW-2026-07-24.md M-new-1/2/4, C8

backup-cloud ran the kubectl-dependent dumps first under 'set -e', so on
2026-07-25 a single unreachable pod aborted the whole run — and the etcd
snapshots and k3s server token, which are plain file copies with no cluster
dependency and are the most critical items in the backup, never reached R2.
One flaky node cost a day of offsite cluster state for a reason unrelated
to cluster state.

Now: dependency-free items are staged first, each cluster-dependent dump is
individually non-fatal, the restic backup always runs, and the script exits
non-zero at the end if anything was missed. A bad node costs one component
and an alert instead of everything.

Postgres dumps are validated on DECOMPRESSED size. An empty pg_dump gzips to
a ~20-byte archive that passes both a size test and 'gzip -t' — the same
well-formed-but-empty shape as the 4 KB etcd snapshots that started this.

Also:
- backup-verify no longer uses restic's directory-filter form of 'ls'.
  Current restic documents it; 0.12.1 does not behave the same way, and it
  reported the server token missing from a snapshot that contained it.
- Vault preflight in backup-cloud. The reads carry no_log, which hides the
  error as well as the secret, so an expired token surfaced only as
  'censored due to no_log'.
- check_mode: false on the read-only Vault probes. Without it --check
  skipped them and then died parsing nothing behind no_log.
- TMPDIR renamed to STAGE; it is rm -rf'd and should not shadow the
  conventional temp variable.
- New apport.yml. A wedged apport had burned a full core on n150-2 for
  3 weeks, taking systemd with it: k3s down, etcd quorum to 2/3, lldap
  stranded on its local-path PVC, SSO down, recovery via SysRq. n150-1 was
  found in the same state, one crash from the same outage.

Refs: docs/REVIEW-2026-07-24.md M-new-1/2/4, C8
@swares
swares merged commit 1ac7348 into main Jul 25, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant