fix(backup): isolate component failures; disable apport fleet-wide#293
Merged
Conversation
backup-cloud ran the kubectl-dependent dumps first under 'set -e', so on 2026-07-25 a single unreachable pod aborted the whole run — and the etcd snapshots and k3s server token, which are plain file copies with no cluster dependency and are the most critical items in the backup, never reached R2. One flaky node cost a day of offsite cluster state for a reason unrelated to cluster state. Now: dependency-free items are staged first, each cluster-dependent dump is individually non-fatal, the restic backup always runs, and the script exits non-zero at the end if anything was missed. A bad node costs one component and an alert instead of everything. Postgres dumps are validated on DECOMPRESSED size. An empty pg_dump gzips to a ~20-byte archive that passes both a size test and 'gzip -t' — the same well-formed-but-empty shape as the 4 KB etcd snapshots that started this. Also: - backup-verify no longer uses restic's directory-filter form of 'ls'. Current restic documents it; 0.12.1 does not behave the same way, and it reported the server token missing from a snapshot that contained it. - Vault preflight in backup-cloud. The reads carry no_log, which hides the error as well as the secret, so an expired token surfaced only as 'censored due to no_log'. - check_mode: false on the read-only Vault probes. Without it --check skipped them and then died parsing nothing behind no_log. - TMPDIR renamed to STAGE; it is rm -rf'd and should not shadow the conventional temp variable. - New apport.yml. A wedged apport had burned a full core on n150-2 for 3 weeks, taking systemd with it: k3s down, etcd quorum to 2/3, lldap stranded on its local-path PVC, SSO down, recovery via SysRq. n150-1 was found in the same state, one crash from the same outage. Refs: docs/REVIEW-2026-07-24.md M-new-1/2/4, C8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
backup-cloud ran the kubectl-dependent dumps first under 'set -e', so on
2026-07-25 a single unreachable pod aborted the whole run — and the etcd
snapshots and k3s server token, which are plain file copies with no cluster
dependency and are the most critical items in the backup, never reached R2.
One flaky node cost a day of offsite cluster state for a reason unrelated
to cluster state.
Now: dependency-free items are staged first, each cluster-dependent dump is
individually non-fatal, the restic backup always runs, and the script exits
non-zero at the end if anything was missed. A bad node costs one component
and an alert instead of everything.
Postgres dumps are validated on DECOMPRESSED size. An empty pg_dump gzips to
a ~20-byte archive that passes both a size test and 'gzip -t' — the same
well-formed-but-empty shape as the 4 KB etcd snapshots that started this.
Also:
Current restic documents it; 0.12.1 does not behave the same way, and it
reported the server token missing from a snapshot that contained it.
error as well as the secret, so an expired token surfaced only as
'censored due to no_log'.
skipped them and then died parsing nothing behind no_log.
conventional temp variable.
3 weeks, taking systemd with it: k3s down, etcd quorum to 2/3, lldap
stranded on its local-path PVC, SSO down, recovery via SysRq. n150-1 was
found in the same state, one crash from the same outage.
Refs: docs/REVIEW-2026-07-24.md M-new-1/2/4, C8