fix/backups#291
Merged
Merged
Conversation
swares
commented
Jul 25, 2026
swares
commented
Owner
- fix: remove stale Kyverno PolicyExceptions — ollama pinned to 0.32.1, whisper to 0.6.0-rc.3-cpu
- feat: add Groq + Gemini cloud fallback to LiteLLM
- docs: document Groq/Gemini cloud fallback, fix litellm-vip placeholder URL
- feat: gate LiteLLM UI with Authelia ForwardAuth
- docs: session notes 2026-07-23
- feat: back up OpenTofu state from MinIO to Cloudflare R2
- fix: cap systemd journal to 500M on all Linux hosts
- fix(backup): make backups restorable, bounded, and self-verifying
… whisper to 0.6.0-rc.3-cpu
- ExternalSecret pulls groq-api-key + gemini-api-key from secret/lab/cloud-ai - Deployment mounts litellm-cloud-ai Secret as env vars - ConfigMap adds 'cloud' model group (llama-3.3-70b-versatile + gemini-2.0-flash) - Router fallbacks: chat → cloud, chat-cpu → cloud on any failure - NetworkPolicy allows litellm HTTPS egress to public internet for cloud APIs
- Authelia: add bypass rules for /v1/ and /health/ on ai.apps.lab.home.arpa - ai-gateway ingress: add authelia-forwardauth middleware - SSO.md: move LiteLLM to Completed
Captures work done and deferred across the 2026-07-14–23 sessions: - healthchecks.io dead-man's switch (5 checks) - Groq + Gemini cloud fallback in LiteLLM - LiteLLM ForwardAuth SSO via Authelia - Kyverno PolicyExceptions cleanup - lldap SQLite backup wired - OVMS / Vault OIDC / rknpu / offsite Track 2 deferred with rationale - Current lab state snapshot
Add awscli install and a tofu-state copy step to backup-cloud.yml. Before the nightly restic snapshot, the script now pulls the tofu-state bucket from MinIO (using root creds from Vault secret/lab/minio) into $TMPDIR, which restic then includes in the R2 snapshot. MinIO creds are written to /etc/restic/cloud.env alongside the existing R2 credentials so no separate env file is needed. aws s3 cp overrides AWS_ACCESS_KEY_ID/SECRET with the MinIO root creds for that one call; the rest of the script uses the R2 creds from the EnvironmentFile as before.
Unbounded journal growth on n150-1 caused disk pressure (91% of 49G root) → kubelet evicted all monitoring namespace pods including Grafana, Alertmanager, and Prometheus (2026-07-23). Adds a journald drop-in (SystemMaxUse=500M, SystemKeepFree=1G) and vacuums existing journals on first run. Targets all:!x86_nodes:!embedded:!standalone_vms.
Follows the backup-etcd fix. Four defects, all of the same shape — a job that runs, exits 0, and does not do its job: - The k3s server token was backed up nowhere. k3s derives the datastore's AES-256 encryption key from it, so restoring onto new hardware is impossible without it. Losing the H4 alone was survivable (n150-1/2 hold the same token), but total-site loss meant every snapshot in R2 - the tier that exists for precisely that case - was unrestorable. backup-cloud now stages it and backup-verify asserts its presence. - R2 retention never deleted anything. A mktemp staging dir put a unique path in every snapshot, and restic forget groups by host+paths, so each snapshot formed its own retention group. Fixed with a stable staging path and --group-by host,tags. - The cold-sec copy was never pruned (40 snapshots vs the primary's 19) and its failures were swallowed by a '-' prefix on ExecStartPost. Now bounded at 14d/8w/12m - deliberately deeper than the primary, it is the long-history archive - and failures are fatal. - restic 0.12.1 is a latent hazard: the copy command's argument semantics changed and --repo2 no longer appears in current docs, so an apt upgrade could silently reverse the copy direction. backup-nas-copy.sh detects the version and uses the unambiguous form for it, refusing to run if it cannot parse a numeric version. Adds backup-verify.timer (weekly, read-only) which checks repo integrity AND asserts content - a clean 'restic check' proves a repo is well-formed, not that the right files are in it. Adds docs/BACKUP-RESTORE.md with a corrected 3-server HA etcd restore procedure, verified against current k3s docs. The procedure previously in RUNBOOK.md used a subcommand that does not exist, the wrong directory, and single-node steps for a 3-server cluster; it has been removed and now points at the new document. Refs: docs/REVIEW-2026-07-24.md C8, M-new-1, M-new-2, M-new-4
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.