fix: cap systemd journal to 500M on all Linux hosts#290
Merged
Conversation
… whisper to 0.6.0-rc.3-cpu
- ExternalSecret pulls groq-api-key + gemini-api-key from secret/lab/cloud-ai - Deployment mounts litellm-cloud-ai Secret as env vars - ConfigMap adds 'cloud' model group (llama-3.3-70b-versatile + gemini-2.0-flash) - Router fallbacks: chat → cloud, chat-cpu → cloud on any failure - NetworkPolicy allows litellm HTTPS egress to public internet for cloud APIs
- Authelia: add bypass rules for /v1/ and /health/ on ai.apps.lab.home.arpa - ai-gateway ingress: add authelia-forwardauth middleware - SSO.md: move LiteLLM to Completed
Captures work done and deferred across the 2026-07-14–23 sessions: - healthchecks.io dead-man's switch (5 checks) - Groq + Gemini cloud fallback in LiteLLM - LiteLLM ForwardAuth SSO via Authelia - Kyverno PolicyExceptions cleanup - lldap SQLite backup wired - OVMS / Vault OIDC / rknpu / offsite Track 2 deferred with rationale - Current lab state snapshot
Add awscli install and a tofu-state copy step to backup-cloud.yml. Before the nightly restic snapshot, the script now pulls the tofu-state bucket from MinIO (using root creds from Vault secret/lab/minio) into $TMPDIR, which restic then includes in the R2 snapshot. MinIO creds are written to /etc/restic/cloud.env alongside the existing R2 credentials so no separate env file is needed. aws s3 cp overrides AWS_ACCESS_KEY_ID/SECRET with the MinIO root creds for that one call; the rest of the script uses the R2 creds from the EnvironmentFile as before.
Unbounded journal growth on n150-1 caused disk pressure (91% of 49G root) → kubelet evicted all monitoring namespace pods including Grafana, Alertmanager, and Prometheus (2026-07-23). Adds a journald drop-in (SystemMaxUse=500M, SystemKeepFree=1G) and vacuums existing journals on first run. Targets all:!x86_nodes:!embedded:!standalone_vms.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Root cause of monitoring outage 2026-07-23: n150-1 root disk hit 91% from 4.1G unbounded journal → DiskPressure → mass eviction of monitoring namespace.
Fixed by manual vacuum (3.5G freed) and this playbook wiring a permanent cap.