From 95cd1b189740c0fdb7c3adec5290fb41c2ef78ed Mon Sep 17 00:00:00 2001 From: Scott Wares Date: Sat, 25 Jul 2026 18:36:24 +0000 Subject: [PATCH] fix(monitoring): route critical alerts off the LAN, and watch the watcher Alertmanager had two receivers: null and a webhook to an M5Stack display on the LAN. Asleep, out, or display off meant every alert was silently discarded. A wedged apport burned a core on n150-1 for three weeks and nothing said a word. - severity=critical now also routes to ntfy (phone). continue: true keeps the M5Stack in the path, so nothing is lost. max_alerts: 5 caps a storm, because a phone that buzzes forty times gets muted. - The Watchdog alert, which kube-prometheus-stack fires continuously by design, went to null - discarding exactly the signal that proves monitoring is alive. It now pings a healthchecks.io check every 5m against a 15m period, so if Prometheus, Alertmanager or n150-1 dies, healthchecks alerts from outside the lab. The only prior heartbeat was h4-core. - Endpoint URLs come from Vault via ESO and are read with url_file. The ntfy topic IS the password; it must not be in git. Scrape targets (H28): dropped decommissioned 192.168.1.70, which had kept LabExternalHostDown firing critical for three weeks. Added .116, .217, .128 (Vault - every ExternalSecret depends on it) and .64. LabVMUnexpectedShutoff filtered on domain=~ldap-1, decommissioned 2026-07-04, so the rule was structurally unable to fire. Retargeted at gitlab-1. Refs: docs/REVIEW-2026-07-24.md H25, H28 --- docs/REVIEW-2026-07-24.md | 4 +- gitops/apps/monitoring.yaml | 95 +++++++++++++++++++-- gitops/workloads/monitoring/lab-alerts.yaml | 8 +- 3 files changed, 96 insertions(+), 11 deletions(-) diff --git a/docs/REVIEW-2026-07-24.md b/docs/REVIEW-2026-07-24.md index 2e99b7b..a9a9733 100644 --- a/docs/REVIEW-2026-07-24.md +++ b/docs/REVIEW-2026-07-24.md @@ -302,10 +302,10 @@ directory carries the password. `LAB-URLS.md` has the same shape of problem | # | Finding | File | |---|---------|------| -| H25 | **Nothing pages a human.** Alertmanager has exactly two receivers: `null` and `m5stack` (an HTTP POST to an ESP32 on the LAN). ArgoCD notifications go to the same device. Asleep or display off = every alert silently discarded | `gitops/apps/monitoring.yaml:166-192` | +| H25 | **FIXED 2026-07-25** (branch `fix/alert-routing`). Was: Alertmanager had exactly two receivers, `null` and `m5stack` (an HTTP POST to an ESP32 on the LAN), so asleep or display-off meant every alert was silently discarded. Now `severity=critical` also routes to ntfy (topic URL in Vault, `continue: true` so the display still sees it), and the always-firing `Watchdog` alert — previously discarded to `null` — forwards to a healthchecks.io check, making the monitoring stack its own dead-man's switch. ArgoCD notifications still go only to the M5Stack; separate item. | `gitops/apps/monitoring.yaml` | | H26 | `kubeEtcd: enabled: false` — **zero etcd metrics, therefore zero quorum-loss alerting** on a 3-node HA cluster | `gitops/apps/monitoring.yaml:221-222` | | H27 | `LabBackupEtcdSilent` fires at >25 h against a **weekly** timer — permanently critical 6 days out of 7, training you to ignore the one alert group that matters most (see C1) | `lab-alerts.yaml:80-95` vs `backup-etcd.timer.j2:5` | -| H28 | Prometheus scrapes decommissioned `192.168.1.70` (ldap-1) — `LabExternalHostDown` critical has been firing for ~3 weeks — while `.116` (Pi-hole secondary), `.217` (4th DNS) and `.128` (**Vault**) are not scraped at all | `gitops/apps/monitoring.yaml:129-134` | +| H28 | **FIXED 2026-07-25.** Dropped decommissioned `192.168.1.70`; added `.116` (Pi-hole secondary), `.217` (4th DNS), `.128` (**Vault** — every ExternalSecret depends on it) and `.64` (build agent). Also found and fixed a second stale reference: `LabVMUnexpectedShutoff` filtered on `domain=~"ldap-1"`, a VM decommissioned 2026-07-04, so the rule had been incapable of firing for three weeks — retargeted at `gitlab-1`. | `gitops/apps/monitoring.yaml`, `lab-alerts.yaml:137` | | H29 | No cert-manager ServiceMonitor and no expiry rule; no blackbox exporter, so nothing verifies that `api.lab.home.arpa` and `*.apps.lab.home.arpa` actually resolve — despite `CLAUDE.md:49` calling DNS load-bearing | `gitops/workloads/cert-manager/` | | H30 | `.claude/settings.json` is a MicroShift artifact: it gates `oc apply`/`oc delete`/`oc adm`, which `CLAUDE.md:42` forbids using. `kubectl` appears **3 times, all in `allow`** — every `kubectl` mutation is unguarded, directly contradicting the table at `docs/SECURITY.md:52-54`. Also denies `systemctl stop nfs-server` while Ubuntu's unit is `nfs-kernel-server`, and no `sudo`-prefixed variants of any deny rule | `.claude/settings.json:4-57` | diff --git a/gitops/apps/monitoring.yaml b/gitops/apps/monitoring.yaml index 337b8a7..d245283 100644 --- a/gitops/apps/monitoring.yaml +++ b/gitops/apps/monitoring.yaml @@ -127,12 +127,25 @@ spec: # Keeping them here caused port 9100 conflicts with the DaemonSet. static_configs: - targets: + # Corrected 2026-07-25 (H28). Previously this list scraped + # ldap-1 (192.168.1.70), decommissioned 2026-07-04 — so + # LabExternalHostDown had been firing critical for ~3 weeks, + # training the eye to ignore it. Meanwhile Vault, the Pi-hole + # secondary, the 4th DNS server and the build agent were not + # scraped at all: real outages there were invisible. + # opi-zero2w-4 was already listed and has in fact been + # unreachable since at least 2026-07-25 (M-new-5) — the + # target was right, nothing was routing the alert anywhere. - 192.168.1.50:9100 # gitlab-1 (GitLab CE VM) - - 192.168.1.70:9100 # ldap-1 (lldap VM) - - 192.168.1.148:9100 # octopi (DNS primary) - - 192.168.1.184:9100 # opi-zero2w-1 (DNS secondary) - - 192.168.1.188:9100 # opi-zero2w-2 (MQTT) - - 192.168.1.99:9100 # opi-zero2w-4 + - 192.168.1.148:9100 # octopi (DNS primary, Pi-hole) + - 192.168.1.116:9100 # rpi4b (DNS secondary, Pi-hole) + - 192.168.1.184:9100 # opi-zero2w-1 (DNS tertiary, dnsmasq) + - 192.168.1.217:9100 # opi-zero2w-3 (DNS quaternary, dnsmasq) + - 192.168.1.188:9100 # opi-zero2w-2 (MQTT primary) + - 192.168.1.99:9100 # opi-zero2w-4 (MQTT secondary/HA bridge) + - 192.168.1.128:9100 # rpi5 (HashiCorp Vault) — every + # ExternalSecret in the lab depends on it + - 192.168.1.64:9100 # xu3-1 (build agent) relabel_configs: - source_labels: [__address__] regex: '([^:]+):\d+' @@ -155,6 +168,12 @@ spec: alertmanagerSpec: nodeSelector: kubernetes.io/hostname: n150-1 + # Mounts the ExternalSecret at /etc/alertmanager/secrets// + # so receivers can reference URLs via url_file and keep the ntfy topic + # (which IS the password) out of git. + # See gitops/workloads/monitoring/alertmanager-externalsecret.yaml + secrets: + - alertmanager-endpoints storage: volumeClaimTemplate: spec: @@ -163,8 +182,30 @@ spec: resources: requests: storage: 2Gi - # Routing: all real alerts → m5stack-adapter webhook. - # Watchdog and InfoInhibitor go to the null receiver (expected noise). + # ── Alert routing ──────────────────────────────────────────────────── + # + # Until 2026-07-25 every alert went to exactly one place: a webhook to + # an M5Stack display on the LAN. If you were asleep, out, or the display + # was off, the alert was discarded — silently. A crash reporter burned a + # CPU core on n150-1 for three weeks and nothing said a word. + # + # Three destinations now, each for a different failure class: + # + # m5stack — everything, ambient. Unchanged; still the at-a-glance view. + # ntfy — severity=critical only, to your phone. Low volume so it + # stays trustworthy; priority/tags are set in the Vault URL. + # watchdog — the Watchdog alert, which kube-prometheus-stack fires + # CONTINUOUSLY by design, forwarded to a healthchecks.io + # check. This is the one that watches the watcher: if + # Prometheus, Alertmanager, or n150-1 dies, the pings stop + # and healthchecks.io alerts from OUTSIDE the lab. + # Previously Watchdog was routed to 'null' — discarding + # precisely the signal that proves monitoring is alive. + # + # Note what this does and does not fix: Alertmanager runs IN the cluster, + # so a new receiver only helps when the cluster is up. Cluster-down is + # covered by the Watchdog route above and by healthchecks.io's h4-core + # heartbeat. See docs/REVIEW-2026-07-24.md H25. config: global: resolve_timeout: 5m @@ -180,17 +221,55 @@ spec: repeat_interval: 4h receiver: 'null' routes: + # Dead-man's switch. Watchdog always fires, so this route always + # has traffic; repeat_interval drives the ping cadence. 5m repeat + # against a 15m healthchecks Period leaves room for two missed + # pings before it alarms — tight enough to notice, loose enough + # not to flap on a slow reconcile. - matchers: - - alertname =~ "InfoInhibitor|Watchdog" + - alertname = "Watchdog" + receiver: watchdog + group_wait: 0s + group_interval: 5m + repeat_interval: 5m + continue: false + + - matchers: + - alertname = "InfoInhibitor" receiver: 'null' + + # Critical goes to BOTH the phone and the display. `continue: true` + # is what makes the m5stack route below still see it — without it, + # Alertmanager stops at the first matching route and the display + # would go quiet for exactly the alerts that matter most. + - matchers: + - severity = "critical" + receiver: ntfy + continue: true + - receiver: m5stack receivers: - name: 'null' + - name: m5stack webhook_configs: - url: 'http://m5stack-adapter.monitoring.svc.cluster.local:8080/webhook' send_resolved: true + # url_file, not url: keeps the ntfy topic out of git. The topic is + # the only credential ntfy has — anyone who knows it can read every + # alert and publish fakes. + - name: ntfy + webhook_configs: + - url_file: /etc/alertmanager/secrets/alertmanager-endpoints/ntfy-url + send_resolved: true + max_alerts: 5 + + - name: watchdog + webhook_configs: + - url_file: /etc/alertmanager/secrets/alertmanager-endpoints/watchdog-url + send_resolved: false + # ---- Node exporter — runs on all nodes ---- prometheus-node-exporter: tolerations: diff --git a/gitops/workloads/monitoring/lab-alerts.yaml b/gitops/workloads/monitoring/lab-alerts.yaml index f03f45b..35abd7f 100644 --- a/gitops/workloads/monitoring/lab-alerts.yaml +++ b/gitops/workloads/monitoring/lab-alerts.yaml @@ -133,8 +133,14 @@ spec: - alert: LabVMUnexpectedShutoff # State 5=shut off. Fires only for VMs that should always be running. # Adjust the domain filter to match your always-on VMs. + # + # Corrected 2026-07-25: this filtered on `ldap-1`, a VM decommissioned + # on 2026-07-04 when lldap moved into k3s. The rule had therefore been + # dead for three weeks — it could never fire, so an actual always-on VM + # shutting down would have gone unnoticed. gitlab-1 is now the only + # always-on VM in the lab (see README fleet table). expr: | - libvirt_domain_state_code{domain=~"ldap-1"} == 5 + libvirt_domain_state_code{domain=~"gitlab-1"} == 5 for: 5m labels: severity: warning