Skip to content

fix(monitoring): route critical alerts off the LAN, and watch the watcher #295

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
swares merged 1 commit into from
Jul 25, 2026
+96 −11
Merged

fix(monitoring): route critical alerts off the LAN, and watch the watcher #295

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/REVIEW-2026-07-24.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -302,10 +302,10 @@ directory carries the password. `LAB-URLS.md` has the same shape of problem

| # | Finding | File |
|---|---------|------|
| H25 | **Nothing pages a human.** Alertmanager has exactly two receivers: `null` and `m5stack` (an HTTP POST to an ESP32 on the LAN). ArgoCD notifications go to the same device. Asleep or display off = every alert silently discarded | `gitops/apps/monitoring.yaml:166-192` |
| H25 | **FIXED 2026-07-25** (branch `fix/alert-routing`). Was: Alertmanager had exactly two receivers, `null` and `m5stack` (an HTTP POST to an ESP32 on the LAN), so asleep or display-off meant every alert was silently discarded. Now `severity=critical` also routes to ntfy (topic URL in Vault, `continue: true` so the display still sees it), and the always-firing `Watchdog` alert — previously discarded to `null` — forwards to a healthchecks.io check, making the monitoring stack its own dead-man's switch. ArgoCD notifications still go only to the M5Stack; separate item. | `gitops/apps/monitoring.yaml` |
| H26 | `kubeEtcd: enabled: false` — **zero etcd metrics, therefore zero quorum-loss alerting** on a 3-node HA cluster | `gitops/apps/monitoring.yaml:221-222` |
| H27 | `LabBackupEtcdSilent` fires at >25 h against a **weekly** timer — permanently critical 6 days out of 7, training you to ignore the one alert group that matters most (see C1) | `lab-alerts.yaml:80-95` vs `backup-etcd.timer.j2:5` |
| H28 | Prometheus scrapes decommissioned `192.168.1.70` (ldap-1) — `LabExternalHostDown` critical has been firing for ~3 weeks — while `.116` (Pi-hole secondary), `.217` (4th DNS) and `.128` (**Vault**) are not scraped at all | `gitops/apps/monitoring.yaml:129-134` |
| H28 | **FIXED 2026-07-25.** Dropped decommissioned `192.168.1.70`; added `.116` (Pi-hole secondary), `.217` (4th DNS), `.128` (**Vault** — every ExternalSecret depends on it) and `.64` (build agent). Also found and fixed a second stale reference: `LabVMUnexpectedShutoff` filtered on `domain=~"ldap-1"`, a VM decommissioned 2026-07-04, so the rule had been incapable of firing for three weeks — retargeted at `gitlab-1`. | `gitops/apps/monitoring.yaml`, `lab-alerts.yaml:137` |
| H29 | No cert-manager ServiceMonitor and no expiry rule; no blackbox exporter, so nothing verifies that `api.lab.home.arpa` and `*.apps.lab.home.arpa` actually resolve — despite `CLAUDE.md:49` calling DNS load-bearing | `gitops/workloads/cert-manager/` |
| H30 | `.claude/settings.json` is a MicroShift artifact: it gates `oc apply`/`oc delete`/`oc adm`, which `CLAUDE.md:42` forbids using. `kubectl` appears **3 times, all in `allow`** — every `kubectl` mutation is unguarded, directly contradicting the table at `docs/SECURITY.md:52-54`. Also denies `systemctl stop nfs-server` while Ubuntu's unit is `nfs-kernel-server`, and no `sudo`-prefixed variants of any deny rule | `.claude/settings.json:4-57` |

Expand Down
95 changes: 87 additions & 8 deletions gitops/apps/monitoring.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,12 +127,25 @@ spec:
# Keeping them here caused port 9100 conflicts with the DaemonSet.
static_configs:
- targets:
# Corrected 2026-07-25 (H28). Previously this list scraped
# ldap-1 (192.168.1.70), decommissioned 2026-07-04 — so
# LabExternalHostDown had been firing critical for ~3 weeks,
# training the eye to ignore it. Meanwhile Vault, the Pi-hole
# secondary, the 4th DNS server and the build agent were not
# scraped at all: real outages there were invisible.
# opi-zero2w-4 was already listed and has in fact been
# unreachable since at least 2026-07-25 (M-new-5) — the
# target was right, nothing was routing the alert anywhere.
- 192.168.1.50:9100 # gitlab-1 (GitLab CE VM)
- 192.168.1.70:9100 # ldap-1 (lldap VM)
- 192.168.1.148:9100 # octopi (DNS primary)
- 192.168.1.184:9100 # opi-zero2w-1 (DNS secondary)
- 192.168.1.188:9100 # opi-zero2w-2 (MQTT)
- 192.168.1.99:9100 # opi-zero2w-4
- 192.168.1.148:9100 # octopi (DNS primary, Pi-hole)
- 192.168.1.116:9100 # rpi4b (DNS secondary, Pi-hole)
- 192.168.1.184:9100 # opi-zero2w-1 (DNS tertiary, dnsmasq)
- 192.168.1.217:9100 # opi-zero2w-3 (DNS quaternary, dnsmasq)
- 192.168.1.188:9100 # opi-zero2w-2 (MQTT primary)
- 192.168.1.99:9100 # opi-zero2w-4 (MQTT secondary/HA bridge)
- 192.168.1.128:9100 # rpi5 (HashiCorp Vault) — every
# ExternalSecret in the lab depends on it
- 192.168.1.64:9100 # xu3-1 (build agent)
relabel_configs:
- source_labels: [__address__]
regex: '([^:]+):\d+'
Expand All @@ -155,6 +168,12 @@ spec:
alertmanagerSpec:
nodeSelector:
kubernetes.io/hostname: n150-1
# Mounts the ExternalSecret at /etc/alertmanager/secrets/<name>/<key>
# so receivers can reference URLs via url_file and keep the ntfy topic
# (which IS the password) out of git.
# See gitops/workloads/monitoring/alertmanager-externalsecret.yaml
secrets:
- alertmanager-endpoints
storage:
volumeClaimTemplate:
spec:
Expand All @@ -163,8 +182,30 @@ spec:
resources:
requests:
storage: 2Gi
# Routing: all real alerts → m5stack-adapter webhook.
# Watchdog and InfoInhibitor go to the null receiver (expected noise).
# ── Alert routing ────────────────────────────────────────────────────
#
# Until 2026-07-25 every alert went to exactly one place: a webhook to
# an M5Stack display on the LAN. If you were asleep, out, or the display
# was off, the alert was discarded — silently. A crash reporter burned a
# CPU core on n150-1 for three weeks and nothing said a word.
#
# Three destinations now, each for a different failure class:
#
# m5stack — everything, ambient. Unchanged; still the at-a-glance view.
# ntfy — severity=critical only, to your phone. Low volume so it
# stays trustworthy; priority/tags are set in the Vault URL.
# watchdog — the Watchdog alert, which kube-prometheus-stack fires
# CONTINUOUSLY by design, forwarded to a healthchecks.io
# check. This is the one that watches the watcher: if
# Prometheus, Alertmanager, or n150-1 dies, the pings stop
# and healthchecks.io alerts from OUTSIDE the lab.
# Previously Watchdog was routed to 'null' — discarding
# precisely the signal that proves monitoring is alive.
#
# Note what this does and does not fix: Alertmanager runs IN the cluster,
# so a new receiver only helps when the cluster is up. Cluster-down is
# covered by the Watchdog route above and by healthchecks.io's h4-core
# heartbeat. See docs/REVIEW-2026-07-24.md H25.
config:
global:
resolve_timeout: 5m
Expand All @@ -180,17 +221,55 @@ spec:
repeat_interval: 4h
receiver: 'null'
routes:
# Dead-man's switch. Watchdog always fires, so this route always
# has traffic; repeat_interval drives the ping cadence. 5m repeat
# against a 15m healthchecks Period leaves room for two missed
# pings before it alarms — tight enough to notice, loose enough
# not to flap on a slow reconcile.
- matchers:
- alertname =~ "InfoInhibitor|Watchdog"
- alertname = "Watchdog"
receiver: watchdog
group_wait: 0s
group_interval: 5m
repeat_interval: 5m
continue: false

- matchers:
- alertname = "InfoInhibitor"
receiver: 'null'

# Critical goes to BOTH the phone and the display. `continue: true`
# is what makes the m5stack route below still see it — without it,
# Alertmanager stops at the first matching route and the display
# would go quiet for exactly the alerts that matter most.
- matchers:
- severity = "critical"
receiver: ntfy
continue: true

- receiver: m5stack
receivers:
- name: 'null'

- name: m5stack
webhook_configs:
- url: 'http://m5stack-adapter.monitoring.svc.cluster.local:8080/webhook'
send_resolved: true

# url_file, not url: keeps the ntfy topic out of git. The topic is
# the only credential ntfy has — anyone who knows it can read every
# alert and publish fakes.
- name: ntfy
webhook_configs:
- url_file: /etc/alertmanager/secrets/alertmanager-endpoints/ntfy-url
send_resolved: true
max_alerts: 5

- name: watchdog
webhook_configs:
- url_file: /etc/alertmanager/secrets/alertmanager-endpoints/watchdog-url
send_resolved: false

# ---- Node exporter — runs on all nodes ----
prometheus-node-exporter:
tolerations:
Expand Down
8 changes: 7 additions & 1 deletion gitops/workloads/monitoring/lab-alerts.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,8 +133,14 @@ spec:
- alert: LabVMUnexpectedShutoff
# State 5=shut off. Fires only for VMs that should always be running.
# Adjust the domain filter to match your always-on VMs.
#
# Corrected 2026-07-25: this filtered on `ldap-1`, a VM decommissioned
# on 2026-07-04 when lldap moved into k3s. The rule had therefore been
# dead for three weeks — it could never fire, so an actual always-on VM
# shutting down would have gone unnoticed. gitlab-1 is now the only
# always-on VM in the lab (see README fleet table).
expr: |
libvirt_domain_state_code{domain=~"ldap-1"} == 5
libvirt_domain_state_code{domain=~"gitlab-1"} == 5
for: 5m
labels:
severity: warning
Expand Down