fix(monitoring): route critical alerts off the LAN, and watch the watcher#295
Merged
Conversation
…cher Alertmanager had two receivers: null and a webhook to an M5Stack display on the LAN. Asleep, out, or display off meant every alert was silently discarded. A wedged apport burned a core on n150-1 for three weeks and nothing said a word. - severity=critical now also routes to ntfy (phone). continue: true keeps the M5Stack in the path, so nothing is lost. max_alerts: 5 caps a storm, because a phone that buzzes forty times gets muted. - The Watchdog alert, which kube-prometheus-stack fires continuously by design, went to null - discarding exactly the signal that proves monitoring is alive. It now pings a healthchecks.io check every 5m against a 15m period, so if Prometheus, Alertmanager or n150-1 dies, healthchecks alerts from outside the lab. The only prior heartbeat was h4-core. - Endpoint URLs come from Vault via ESO and are read with url_file. The ntfy topic IS the password; it must not be in git. Scrape targets (H28): dropped decommissioned 192.168.1.70, which had kept LabExternalHostDown firing critical for three weeks. Added .116, .217, .128 (Vault - every ExternalSecret depends on it) and .64. LabVMUnexpectedShutoff filtered on domain=~ldap-1, decommissioned 2026-07-04, so the rule was structurally unable to fire. Retargeted at gitlab-1. Refs: docs/REVIEW-2026-07-24.md H25, H28
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Alertmanager had two receivers: null and a webhook to an M5Stack display on
the LAN. Asleep, out, or display off meant every alert was silently
discarded. A wedged apport burned a core on n150-1 for three weeks and
nothing said a word.
the M5Stack in the path, so nothing is lost. max_alerts: 5 caps a storm,
because a phone that buzzes forty times gets muted.
design, went to null - discarding exactly the signal that proves
monitoring is alive. It now pings a healthchecks.io check every 5m
against a 15m period, so if Prometheus, Alertmanager or n150-1 dies,
healthchecks alerts from outside the lab. The only prior heartbeat was
h4-core.
ntfy topic IS the password; it must not be in git.
Scrape targets (H28): dropped decommissioned 192.168.1.70, which had kept
LabExternalHostDown firing critical for three weeks. Added .116, .217,
.128 (Vault - every ExternalSecret depends on it) and .64.
LabVMUnexpectedShutoff filtered on domain=~ldap-1, decommissioned
2026-07-04, so the rule was structurally unable to fire. Retargeted at
gitlab-1.
Refs: docs/REVIEW-2026-07-24.md H25, H28