diff --git a/deploy/config.example.yaml b/deploy/config.example.yaml index 34cfb33..a4dc26f 100644 --- a/deploy/config.example.yaml +++ b/deploy/config.example.yaml @@ -271,9 +271,9 @@ ingest: key_file: /etc/monitor/tls/ingest.key trusted_proxies: [] max_body: 64KiB - rate_limit: # per ingest token (box.agent beats once a second) - per_second: 2 - burst: 20 + rate_limit: # per ingest token + per_second: 1 + burst: 10 mqtt: # Embedded broker. External-broker mode is configured per tenant through diff --git a/docs/features/F08-push-ingestion.md b/docs/features/F08-push-ingestion.md index 96c3b75..cade69e 100644 --- a/docs/features/F08-push-ingestion.md +++ b/docs/features/F08-push-ingestion.md @@ -91,12 +91,12 @@ Pushed timestamps are accepted from up to 24 h in the past (devices that buffer - **Billing.** Weight 2 per monitored VM (`vm.agent`, confirmed by PlusClouds on 2026-10-07, from `v0.12.1`); objects (disks, interfaces) are not billed. - **Not yet:** service states (the agent sends none in telemetry; a watch list in the agent is planned), heartbeats on `agent.vm..evt`. -## As built: box.agent (`v0.13.0`) +## As built: box.agent (`v0.13.0`, 5 s from `v0.13.1`) -- **Source.** `box.agent` (Go, on servers that run vLLM) POSTs a JSON heartbeat about once a second to `POST /ingest/v1/{check_id}` with `Authorization: Bearer mpush_…`. No NATS: it is an independent service. The token is minted for monitoring alone (not the token box.agent uses with llmocean.api). -- **Check.** `box.agent`, config `{"box_id": "", "missed_count": 5}`, unique on the server (409). The provisioner (llmocean.api) creates the device (external `{source: "greenference", type: "llmbox", id: }`) and the check with `interval_seconds: 1`, receives `push_token` once, and gives the box `-monitor-url`/`-monitor-token`. `POST /v1/checks/{id}/rotate-token` rotates it; disabling or deleting the check on deprovision stops the silence alert. The tenant's `min_check_interval_seconds` must be 1 or less. +- **Source.** `box.agent` (Go, on servers that run vLLM) POSTs a JSON heartbeat every 5 seconds to `POST /ingest/v1/{check_id}` with `Authorization: Bearer mpush_…`. No NATS: it is an independent service. The token is minted for monitoring alone (not the token box.agent uses with llmocean.api). +- **Check.** `box.agent`, config `{"box_id": "", "missed_count": 3}`, unique on the server (409). The provisioner (llmocean.api) creates the device (external `{source: "greenference", type: "llmbox", id: }`) and the check with `interval_seconds: 5`, receives `push_token` once, and gives the box `-monitor-url`/`-monitor-token`. `POST /v1/checks/{id}/rotate-token` rotates it; disabling or deleting the check on deprovision stops the silence alert. The plugin's own 5 s floor holds in every tenant (`Manifest.TenantFloorExempt`): the tenant's `min_check_interval_seconds` does not apply to `box.agent`, so no tenant setting is needed. - **Objects.** `host` (load, memory, disk, OOM kills, router connection, the box's own count of this server's answers by class), `vllm` (up, health, restarts, queue) and `gpu:` (temperature, utilisation, memory, ECC, last XID, throttle). A missing GPU array is unknown, not failed. -- **State.** XID 79, 48 or 64, or uncorrectable ECC, is CRITICAL at once. `booting`, `downloading`, `restarting` and `stopped` are never incidents. Otherwise the agent's `severity` (ok, degraded, failed) maps to OK, WARNING, CRITICAL; the server decides what happens next. Silence for `missed_count` intervals (5) is CRITICAL through the last-seen rule (swept every 5 s). +- **State.** XID 79, 48 or 64, or uncorrectable ECC, is CRITICAL at once. `booting`, `downloading`, `restarting` and `stopped` are never incidents. Otherwise the agent's `severity` (ok, degraded, failed) maps to OK, WARNING, CRITICAL; the server decides what happens next. Silence for `missed_count` intervals (3 x 5 s = 15 s) is CRITICAL through the last-seen rule (swept every 5 s). - **Counters.** `ingest_http_responses_total{code}` counts every answer of the push endpoint by status code. - **Billing.** Weight 2 per box is a placeholder until PlusClouds confirms it. - **Not yet:** the alert route to the llmocean.api webhook and the provisioning calls are llmocean.api's side. diff --git a/docs/progress.md b/docs/progress.md index 06fa7f8..b56a328 100644 --- a/docs/progress.md +++ b/docs/progress.md @@ -167,9 +167,13 @@ Suppression: a pool host that goes down suppresses its VMs' incidents and the ch Verified by `TestPool`, `TestPoolFollowsMaster`, `TestPoolFailures` (fake pool), `TestDiscoveredDevices` (engine, devices, incidents, metrics, limits, retention) and `TestDiscoveredHostSuppression`. +### v0.13.1 — box.agent every 5 s in every tenant + +`box.agent` beats every 5 s (was planned at 1 s): `DefaultInterval` and `MinInterval` are 5 s, `missed_count` defaults to 3 (silence after 15 s), and `Manifest.TenantFloorExempt` makes the 5 s floor hold in every tenant, so the tenant's 30 s or 60 s `min_check_interval_seconds` no longer blocks box provisioning. The ingest rate limit stays at 1/s. Verified by `TestBoxAgent`. + ### v0.13.0 — box.agent heartbeats -`box.agent` checks receive an LLM box's 1 s heartbeat over HTTPS with their own push token (separate from the token box.agent uses with llmocean.api): host, vllm and GPU objects, hard GPU faults as immediate CRITICAL, boot states exempt, silence as CRITICAL after 5 beats. Push answers are counted by status code (`ingest_http_responses_total`). Verified by `TestBoxAgent` (not yet run in CI at the time of writing) and `FuzzDecode`. +`box.agent` checks receive an LLM box's heartbeat (1 s planned, 5 s from v0.13.1) over HTTPS with their own push token (separate from the token box.agent uses with llmocean.api): host, vllm and GPU objects, hard GPU faults as immediate CRITICAL, boot states exempt, silence as CRITICAL after missed beats. Push answers are counted by status code (`ingest_http_responses_total`). Verified by `TestBoxAgent` (not yet run in CI at the time of writing) and `FuzzDecode`. ### v0.12.0 — PlusClouds VM agent telemetry diff --git a/internal/api/boxagent_integration_test.go b/internal/api/boxagent_integration_test.go index c28b95b..49f6c28 100644 --- a/internal/api/boxagent_integration_test.go +++ b/internal/api/boxagent_integration_test.go @@ -39,7 +39,7 @@ func boxBeat(box, state, severity string, xid float64, extra map[string]any) str // box.agent (LLM box heartbeats over HTTPS with their own push token): the // provisioner creates a device and a check per box and gets the token once; -// heartbeats become host, vllm and GPU objects; boot states are not +// heartbeats (every 5 s) become host, vllm and GPU objects; boot states are not // incidents, a hard GPU fault and silence are; a rotated token stops the // old one; every answer is counted by status code. func TestBoxAgent(t *testing.T) { @@ -48,15 +48,17 @@ func TestBoxAgent(t *testing.T) { const box = "box-7f3a" dev := x.id(x.must(x.do("POST", "/v1/devices", x.key, map[string]any{"name": box, "type": "server", "external": map[string]any{"source": "greenference", "type": "llmbox", "id": box}}), 201)) - body := map[string]any{"name": "box", "plugin": "box.agent", "is_host_check": true, "interval_seconds": 1, + body := map[string]any{"name": "box", "plugin": "box.agent", "is_host_check": true, "interval_seconds": 5, "config": map[string]any{"box_id": box}, "thresholds": []any{map[string]any{"metric": "gpu_temp_c", "object": "*", "warning": map[string]any{"op": ">", "value": 85}}}} - // The tenant's minimum interval applies: a 1 s check needs it lowered. - x.must(x.do("POST", "/v1/devices/"+dev+"/checks", x.key, body), 422) - if _, err := db.System.Exec(ctx, `UPDATE tenants SET min_check_interval_seconds = 1 WHERE id = $1`, x.tenant); err != nil { + // The plugin's own 5 s floor holds in every tenant; the tenant's minimum + // (30 s here) does not apply to box.agent. + if _, err := db.System.Exec(ctx, `UPDATE tenants SET min_check_interval_seconds = 30 WHERE id = $1`, x.tenant); err != nil { t.Fatal(err) } + faster := map[string]any{"name": "box", "plugin": "box.agent", "interval_seconds": 1, "config": map[string]any{"box_id": box}} + x.must(x.do("POST", "/v1/devices/"+dev+"/checks", x.key, faster), 422) created := x.must(x.do("POST", "/v1/devices/"+dev+"/checks", x.key, body), 201) chk := x.id(created) token, _ := created.body["push_token"].(string) @@ -69,9 +71,9 @@ func TestBoxAgent(t *testing.T) { // One check per box on the whole server; a bad box_id is refused. other := x.device("box-copy") x.must(x.do("POST", "/v1/devices/"+other+"/checks", x.key, map[string]any{"name": "box", "plugin": "box.agent", - "interval_seconds": 1, "config": map[string]any{"box_id": strings.ToUpper(box)}}), 409) + "interval_seconds": 5, "config": map[string]any{"box_id": strings.ToUpper(box)}}), 409) x.must(x.do("POST", "/v1/devices/"+other+"/checks", x.key, map[string]any{"name": "box", "plugin": "box.agent", - "interval_seconds": 1, "config": map[string]any{"box_id": "bad id"}}), 422) + "interval_seconds": 5, "config": map[string]any{"box_id": "bad id"}}), 422) results := make(chan runner.Result, 100) reg := prometheus.NewRegistry() @@ -196,7 +198,7 @@ func TestBoxAgent(t *testing.T) { t.Errorf("after recovery: %v", st) } - // Silence: 5 intervals without a beat is CRITICAL, once per interval. + // Silence: 3 intervals without a beat is CRITICAL, once per interval. if _, err := db.System.Exec(ctx, `UPDATE push_sources SET last_push_at = now() - interval '1 minute', created_at = now() - interval '1 hour' WHERE check_id = $1`, chk); err != nil { t.Fatal(err) @@ -205,7 +207,7 @@ func TestBoxAgent(t *testing.T) { t.Fatalf("sweep: %d %v", n, err) } apply() - if st := state(); st["phase"] != "PROBLEM" || !strings.Contains(st["last_output"].(string), "5 intervals") { + if st := state(); st["phase"] != "PROBLEM" || !strings.Contains(st["last_output"].(string), "3 intervals") { t.Errorf("silent: %v", st) } if code := post(chk, token, boxBeat(box, "ready", "ok", 0, nil)); code != 202 { diff --git a/internal/config/defaults.go b/internal/config/defaults.go index 77c3200..72184d3 100644 --- a/internal/config/defaults.go +++ b/internal/config/defaults.go @@ -101,7 +101,7 @@ func Default() Config { Enabled: true, Listen: ":9443", MaxBody: 64 << 10, - RateLimit: Rate{PerSecond: 2, Burst: 20}, // box.agent beats once a second + RateLimit: Rate{PerSecond: 1, Burst: 10}, }, MQTT: IngestMQTT{ Enabled: true, diff --git a/internal/ingest/ingest.go b/internal/ingest/ingest.go index c8b2600..de50e3f 100644 --- a/internal/ingest/ingest.go +++ b/internal/ingest/ingest.go @@ -353,14 +353,11 @@ func (s *Server) counted(h http.HandlerFunc) http.HandlerFunc { // missedCount is a push check's missed_count (push.http and push.mqtt // share the field), as the sweep query reads it. -func missedCount(kind string, cfg json.RawMessage) int { +func missedCount(cfg json.RawMessage) int { var c struct { MissedCount int `json:"missed_count"` } if json.Unmarshal(cfg, &c) != nil || c.MissedCount < 1 { - if kind == "box.agent" { - return 5 - } return 3 } return c.MissedCount @@ -376,7 +373,7 @@ func (s *Server) Sweep(ctx context.Context) (int, error) { FROM checks c, tenants t WHERE c.id = ps.check_id AND t.id = c.tenant_id AND c.enabled AND t.status = 'active' AND greatest(ps.last_push_at, ps.created_at) < - now() - make_interval(secs => c.interval_seconds * coalesce((c.config->>'missed_count')::int, CASE c.plugin WHEN 'box.agent' THEN 5 ELSE 3 END)) + now() - make_interval(secs => c.interval_seconds * coalesce((c.config->>'missed_count')::int, 3)) AND (ps.stale_reported_at IS NULL OR ps.stale_reported_at < now() - make_interval(secs => c.interval_seconds)) RETURNING ps.check_id, c.tenant_id, c.device_id, c.plugin, c.config, c.interval_seconds, ps.last_push_at`) if err != nil { @@ -403,7 +400,7 @@ func (s *Server) Sweep(ctx context.Context) (int, error) { if p, ok := plugin.Lookup(x.plugin); ok { layout = plugin.MetricsOf(p, x.config) } - missed := missedCount(x.plugin, x.config) + missed := missedCount(x.config) out := "no data received yet" if x.last != nil { out = "no data since " + x.last.UTC().Format(time.RFC3339) + " (" + strconv.Itoa(missed) + " intervals of " + diff --git a/internal/inventory/checks.go b/internal/inventory/checks.go index 159bd35..2b73ecb 100644 --- a/internal/inventory/checks.go +++ b/internal/inventory/checks.go @@ -93,9 +93,9 @@ func prepare(ctx context.Context, tx pgx.Tx, in CheckInput, lim Limits) (CheckIn return in, errs.Invalidf("config", "%s", err.Error()) } if in.IntervalSeconds == 0 { - in.IntervalSeconds = max(int(m.DefaultInterval/time.Second), lim.MinCheckIntervalSeconds) + in.IntervalSeconds = max(int(m.DefaultInterval/time.Second), tenantFloor(m, lim)) } - minInterval := max(int(m.MinInterval/time.Second), lim.MinCheckIntervalSeconds) + minInterval := max(int(m.MinInterval/time.Second), tenantFloor(m, lim)) if in.IntervalSeconds < minInterval { return in, errs.Invalidf("interval_seconds", "must be at least %d for %s in this tenant", minInterval, m.Type) } @@ -166,6 +166,15 @@ const checkCols = `c.id, c.tenant_id, c.device_id, c.name, c.plugin, c.config, c (SELECT jsonb_build_object('token_prefix', ps.token_prefix, 'token_created_at', ps.token_created_at, 'last_push_at', ps.last_push_at) FROM push_sources ps WHERE ps.check_id = c.id)` +// tenantFloor is the tenant's minimum check interval that applies to a plugin: +// none for a plugin that is exempt (its own MinInterval is the floor). +func tenantFloor(m plugin.Manifest, lim Limits) int { + if m.TenantFloorExempt { + return 0 + } + return lim.MinCheckIntervalSeconds +} + var checkConstraints = map[string]string{ "checks_device_id_name_key": "the device already has a check with this name", "checks_one_host_check": "the device already has a host check", diff --git a/pkg/plugin/plugin.go b/pkg/plugin/plugin.go index fc1829e..2bf6fa8 100644 --- a/pkg/plugin/plugin.go +++ b/pkg/plugin/plugin.go @@ -87,11 +87,16 @@ type Manifest struct { Metrics []MetricDef `json:"metrics"` // fixed layout; Result.Metrics aligns with it DefaultInterval time.Duration `json:"-"` MinInterval time.Duration `json:"-"` - NeedsRawSocket bool `json:"needs_raw_socket"` - MaxConcurrency int `json:"-"` // per runner; 0 = runner default - PerTargetLimit int `json:"-"` // concurrent runs against one address; 0 = runner default - Slow bool `json:"-"` // runs in the slow pool (F04) - BillingClass string `json:"billing_class"` + // TenantFloorExempt: the plugin's own MinInterval is the floor in every + // tenant, not the tenant's min_check_interval_seconds. For ingesters whose + // agents beat at a fixed short rate (box.agent) and whose interval only + // sets the silence window. + TenantFloorExempt bool `json:"-"` + NeedsRawSocket bool `json:"needs_raw_socket"` + MaxConcurrency int `json:"-"` // per runner; 0 = runner default + PerTargetLimit int `json:"-"` // concurrent runs against one address; 0 = runner default + Slow bool `json:"-"` // runs in the slow pool (F04) + BillingClass string `json:"billing_class"` // WhoopsyMetric is the metric Whoopsy! (band alerting) watches unless // the check names another: response time for http, round-trip time // for icmp. Empty: the check must name one. diff --git a/plugins/boxagent/boxagent.go b/plugins/boxagent/boxagent.go index 1b2a0fe..e926ca6 100644 --- a/plugins/boxagent/boxagent.go +++ b/plugins/boxagent/boxagent.go @@ -1,6 +1,6 @@ // Package boxagent implements box.agent: the heartbeat the box.agent service // (a WebSocket proxy in front of vLLM on an LLM host) POSTs to the ingest -// listener about once a second, with a token minted for monitoring alone +// listener every 5 seconds, with a token minted for monitoring alone // (push token, F08). A box is one check whose objects are the host, the // vLLM backend and each GPU, so each has its own thresholds, state and graphs. // @@ -80,18 +80,18 @@ var defs = []plugin.MetricDef{ } func g(name, unit, desc string) plugin.MetricDef { - // The 1 s stream is for liveness and fast thresholds; the standard - // class keeps its storage and downsampling in line with other plugins. + // A 5 s stream for liveness and thresholds; the standard class keeps its + // storage and downsampling in line with other plugins. return plugin.MetricDef{Name: name, Unit: unit, Description: desc, Kind: "gauge", RetentionClass: plugin.RetentionStandard} } // Config of a box.agent check: which box's heartbeats it receives. type Config struct { BoxID string `json:"box_id" jsonschema:"required,maxLength=128,description=The box's ID (the agent instance ID llmocean.api uses). Unique on the server; every heartbeat must carry it"` - MissedCount int `json:"missed_count,omitempty" jsonschema:"minimum=1,maximum=100,default=5,description=CRITICAL after this many intervals without a heartbeat"` + MissedCount int `json:"missed_count,omitempty" jsonschema:"minimum=1,maximum=100,default=3,description=CRITICAL after this many intervals without a heartbeat"` } -var defaults = Config{MissedCount: 5} +var defaults = Config{MissedCount: 3} var boxID = regexp.MustCompile(`^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$`) @@ -102,14 +102,15 @@ func (*Agent) Manifest() plugin.Manifest { return plugin.Manifest{ Type: "box.agent", Kind: plugin.KindIngester, - Description: "An LLM box's own heartbeat from box.agent (POST to the ingest URL with the check's token, about every second): " + + Description: "An LLM box's own heartbeat from box.agent (POST to the ingest URL with the check's token, every 5 seconds): " + "host, vLLM backend and each GPU as their own objects. CRITICAL when no heartbeat arrives for missed_count intervals, " + "or on a hard GPU fault (XID 79, uncorrectable ECC). Boot, model download and vLLM restarts are not incidents.", - ConfigSchema: plugin.SchemaFor[Config](), - DefaultInterval: time.Second, - MinInterval: time.Second, - BillingClass: plugin.BillingStandard, - Objects: true, + ConfigSchema: plugin.SchemaFor[Config](), + DefaultInterval: 5 * time.Second, + MinInterval: 5 * time.Second, + TenantFloorExempt: true, // 5 s in every tenant: the agent beats at this rate + BillingClass: plugin.BillingStandard, + Objects: true, } } diff --git a/plugins/boxagent/boxagent_test.go b/plugins/boxagent/boxagent_test.go index cde6c7d..d6ae88d 100644 --- a/plugins/boxagent/boxagent_test.go +++ b/plugins/boxagent/boxagent_test.go @@ -140,7 +140,7 @@ func TestValidate(t *testing.T) { } func TestMissedCount(t *testing.T) { - if MissedCount(cfg()) != 5 || MissedCount(json.RawMessage(`{"box_id":"x","missed_count":8}`)) != 8 { + if MissedCount(cfg()) != 3 || MissedCount(json.RawMessage(`{"box_id":"x","missed_count":8}`)) != 8 { t.Error("missed_count") } } diff --git a/scripts/box-monitor-token.sh b/scripts/box-monitor-token.sh new file mode 100755 index 0000000..d7f4f45 --- /dev/null +++ b/scripts/box-monitor-token.sh @@ -0,0 +1,157 @@ +#!/usr/bin/env bash +# Creates, rotates or removes the monitoring token of one box.agent box, through +# the PlusClouds API (api.plusclouds.com/monitoring), so the caller needs a +# PlusClouds API token and no monitoring.server key. +# +# create Makes the host (external type "llmbox", external id = the box ID) and +# its box.agent check, then prints the two settings the box needs: +# MONITOR_URL=https://monitoring.plusclouds.com/ingest/v1/ +# MONITOR_TOKEN=mpush_... +# The token is shown once; only these two lines go to stdout, so +# `eval "$(... create BOX_ID)"` or `>> box.env` works. Everything else goes to stderr. +# Re-running for a box that already has a check refuses; use rotate. +# rotate Replaces the token (the old one stops working at once), prints the same two lines. +# delete Removes the check and the host (use when the box is deprovisioned; +# otherwise its silence raises a CRITICAL incident). +# +# Usage: +# PLUSCLOUDS_TOKEN=... scripts/box-monitor-token.sh create BOX_ID +# PLUSCLOUDS_TOKEN=... scripts/box-monitor-token.sh rotate BOX_ID +# PLUSCLOUDS_TOKEN=... scripts/box-monitor-token.sh delete BOX_ID +# +# Environment: +# PLUSCLOUDS_TOKEN PlusClouds API token (required). Never printed. +# API_URL default https://api.plusclouds.com +# INTERVAL check interval in seconds, default 5 (the box.agent floor). If a server +# older than v0.13.1 enforces a higher tenant minimum, the script uses +# that minimum and says so (silence is detected after MISSED_COUNT x interval). +# MISSED_COUNT silent intervals before CRITICAL, default 3 +# +# Needs: bash, curl, jq. + +set -euo pipefail + +API_URL="${API_URL:-https://api.plusclouds.com}" +INTERVAL="${INTERVAL:-5}" +MISSED_COUNT="${MISSED_COUNT:-3}" +BASE="${API_URL%/}/monitoring" + +die() { echo "error: $*" >&2; exit 1; } +log() { echo "$*" >&2; } + +usage() { sed -n '2,29p' "$0" | sed 's/^# \{0,1\}//' >&2; exit "${1:-1}"; } + +[ $# -eq 2 ] || [ "${1:-}" = "-h" ] || [ "${1:-}" = "--help" ] || usage 1 +case "${1:-}" in -h|--help) usage 0 ;; esac +CMD="$1" +BOX_ID="$2" + +[ -n "${PLUSCLOUDS_TOKEN:-}" ] || die "set PLUSCLOUDS_TOKEN to a PlusClouds API token" +command -v curl >/dev/null || die "curl is required" +command -v jq >/dev/null || die "jq is required" +[[ "$BOX_ID" =~ ^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$ ]] || die "BOX_ID: letters, digits, . _ -, at most 128 characters" +[[ "$INTERVAL" =~ ^[0-9]+$ ]] && [ "$INTERVAL" -ge 1 ] || die "INTERVAL must be a whole number of seconds, at least 1" + +# api METHOD PATH [JSON] sets STATUS and BODY. The token goes through a curl +# config on stdin, so it never appears in the process list. +api() { + local method="$1" path="$2" data="${3:-}" out + local args=(-sS -m 30 -w $'\n%{http_code}' -X "$method" -K - -H "Accept: application/json") + if [ -n "$data" ]; then args+=(-H "Content-Type: application/json" --data "$data"); fi + out="$(printf 'header = "Authorization: Bearer %s"\n' "$PLUSCLOUDS_TOKEN" | curl "${args[@]}" "$BASE$path")" \ + || die "cannot reach $BASE$path" + STATUS="${out##*$'\n'}" + BODY="${out%$'\n'*}" +} + +fail() { # fail WHAT + local msg + msg="$(printf '%s' "$BODY" | jq -r '.error.message // .message // empty' 2>/dev/null || true)" + die "$1: HTTP $STATUS${msg:+ - $msg}" +} + +find_host() { # prints the host id for BOX_ID, or nothing + api GET "/hosts?type=server" + [ "$STATUS" = 200 ] || fail "list hosts" + printf '%s' "$BODY" | jq -r --arg id "$BOX_ID" \ + '(.data | if type == "array" then . else (.items // []) end)[] | select(.external_id == $id) | .id' | head -n1 +} + +find_check() { # find_check HOST_ID prints the box.agent check id, or nothing + api GET "/hosts/$1/checks" + [ "$STATUS" = 200 ] || fail "list checks" + printf '%s' "$BODY" | jq -r \ + '(.data | if type == "array" then . else (.items // []) end)[] | select(.plugin == "box.agent") | .id' | head -n1 +} + +print_settings() { # print_settings URL TOKEN + printf 'MONITOR_URL=%s\nMONITOR_TOKEN=%s\n' "$1" "$2" +} + +create_check() { # create_check HOST_ID [INTERVAL] sets STATUS and BODY + local payload + payload="$(jq -nc --arg box "$BOX_ID" --argjson interval "$2" --argjson missed "$MISSED_COUNT" '{ + name: "box", plugin: "box.agent", is_host_check: true, interval_seconds: $interval, + config: {box_id: $box, missed_count: $missed}, + thresholds: [{metric: "gpu_temp_c", object: "*", warning: {op: ">", value: 85}}]}')" + api POST "/hosts/$1/checks" "$payload" +} + +case "$CMD" in +create) + host="$(find_host)" + if [ -z "$host" ]; then + payload="$(jq -nc --arg id "$BOX_ID" '{name: $id, type: "server", external_id: $id, external_type: "llmbox"}')" + api POST "/hosts" "$payload" + [ "$STATUS" = 201 ] || fail "create host" + host="$(printf '%s' "$BODY" | jq -r '.data.id')" + log "created host $host" + else + log "host $host already exists" + [ -z "$(find_check "$host")" ] || die "box $BOX_ID already has a box.agent check; use rotate to get a new token" + fi + + interval="$INTERVAL" + create_check "$host" "$interval" + if [ "$STATUS" = 422 ]; then # the tenant's minimum interval is higher + min="$(printf '%s' "$BODY" | jq -r '.error.message // ""' | sed -n 's/.*must be at least \([0-9][0-9]*\).*/\1/p')" + if [ -n "$min" ] && [ "$min" -gt "$interval" ]; then + log "warning: this tenant allows checks no faster than ${min}s; using ${min}s (silence is detected after $((MISSED_COUNT * min))s)." + interval="$min" + create_check "$host" "$interval" + fi + fi + [ "$STATUS" = 201 ] || fail "create check" + url="$(printf '%s' "$BODY" | jq -r '.data.push.ingest_url // empty')" + token="$(printf '%s' "$BODY" | jq -r '.data.push_token // empty')" + [ -n "$url" ] && [ -n "$token" ] || die "the API returned no ingest URL or token" + log "created box.agent check $(printf '%s' "$BODY" | jq -r '.data.id') (interval ${interval}s)" + print_settings "$url" "$token" + ;; +rotate) + host="$(find_host)" + [ -n "$host" ] || die "no host for box $BOX_ID" + check="$(find_check "$host")" + [ -n "$check" ] || die "box $BOX_ID has no box.agent check; use create" + api POST "/checks/$check/rotate-token" + [ "$STATUS" = 200 ] || fail "rotate token" + token="$(printf '%s' "$BODY" | jq -r '.data.push_token // empty')" + url="$(printf '%s' "$BODY" | jq -r '.data.push.ingest_url // empty')" + [ -n "$token" ] || die "the API returned no token" + log "rotated: the old token stopped working; deploy the new one to the box" + print_settings "$url" "$token" + ;; +delete) + host="$(find_host)" + [ -n "$host" ] || die "no host for box $BOX_ID" + check="$(find_check "$host")" + if [ -n "$check" ]; then + api DELETE "/checks/$check" + [ "$STATUS" = 204 ] || [ "$STATUS" = 200 ] || fail "delete check" + fi + api DELETE "/hosts/$host" + [ "$STATUS" = 204 ] || [ "$STATUS" = 200 ] || fail "delete host" + log "deleted the check and host of box $BOX_ID" + ;; +*) usage 1 ;; +esac