From 18838868975f49d729ae31adaae6786326f226dd Mon Sep 17 00:00:00 2001 From: Sandipan Das Date: Sun, 20 Sep 2026 20:54:56 -0400 Subject: [PATCH 1/2] feat: real container restart remediation via argus-executor - Add dedicated argus-executor service that owns /var/run/docker.sock - execute_runbook_step no longer accepts a command string; it looks up typed runbook steps and routes restart through the executor - Enforce per-incident remediation attempt cap (default 3, env configurable) - Keep /recover endpoints for manual/demo reset but remove from agent path - Refuse execution in native modes where no executor is configured - Document all seven demo scenarios including the unprimed api-oom case - Add remediation-cap test and update approval-gate test for new tool signature --- ARCHITECTURE.md | 49 +++-- README.md | 41 ++++- argus-executor/Dockerfile | 22 +++ argus-executor/main.ts | 125 +++++++++++++ argus-executor/package.json | 6 + argus-server/agent/system-prompt.ts | 2 +- argus-server/db/api.ts | 10 ++ argus-server/db/schema.ts | 7 + argus-server/db/types.ts | 1 + argus-server/server.ts | 31 +++- argus-server/tests/approval-gate.test.ts | 9 +- argus-server/tests/remediation-cap.test.ts | 102 +++++++++++ argus-server/tools/argus-tools.ts | 198 ++++++++++++++------- docker-compose.yml | 23 +++ runbooks/services/app-api.yaml | 2 +- runbooks/services/app-ui.yaml | 2 +- runbooks/services/auth.yaml | 2 +- runbooks/services/gateway.yaml | 2 +- runbooks/services/pgsql.yaml | 2 +- runbooks/services/redis.yaml | 13 +- 20 files changed, 542 insertions(+), 107 deletions(-) create mode 100644 argus-executor/Dockerfile create mode 100644 argus-executor/main.ts create mode 100644 argus-executor/package.json create mode 100644 argus-server/tests/remediation-cap.test.ts diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index dd99cd5..fd3cb66 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -5,11 +5,10 @@ demo microservices stack, and when services degrade it investigates, opens an in proposes a fix, waits for a human to approve, executes the remediation, and writes a root-cause analysis, mostly on its own. -It is a **demo / reference system, not a product**. The failure scenarios are scripted, the -runbook "commands" hit mock `/recover` endpoints, and a single sequential agent session -won't scale to a real org's alert volume. What it _is_ meant to be is an honest, working -illustration of how to wire an LLM agent into an operational control loop, and more specifically -of **where such a system needs determinism and where it needs judgment.** +It is a **demo / reference system, not a product**. The failure scenarios are scripted, and a +single sequential agent session won't scale to a real org's alert volume. What it _is_ meant to +be is an honest, working illustration of how to wire an LLM agent into an operational control +loop, and more specifically of **where such a system needs determinism and where it needs judgment.** That boundary is the thesis of this document. @@ -35,6 +34,10 @@ graph TB db[(SQLite)] end + subgraph executor["argus-executor — Docker only"] + ex[POST /execute] + end + ui[argus-ui — React/Vite, :5173] demo -- GET /metrics --> scr @@ -44,7 +47,8 @@ graph TB rec -- reads desired state --> db rec -- enqueues tasks --> ag ag -- tool calls --> db - ag -- POST /recover --> demo + ag -- POST /execute --> ex + ex -- docker restart --> demo rest --> db ui -- poll /api/workflows, /api/chat --> rest rest -- WS broadcast (tagged by workflowId) --> ui @@ -54,6 +58,7 @@ graph TB |---|---|---|---| | **argus-server** | Bun | 3000 | Headless agent, reconciler, REST API, alert scraper, WebSocket | | **argus-ui** | React + Vite | 5173 | Display-only triage console (pipeline, incidents, chat) | +| **argus-executor** | Bun + Docker CLI | 3001 | Dedicated remediation worker: the only container with Docker socket access | | **demo-services** | Bun | 8081–8086 | Six mock services with `GET /metrics`, `POST /fail`, `POST /recover` | | **infra** | Docker Compose | — | VictoriaMetrics + vmalert + Alertmanager (optional alert source) | | **runbooks/** | YAML | — | Per-service triggers, steps, known failure patterns | @@ -226,8 +231,10 @@ without a human approval on record.** The flow: row to `approved`. That's all the endpoint does; the approval is now durable. 3. The reconciler sees `approved + not mitigated + alerts firing` and enqueues execution. 4. The agent calls `execute_runbook_step`, which **re-checks the gate** (refuses unless the - confirmation is `approved`), then performs the real recovery: `POST /recover` on the - affected services, resolves their alerts, marks the incident mitigated. + confirmation is `approved`), looks up the typed runbook step (`action: executable`, `target`), + validates the target against the executor allowlist, and routes the real container restart + through `argus-executor`. It polls the service until metrics verify healthy, then marks the + incident `mitigated`. Alert resolution is left to the detection path. The gate is enforced in two independent places: the UI flow and the tool itself. The agent cannot talk itself past it. @@ -326,15 +333,21 @@ The line between deterministic code and the LLM is the thing to take from this a | Deciding *when* to act (reconciler) | Diagnosing the root cause | | Computing workflow / pipeline state | Proposing a remediation step | | Enforcing the approval gate | Writing the incident + RCA narrative | -| Performing the DB writes & `/recover` call | Asking the operator the right question | +| Performing the typed container restart via `argus-executor` | Asking the operator the right question | | Persisting transcript & aliasing threads | Judgment under ambiguity | | Shipping & indexing logs (future) | Reading a log line and deciding what it means | -Detection, scheduling, state, and safety are **mechanism**: they must be predictable, so they -are code. Correlation, diagnosis, and explanation are **judgment**: they benefit from a model. -Keeping the model out of the control loop's mechanics is what makes an autonomous agent safe to -run unattended. Not everything splits that cleanly; the last row in the table straddles the -boundary, and that's where the interesting design questions live. +The remediation row is the important one for the safety argument: the model names a +pre-approved runbook step, but deterministic code decides what that step maps to, validates +the target against an allowlist, and routes the real container restart through a dedicated +executor that owns the Docker socket. The model never supplies a raw command string, and the +executor validates independently even of `argus-server`. + +The three primed scenarios (`db`, `cache`, `auth`) exercise the control loop against +runbooks that pre-map symptoms to root causes. The `api-oom` scenario is different: +`app-api` fails directly, with no upstream cause and no matching `known_patterns` entry, +so the agent must reason from live metrics and runbook steps without the shortcut. +That is the unprimed case, and it is what tests the judgment side of the boundary. ### 7.1 Instrumenting the agentic side @@ -400,13 +413,19 @@ All access goes through `argus-server/db/api.ts`; tools and handlers never embed - Failure scenarios are **scripted** (`POST /fail` / `POST /recover`); this is not chaos engineering. -- Remediation is **simulated** against mock endpoints. No real infrastructure is touched. +- The full **container restart** remediation path only runs under `docker compose up` / + `mise run infra:up`, where an `argus-executor` container owns the Docker socket. In native + modes (`mise run dev`, `mise run infra:start`) there are no containers to restart, so + `execute_runbook_step` refuses rather than simulating. - A single session won't handle **production alert volume**; this is an architecture demo. - Autonomy **requires the model to be reachable** (Ollama in dev); if it's down, detection and state still work, but triage pauses (the reconciler no-ops until the agent is up). - Narrow restart edge: if the server restarts in the seconds between an investigation finishing and the incident being aliased, that one transcript can stay under the pending thread. +The `POST /recover` endpoints remain on the demo services for manual resets and the `recover` +demo scenario; the agent no longer calls them. + --- ## 11. Repository layout diff --git a/README.md b/README.md index c1d02db..d52387f 100644 --- a/README.md +++ b/README.md @@ -82,11 +82,13 @@ mise run dev # server + UI + services, with bun --watch hot-reload ## Demo scenarios ```bash -mise run infra:demo -- db # database cascade -mise run infra:demo -- cache # redis eviction storm -mise run infra:demo -- auth # auth outage -mise run infra:demo -- all # everything fails -mise run infra:demo -- recover # restore all services +mise run infra:demo -- db # database cascade +mise run infra:demo -- cache # redis eviction storm +mise run infra:demo -- auth # auth outage +mise run infra:demo -- frontend # app-ui failure +mise run infra:demo -- api-oom # app-api crash, no runbook match +mise run infra:demo -- all # everything fails +mise run infra:demo -- recover # restore all services ``` ## URLs @@ -119,6 +121,12 @@ Each service exposes `GET /metrics` (Prometheus format), `POST /fail`, and `POST 2. **Auth outage:** auth fails → token validation errors → app-api 401s → gateway errors. Root cause: auth 3. **Cache eviction storm:** redis memory pressure → cache misses → app-api latency → pgsql pool exhaustion. Root cause: redis +## The unprimed scenario (`api-oom`) + +`api-oom` fails the `app-api` process directly. The agent can read live metrics, check that `auth`, `pgsql`, and `redis` are healthy, and follow the `app-api` runbook steps, but the runbook's `known_patterns` all point to upstream dependencies. There is no named pattern for a direct app-api failure, so the agent has to reason without being handed the answer. Watch whether it proposes a sensible restart anyway, or whether it gets stuck looking for an upstream cause that does not exist. + +This is the scenario that tests the judgment side of the determinism boundary, not just the control loop. + ## Prerequisites - [mise](https://mise.jdx.dev) — **required**: it runs every `mise run …` command, pins the @@ -172,8 +180,10 @@ Each monitored service has a YAML runbook in [`runbooks/services/`](runbooks/ser the operational knowledge an on-call engineer would reach for. A runbook declares: - **`triggers`** — the metrics and thresholds that signal trouble (e.g. `pgsql_connection_pool_used > 0.90`). -- **`steps`** — ordered remediation steps, each typed `manual`, `executable` (with a `command`, - `requires_confirmation`, and a `rollback`), or `notify` (escalate to a channel). +- **`steps`** — ordered remediation steps, each typed `manual`, `executable` (with a `target` + container, `requires_confirmation`, and a `rollback`), or `notify` (escalate to a channel). + The agent names a pre-approved step; deterministic code maps that to a typed action and + runs it through the executor. The model never supplies a raw command string. - **`known_patterns`** — named failure patterns that map a cluster of symptoms to a **root cause**. This is what lets the agent resolve a multi-service cascade to the one service actually at fault. - **`related_runbooks`** — links to dependent services' runbooks. @@ -218,7 +228,7 @@ These runbooks are curated, human-authored knowledge the agent consults. The mod | `get_recent_deployments` | Check recent deployments | | `notify_channel` | Send notification to a Slack channel | | `request_confirmation` | Request human approval before remediation | -| `execute_runbook_step` | Execute a remediation step after confirmation | +| `execute_runbook_step` | Execute an approved, typed runbook step via the executor after verification | ## All mise tasks @@ -237,4 +247,17 @@ These runbooks are curated, human-authored knowledge the agent consults. The mod > `infra:up`/`infra:down` wrap `docker compose` on the top-level `docker-compose.yml`. > The infra-only observability stack (for use with native services) remains -> `infra/docker-compose.yml`, driven by `mise run infra:start`/`infra:stop`. \ No newline at end of file +> `infra/docker-compose.yml`, driven by `mise run infra:start`/`infra:stop`. + +## Remediation and the executor + +In the full Docker topology (`docker compose up` / `mise run infra:up`), approved +remediation runs through a dedicated `argus-executor` container. The executor is the only +component with access to `/var/run/docker.sock`; it accepts a typed `{ action, target }` +request, validates the target against an allowlist of the six demo services, and restarts the +real container. The agent never supplies a raw command string. + +In native modes (`mise run dev` and `mise run infra:start`) there are no containers to +restart, so `execute_runbook_step` refuses with a clear message rather than pretending. The +`/recover` endpoints on the demo services still work for manual resets and the `recover` +demo scenario. \ No newline at end of file diff --git a/argus-executor/Dockerfile b/argus-executor/Dockerfile new file mode 100644 index 0000000..f902381 --- /dev/null +++ b/argus-executor/Dockerfile @@ -0,0 +1,22 @@ +# argus-executor — the only component that touches /var/run/docker.sock. +# It exposes POST /execute and runs only pre-approved actions against an allowlist. +# +# Build context is the argus-executor/ directory: +# docker build -f argus-executor/Dockerfile ./argus-executor + +# Copy a static Docker CLI binary so the Bun image does not need Docker-in-Docker. +FROM docker:27-cli AS docker + +FROM oven/bun:1.3.6 + +COPY --from=docker /usr/local/bin/docker /usr/local/bin/docker + +WORKDIR /app/argus-executor +COPY package.json ./ +RUN bun install + +COPY . ./ + +EXPOSE 3001 +ENV EXECUTOR_PORT=3001 +CMD ["bun", "run", "main.ts"] diff --git a/argus-executor/main.ts b/argus-executor/main.ts new file mode 100644 index 0000000..90ecf65 --- /dev/null +++ b/argus-executor/main.ts @@ -0,0 +1,125 @@ +// Argus executor — tiny, hard-coded remediation worker. +// +// This service is the ONLY component that touches /var/run/docker.sock. It exposes a +// single POST /execute endpoint that accepts a typed { action, target } and runs only +// pre-approved actions against an allowlist of demo service containers. It validates on +// its own; it does not trust argus-server to decide what is safe. + +const PORT = Number.parseInt(process.env.EXECUTOR_PORT || "3001", 10); +const DEFAULT_ALLOWLIST = ["gateway", "app-ui", "app-api", "auth", "pgsql", "redis"]; +const ALLOWLIST = process.env.EXECUTOR_ALLOWLIST + ? process.env.EXECUTOR_ALLOWLIST.split(",").map(s => s.trim()).filter(Boolean) + : DEFAULT_ALLOWLIST; +const ALLOWSET = new Set(ALLOWLIST); + +const DENYLIST = new Set(["argus-server", "argus-executor", "victoriametrics", "vmalert", "alertmanager", "mock-slack"]); + +const ACTIONS = new Set(["restart"]); + +type LogLevel = "info" | "warn" | "error"; +function log(level: LogLevel, msg: string, extra?: Record) { + const line = { ts: new Date().toISOString(), level, msg, ...extra }; + console[level === "error" ? "error" : level === "warn" ? "warn" : "log"](JSON.stringify(line)); +} + +interface ExecuteRequest { + action: string; + target: string; +} + +function isExecuteRequest(body: unknown): body is ExecuteRequest { + if (typeof body !== "object" || body === null) return false; + const b = body as Record; + return typeof b.action === "string" && typeof b.target === "string"; +} + +function jsonResponse(body: unknown, status = 200) { + return new Response(JSON.stringify(body, null, 2), { + status, + headers: { "Content-Type": "application/json" }, + }); +} + +async function runDocker(args: string[], timeoutMs = 60000): Promise<{ ok: boolean; stdout: string; stderr: string }> { + const proc = Bun.spawn(["docker", ...args], { + stdout: "pipe", + stderr: "pipe", + }); + const timeout = setTimeout(() => proc.kill("SIGTERM"), timeoutMs); + try { + const [stdoutBuf, stderrBuf] = await Promise.all([new Response(proc.stdout).arrayBuffer(), new Response(proc.stderr).arrayBuffer()]); + await proc.exited; + return { + ok: proc.exitCode === 0, + stdout: new TextDecoder().decode(stdoutBuf).trim(), + stderr: new TextDecoder().decode(stderrBuf).trim(), + }; + } finally { + clearTimeout(timeout); + } +} + +async function handleExecute(body: ExecuteRequest): Promise { + const { action, target } = body; + + if (!ACTIONS.has(action)) { + log("warn", "rejected action", { action, target, reason: "action_not_allowed" }); + return jsonResponse({ error: `Action "${action}" is not allowed. Allowed actions: ${[...ACTIONS].join(", ")}` }, 400); + } + + if (DENYLIST.has(target)) { + log("warn", "rejected denylisted target", { action, target, reason: "target_denied" }); + return jsonResponse({ error: `Target "${target}" is explicitly denied.` }, 403); + } + + if (!ALLOWSET.has(target)) { + log("warn", "rejected target", { action, target, reason: "target_not_in_allowlist", allowlist: ALLOWLIST }); + return jsonResponse({ error: `Target "${target}" is not in the executor allowlist.` }, 403); + } + + log("info", "executing remediation", { action, target }); + + if (action === "restart") { + const { ok, stdout, stderr } = await runDocker(["restart", target]); + if (!ok) { + log("error", "docker restart failed", { target, stderr, stdout }); + return jsonResponse({ error: `docker restart ${target} failed`, details: stderr || stdout }, 502); + } + log("info", "docker restart succeeded", { target, stdout }); + return jsonResponse({ action, target, status: "restarted", output: stdout }); + } + + // Unreachable — action was validated above. + return jsonResponse({ error: "unexpected action" }, 500); +} + +const server = Bun.serve({ + port: PORT, + hostname: "0.0.0.0", + async fetch(req) { + const url = new URL(req.url); + + if (req.method === "GET" && url.pathname === "/health") { + return jsonResponse({ status: "ok", allowlist: ALLOWLIST }); + } + + if (req.method !== "POST" || url.pathname !== "/execute") { + return jsonResponse({ error: "Not found" }, 404); + } + + let body: unknown; + try { + body = await req.json(); + } catch { + return jsonResponse({ error: "Invalid JSON" }, 400); + } + + if (!isExecuteRequest(body)) { + return jsonResponse({ error: "Request must be { action: string, target: string }" }, 400); + } + + return handleExecute(body); + }, +}); + +log("info", "argus-executor listening", { port: PORT, allowlist: ALLOWLIST }); diff --git a/argus-executor/package.json b/argus-executor/package.json new file mode 100644 index 0000000..160696a --- /dev/null +++ b/argus-executor/package.json @@ -0,0 +1,6 @@ +{ + "name": "argus-executor", + "private": true, + "version": "0.1.0", + "type": "module" +} diff --git a/argus-server/agent/system-prompt.ts b/argus-server/agent/system-prompt.ts index d1aa8fa..3cfebfe 100644 --- a/argus-server/agent/system-prompt.ts +++ b/argus-server/agent/system-prompt.ts @@ -33,7 +33,7 @@ When an alert fires you ALWAYS follow this sequence: 6. create_incident — record your findings with likely_cause 7. notify_channel — structured summary with root cause theory 8. request_confirmation — propose the remediation step and STOP. The operator will approve in the UI. -9. execute_runbook_step — call this to perform the remediation ONLY after the operator has approved. When you are told the approval was granted (you'll be handed the incident_id and confirmation_id), call execute_runbook_step with those plus the step_id, service, and command you proposed. The tool enforces the approval gate and performs the real recovery. After request_confirmation but before approval, end your turn with a brief summary of what you proposed and wait. +9. execute_runbook_step — call this to perform the remediation ONLY after the operator has approved. When you are told the approval was granted (you'll be handed the incident_id and confirmation_id), call execute_runbook_step with those plus the service and step_id you proposed. The tool looks up the actual command in the runbook by deterministic code, enforces the approval gate, restarts the real container through the executor, and verifies the service returns to healthy. After request_confirmation but before approval, end your turn with a brief summary of what you proposed and wait. AUTONOMOUS OPERATION: - When you are told a new alert has fired for a service, begin investigating immediately and create the incident yourself — do NOT wait to be asked. Run the full sequence (steps 1–8) on your own, then STOP at request_confirmation for the operator's approval. diff --git a/argus-server/db/api.ts b/argus-server/db/api.ts index 0ec2db7..1f963a4 100644 --- a/argus-server/db/api.ts +++ b/argus-server/db/api.ts @@ -198,6 +198,16 @@ export function createDeployment(db: Database, { service_id, version, commit_sha return db.query("SELECT * FROM deployments WHERE id = ?").get(Number(result.lastInsertRowid)) as DeploymentRow; } +export function getIncidentRemediationAttempts(db: Database, incidentId: number): number { + const row = db.query("SELECT remediation_attempts FROM incidents WHERE id = ?").get(incidentId) as { remediation_attempts: number } | null; + return row?.remediation_attempts ?? 0; +} + +export function incrementIncidentRemediationAttempts(db: Database, incidentId: number): number { + db.query("UPDATE incidents SET remediation_attempts = remediation_attempts + 1, updated_at = datetime('now') WHERE id = ?").run(incidentId); + return getIncidentRemediationAttempts(db, incidentId); +} + // ── Notifications ── export function createNotification(db: Database, { incident_id, channel, message }: { incident_id: number; channel: string; message: string }): NotificationRow { diff --git a/argus-server/db/schema.ts b/argus-server/db/schema.ts index 6bf3bf2..536bf4b 100644 --- a/argus-server/db/schema.ts +++ b/argus-server/db/schema.ts @@ -37,6 +37,7 @@ CREATE TABLE IF NOT EXISTS incidents ( root_cause_service_id INTEGER REFERENCES services(id), status TEXT NOT NULL DEFAULT 'open' CHECK(status IN ('open', 'investigating', 'mitigated', 'resolved')), runbook_used TEXT, + remediation_attempts INTEGER NOT NULL DEFAULT 0, created_at TEXT DEFAULT (datetime('now')), updated_at TEXT DEFAULT (datetime('now')) ); @@ -125,6 +126,12 @@ function migrate(db: Database) { if (!svcCols.has("recovered_at")) { db.exec("ALTER TABLE services ADD COLUMN recovered_at TEXT"); } + + // incidents.remediation_attempts — count of remediation attempts per incident (loop guard). + const incAttemptCols = new Set((db.query("PRAGMA table_info(incidents)").all() as { name: string }[]).map(c => c.name)); + if (!incAttemptCols.has("remediation_attempts")) { + db.exec("ALTER TABLE incidents ADD COLUMN remediation_attempts INTEGER NOT NULL DEFAULT 0"); + } } export function initDb(dbPath: string): Database { diff --git a/argus-server/db/types.ts b/argus-server/db/types.ts index 4805d86..cf68980 100644 --- a/argus-server/db/types.ts +++ b/argus-server/db/types.ts @@ -49,6 +49,7 @@ export interface IncidentRow { root_cause_service_id: number | null; status: IncidentStatus; runbook_used: string | null; + remediation_attempts: number; created_at: string; updated_at: string; } diff --git a/argus-server/server.ts b/argus-server/server.ts index e9c48fe..d438881 100644 --- a/argus-server/server.ts +++ b/argus-server/server.ts @@ -41,6 +41,7 @@ import { listChat, deleteMessagesForClosedIncidents, isServiceInCooldown, + getIncidentRemediationAttempts, } from "./db/api.js"; const APP_DIR = fileURLToPath(new URL(".", import.meta.url)); @@ -53,9 +54,15 @@ const DB_PATH = process.env.DB_PATH ? resolve(process.env.DB_PATH) : resolve(APP const PORT = Number.parseInt(process.env.PORT || "3000", 10); const LOG_LEVEL = process.env.LOG_LEVEL || "info"; const DISABLE_SCRAPER = process.env.DISABLE_SCRAPER === "1"; +// Maximum remediation attempts the control loop will allow per incident. After this the +// loop stops enqueueing execution and leaves the incident open for operator direction. +const REMEDIATION_ATTEMPT_CAP = Number.parseInt(process.env.REMEDIATION_ATTEMPT_CAP || "3", 10); // External/auto-resolved incidents: how long to WAIT for the operator to say how it was // resolved before falling back to a research-only RCA so the workflow never hangs. const RCA_OPERATOR_TIMEOUT_MS = Number.parseInt(process.env.RCA_OPERATOR_TIMEOUT_SEC || "180", 10) * 1000; +// URL of the dedicated remediation executor. Native modes leave this unset; the execute tool +// refuses rather than pretending it can restart containers. +const EXECUTOR_URL = process.env.EXECUTOR_URL?.trim() || null; // ── Logging ── @@ -638,7 +645,11 @@ function computeWorkflows(): Workflow[] { } const isClosed = (inc: IncidentRow | null | undefined): boolean => - !!inc && (inc.status === "resolved" || inc.status === "mitigated"); + !!inc && inc.status === "resolved"; + // mitigated is intentionally NOT closed here: a mitigated incident with firing alerts is in + // the verifying/cooldown window and should keep owning the service's alerts. Once the + // detection path resolves them, the incident moves to resolved and a fresh failure can + // open a new workflow. const workflows: Workflow[] = []; // Services with an open (non-closed) incident — their firing alerts belong to that @@ -802,7 +813,7 @@ function investigatePrompt(service: string, summary: string): string { return `A new alert has fired for ${service}${summary ? `: ${summary}` : ""}. Investigate now without waiting: correlate the active alerts across services, check ${service} and its dependencies, review recent deployments and runbooks, then create an incident (create_incident) with a likely_cause and root_cause_service. Notify the channel, then propose the remediation step with request_confirmation and STOP — the operator approves in the UI. Do not execute remediation yourself.${history}`; } function executePrompt({ service, incidentId, confirmationId, action }: { service: string; incidentId: number; confirmationId: number; action: string }): string { - return `The operator APPROVED the remediation for incident #${incidentId} (${service}) — confirmation #${confirmationId}${action ? `: "${action}"` : ""}. Execute it now by calling execute_runbook_step with incident_id ${incidentId}, confirmation_id ${confirmationId}, service "${service}", the step_id, and the command you proposed${action ? ` ("${action}")` : ""}. After it returns, briefly confirm what you ran and that the alerts are clearing.`; + return `The operator APPROVED the remediation for incident #${incidentId} (${service}) — confirmation #${confirmationId}${action ? `: "${action}"` : ""}. Execute it now by calling execute_runbook_step with incident_id ${incidentId}, confirmation_id ${confirmationId}, service "${service}", and the step_id you proposed${action ? ` ("${action}")` : ""}. After it returns, briefly confirm what you ran and that the alerts are clearing.`; } // Remediation Argus ran itself — it knows the cause and the fix, so it writes the RCA directly. function rcaRemediatedPrompt({ service, incidentId }: { service: string; incidentId: number }): string { @@ -856,10 +867,12 @@ function reconcileTriage() { // (The investigation transcript is linked to this incident at create_incident time via // linkInvestigationToIncident — see the tool_execution_end handler.) - // 2) Execute: approval on record, agent hasn't run the fix yet (not mitigated), alerts firing. - // Re-derivable after a restart → durable. Guarded once per confirmation per run. + // 2) Execute: approval on record, incident not mitigated/resolved, alerts firing, + // and we have not hit the per-incident remediation attempt cap. + const attemptCapHit = getIncidentRemediationAttempts(db, inc.id) >= REMEDIATION_ATTEMPT_CAP; if (approved && inc.status !== "mitigated" && inc.status !== "resolved" - && wf.firing_alerts.length > 0 && !autoTriage.executed.has(approved.id)) { + && wf.firing_alerts.length > 0 && !autoTriage.executed.has(approved.id) + && !attemptCapHit) { autoTriage.executed.add(approved.id); enqueueAgentTask({ workflowId: wf.id, @@ -868,6 +881,14 @@ function reconcileTriage() { prompt: executePrompt({ service: wf.service_name, incidentId: inc.id, confirmationId: approved.id, action: approved.action }), }); log("info", "triage", `Execute enqueued for incident ${inc.id} (${wf.service_name}) [${wf.id}]`); + } else if (attemptCapHit && approved && !autoTriage.executed.has(approved.id)) { + autoTriage.executed.add(approved.id); + log("warn", "triage", `Skipping execution for incident ${inc.id}: remediation attempt cap (${REMEDIATION_ATTEMPT_CAP}) reached`); + const wid = `inc-${inc.id}`; + const note = `⏸ Remediation attempt cap (${REMEDIATION_ATTEMPT_CAP}) reached for incident #${inc.id}. Execution stopped pending operator direction.`; + try { insertMessage(db, { thread_key: wid, role: "system", content: note, auto: 1 }); } + catch (e) { log("warn", "triage", `Could not persist cap note: ${getErrorMessage(e)}`); } + broadcast({ type: "system_note", workflowId: wid, content: note }); } // 3) RCA: resolved incident with no RCA written yet. diff --git a/argus-server/tests/approval-gate.test.ts b/argus-server/tests/approval-gate.test.ts index d399366..fee32d8 100644 --- a/argus-server/tests/approval-gate.test.ts +++ b/argus-server/tests/approval-gate.test.ts @@ -56,9 +56,8 @@ describe("approval gate — execute_runbook_step", () => { // eslint-disable-next-line @typescript-eslint/no-explicit-any const result = await (executeRunbookStep.execute as any)("test", { incident_id: incidentId, - step_id: 1, + step_id: 2, service: "pgsql", - command: "restart connection pool", confirmation_id: confirmationId, }, undefined, undefined); @@ -74,9 +73,8 @@ describe("approval gate — execute_runbook_step", () => { // eslint-disable-next-line @typescript-eslint/no-explicit-any const result = await (executeRunbookStep.execute as any)("test", { incident_id: incidentId, - step_id: 1, + step_id: 2, service: "pgsql", - command: "restart connection pool", confirmation_id: 9999, }, undefined, undefined); @@ -91,9 +89,8 @@ describe("approval gate — execute_runbook_step", () => { // eslint-disable-next-line @typescript-eslint/no-explicit-any const result = await (executeRunbookStep.execute as any)("test", { incident_id: incidentId, - step_id: 1, + step_id: 2, service: "pgsql", - command: "restart connection pool", confirmation_id: confirmationId, }, undefined, undefined); diff --git a/argus-server/tests/remediation-cap.test.ts b/argus-server/tests/remediation-cap.test.ts new file mode 100644 index 0000000..7dbb28e --- /dev/null +++ b/argus-server/tests/remediation-cap.test.ts @@ -0,0 +1,102 @@ +/** + * Tests for the per-incident remediation attempt cap. + * + * An LLM in a restart loop against a service that will not come back is expensive and + * noisy. execute_runbook_step must stop after a configured number of attempts and leave + * the incident open for operator direction. The reconciler also skips execution once the + * cap is reached. + */ +import { describe, expect, test, beforeEach, afterEach } from "bun:test"; +import { Database } from "bun:sqlite"; +import { initDb } from "../db/schema.ts"; +import { seedIfEmpty } from "../db/seed.ts"; +import { initArgusTools, argusTools } from "../tools/argus-tools.ts"; + +let db: Database; +const ORIGINAL_EXECUTOR_URL = process.env.EXECUTOR_URL; +const ORIGINAL_CAP = process.env.REMEDIATION_ATTEMPT_CAP; + +beforeEach(() => { + db = initDb(":memory:"); + seedIfEmpty(db); + initArgusTools(db); + // Point at a port that will refuse the connection so every call fails deterministically. + process.env.EXECUTOR_URL = "http://127.0.0.1:1"; + process.env.REMEDIATION_ATTEMPT_CAP = "3"; +}); + +afterEach(() => { + process.env.EXECUTOR_URL = ORIGINAL_EXECUTOR_URL; + process.env.REMEDIATION_ATTEMPT_CAP = ORIGINAL_CAP; +}); + +const executeRunbookStep = argusTools.find(t => t.name === "execute_runbook_step")!; + +function parseToolResult(result: { content: { type: string; text?: string }[] }) { + const first = result.content[0]; + if (!first || first.type !== "text" || typeof first.text !== "string") throw new Error("unexpected tool result shape"); + return JSON.parse(first.text) as Record; +} + +function createApprovedIncident() { + const svcId = (db.query("SELECT id FROM services WHERE name = 'pgsql'").get() as { id: number }).id; + + db.query("INSERT INTO incidents (title, severity, status) VALUES (?, ?, ?)").run( + "pgsql pool exhausted", "critical", "investigating" + ); + const incidentId = (db.query("SELECT last_insert_rowid() AS id").get() as { id: number }).id; + + db.query("INSERT INTO confirmations (incident_id, action, status) VALUES (?, ?, ?)").run( + incidentId, "Restart pgsql container", "approved" + ); + const confirmationId = (db.query("SELECT last_insert_rowid() AS id").get() as { id: number }).id; + + return { incidentId, confirmationId, svcId }; +} + +describe("remediation attempt cap", () => { + test("allows attempts up to the cap and then refuses", async () => { + const { incidentId, confirmationId } = createApprovedIncident(); + + for (let i = 1; i <= 3; i++) { + // eslint-disable-next-line @typescript-eslint/no-explicit-any + const result = await (executeRunbookStep.execute as any)("test", { + incident_id: incidentId, + step_id: 2, + service: "pgsql", + confirmation_id: confirmationId, + }, undefined, undefined); + const parsed = parseToolResult(result); + expect(parsed.error).toBeTruthy(); + expect(String(parsed.error)).toContain("executor"); + expect(parsed.attempts).toBe(i); + } + + // Fourth call must be refused by the cap, not by the executor. + // eslint-disable-next-line @typescript-eslint/no-explicit-any + const blocked = await (executeRunbookStep.execute as any)("test", { + incident_id: incidentId, + step_id: 2, + service: "pgsql", + confirmation_id: confirmationId, + }, undefined, undefined); + const parsed = parseToolResult(blocked); + expect(String(parsed.error)).toContain("cap"); + expect(String(parsed.error)).toContain("3"); + }); + + test("refuses a non-executable runbook step", async () => { + const { incidentId, confirmationId } = createApprovedIncident(); + + // eslint-disable-next-line @typescript-eslint/no-explicit-any + const result = await (executeRunbookStep.execute as any)("test", { + incident_id: incidentId, + step_id: 1, + service: "pgsql", + confirmation_id: confirmationId, + }, undefined, undefined); + const parsed = parseToolResult(result); + expect(String(parsed.error)).toContain("manual"); + expect(String(parsed.error)).toContain("not \"executable\""); + }); +}); diff --git a/argus-server/tools/argus-tools.ts b/argus-server/tools/argus-tools.ts index 003e849..25b7e2f 100644 --- a/argus-server/tools/argus-tools.ts +++ b/argus-server/tools/argus-tools.ts @@ -12,7 +12,6 @@ import { updateServiceStatus, listActiveAlerts, listAllAlerts, - resolveAlert, createIncident, getIncident, updateIncident, @@ -23,6 +22,8 @@ import { createNotification, createConfirmation, markServiceRecovered, + getIncidentRemediationAttempts, + incrementIncidentRemediationAttempts, } from "../db/api.js"; let db!: Database; @@ -366,12 +367,11 @@ const requestConfirmationTool = defineTool({ const executeRunbookStepTool = defineTool({ name: "execute_runbook_step", label: "Execute Runbook Step", - description: `Execute a specific runbook step after the operator has APPROVED the confirmation. You MUST call request_confirmation first and only call this once approval is on record. This actually performs the remediation: it recovers the affected service(s) and clears their firing alerts.`, + description: `Execute an approved runbook remediation step. You MUST call request_confirmation first and only call this once approval is on record. The step is identified by service and step_id; the actual command is read from the runbook by deterministic code, not taken from you. After it runs, the tool verifies the target service is healthy and reports what it observed; alert resolution is left to the normal detection path.`, parameters: Type.Object({ incident_id: Type.Number({ description: "The incident ID" }), step_id: Type.Number({ description: "The runbook step ID to execute" }), - service: Type.String({ description: "Service name the step is for" }), - command: Type.String({ description: "The command to execute" }), + service: Type.String({ description: "Service name the runbook belongs to" }), confirmation_id: Type.Number({ description: "The confirmation ID from request_confirmation" }), }), executionMode: "sequential", @@ -386,75 +386,106 @@ const executeRunbookStepTool = defineTool({ return toolResult({ error: `Confirmation ${params.confirmation_id} is "${confirmation.status || "pending"}", not approved. You may only execute after the operator approves in the UI.` }); } - // Build the set of services to recover: the step's target, the incident's - // root-cause service, and any services with alerts correlated to the incident. - const recoverIds = new Set(); - const targetSvc = params.service ? getService(db, params.service) : null; - if (targetSvc) recoverIds.add(targetSvc.id); - - const incident = getIncident(db, params.incident_id); - if (incident?.root_cause_service_id) recoverIds.add(incident.root_cause_service_id); + // Loop guard: cap remediation attempts per incident. + const attempts = getIncidentRemediationAttempts(db, params.incident_id); + const cap = remediationAttemptCap(); + if (attempts >= cap) { + const msg = `Remediation attempt cap (${cap}) reached for incident ${params.incident_id}. The incident stays open pending operator direction.`; + createNotification(db, { incident_id: params.incident_id, channel: "#platform-engineering", message: msg }); + return toolResult({ error: msg, incident_id: params.incident_id, attempts, cap }); + } - for (const a of getIncidentAlerts(db, params.incident_id)) recoverIds.add(a.service_id); + // Native modes have no Docker topology; refuse rather than pretend. + const executorUrl = executorBaseUrl(); + if (!executorUrl) { + return toolResult({ + error: "No remediation executor is configured. Container restart only works in the full Docker topology (`docker compose up`). In native mode, resolve the failure manually and the scraper/Alertmanager path will clear the alerts.", + incident_id: params.incident_id, + }); + } - const recovered: string[] = []; - const failed: string[] = []; - for (const sid of recoverIds) { - const svc = db.query("SELECT * FROM services WHERE id = ?").get(sid) as ServiceRow | null; - if (!svc) continue; - // Best-effort: heal the demo service so its metrics return to healthy. - try { - await fetch(`${serviceBaseUrl(svc)}/recover`, { method: "POST", signal: AbortSignal.timeout(2000) }); - } catch { /* best effort — verified below regardless */ } - - // Verify before declaring victory: poll /metrics until the service reads healthy - // (bounded). This catches a slow/failed /recover instead of silently assuming success, - // and means alerts are only cleared once metrics have actually settled — which (with - // the scraper's post-recovery cooldown) is what kills phantom re-detection. - const healthy = await pollUntilHealthy(svc, 8000, 750); - if (!healthy) { - failed.push(svc.name); - continue; - } - recovered.push(svc.name); - // Stamp the cooldown so the scraper won't re-create alerts on a boundary reading. - markServiceRecovered(db, sid); - // Clear its firing alerts. - const firing = db.query("SELECT id FROM alerts WHERE service_id = ? AND status = 'firing'").all(sid) as { id: number }[]; - for (const a of firing) resolveAlert(db, a.id); + // Load the runbook and step. Deterministic code, not the model, decides what runs. + const runbook = await loadRunbook(params.service); + if (!runbook) { + return toolResult({ error: `No runbook found for service "${params.service}".` }); + } + const step = getRunbookStep(runbook, params.step_id); + if (!step) { + return toolResult({ error: `Step ${params.step_id} not found in ${params.service} runbook.` }); + } + if (step.action !== "executable") { + return toolResult({ error: `Step ${params.step_id} in ${params.service} runbook is "${step.action}", not "executable". Only executable steps can be run through the executor.` }); } + const target = step.target; + if (!target || !EXECUTOR_ALLOWED_TARGETS.has(target)) { + return toolResult({ error: `Step ${params.step_id} targets "${target ?? "unknown"}", which is not in the remediation allowlist. Allowed targets: ${[...EXECUTOR_ALLOWED_TARGETS].join(", ")}.` }); + } + + // Record that we are attempting remediation (atomic increment). + incrementIncidentRemediationAttempts(db, params.incident_id); - // Flip services with no remaining firing alerts back to healthy. - for (const s of db.query("SELECT id FROM services").all() as { id: number }[]) { - const cnt = db.query("SELECT COUNT(*) AS c FROM alerts WHERE service_id = ? AND status = 'firing'").get(s.id) as { c: number }; - if (cnt.c === 0) { - db.query("UPDATE services SET status = 'healthy', last_checked = datetime('now') WHERE id = ?").run(s.id); + // Trigger the real container restart through the dedicated executor. + let executorOutput: string | null = null; + try { + const resp = await fetch(`${executorUrl}/execute`, { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ action: "restart", target }), + signal: AbortSignal.timeout(120000), + }); + const data = await resp.json() as Record; + if (!resp.ok) { + const detail = typeof data.error === "string" ? data.error : JSON.stringify(data); + return toolResult({ error: `Executor refused: ${detail}`, incident_id: params.incident_id, step_id: params.step_id, service: params.service, target }); } + executorOutput = typeof data.output === "string" ? data.output : null; + } catch (err) { + return toolResult({ + error: `Failed to reach remediation executor at ${executorUrl}: ${err instanceof Error ? err.message : String(err)}`, + incident_id: params.incident_id, + attempts: getIncidentRemediationAttempts(db, params.incident_id), + }); } - // Only declare the incident mitigated when every targeted service verified healthy. - // If recovery didn't take, leave the incident open so the operator can retry. - if (failed.length === 0) { - updateIncident(db, { id: params.incident_id, status: "mitigated" }); - // Auto-decline any other pending confirmations for this incident — the fix ran, - // so leftover proposals (e.g. from a declined-then-superseded suggestion) are moot. - db.query( - "UPDATE confirmations SET status = 'declined', resolved_at = datetime('now') WHERE incident_id = ? AND id != ? AND (status IS NULL OR status = 'pending')" - ).run(params.incident_id, params.confirmation_id); + // Verify the restart had the intended effect before declaring the incident mitigated. + const targetSvc = getService(db, target); + if (!targetSvc) { + return toolResult({ error: `Target service "${target}" is not known to Argus.`, incident_id: params.incident_id }); + } + const healthy = await pollUntilHealthy(targetSvc, 8000, 750); + + if (!healthy) { + return toolResult({ + error: `Restarted ${target}, but it did not return to healthy within the timeout — the incident remains open.`, + incident_id: params.incident_id, + step_id: params.step_id, + service: params.service, + target, + attempts: getIncidentRemediationAttempts(db, params.incident_id), + executor_output: executorOutput, + }); } - const ok = failed.length === 0; + // Success: stamp the cooldown so the scraper won't re-create alerts on a boundary reading. + markServiceRecovered(db, targetSvc.id); + + // Mark the incident mitigated. Alert resolution is intentionally left to the detection + // path (built-in scraper or vmalert → Alertmanager), not to this tool. + updateIncident(db, { id: params.incident_id, status: "mitigated" }); + + // Auto-decline any other pending confirmations for this incident — the fix ran. + db.query( + "UPDATE confirmations SET status = 'declined', resolved_at = datetime('now') WHERE incident_id = ? AND id != ? AND (status IS NULL OR status = 'pending')" + ).run(params.incident_id, params.confirmation_id); + return toolResult({ incident_id: params.incident_id, step_id: params.step_id, service: params.service, - command: params.command, - status: ok ? "executed" : "incomplete", - recovered_services: recovered, - failed_services: failed, - message: ok - ? `Executed "${params.command}". Verified ${recovered.join(", ") || "no"} service(s) returned to healthy and cleared their firing alerts.` - : `Executed "${params.command}" but ${failed.join(", ")} did not return to healthy within the timeout — alerts left firing and the incident remains open. The /recover may have failed; investigate and retry.`, + target, + status: "executed", + executor_output: executorOutput, + message: `Restarted ${target} via the executor and verified it returned to healthy. The incident is mitigated; alerts should clear through the normal detection path.`, timestamp: new Date().toISOString(), }); } catch (err) { return toolError(err); } @@ -620,7 +651,28 @@ Use this when you need historical context, metric trends, or infrastructure logs // ── Helpers ── interface RunbookTrigger { pattern: string; [k: string]: unknown } -interface Runbook { service: string; triggers: RunbookTrigger[]; [k: string]: unknown } +interface RunbookStep { + id: number; + title?: string; + action: string; + instruction?: string; + target?: string; + command?: string; + requires_confirmation?: boolean; + rollback?: string; + [k: string]: unknown; +} +interface Runbook { service: string; triggers: RunbookTrigger[]; steps: RunbookStep[]; [k: string]: unknown } + +const EXECUTOR_ALLOWED_TARGETS = new Set(["gateway", "app-ui", "app-api", "auth", "pgsql", "redis"]); +const DEFAULT_REMEDIATION_ATTEMPT_CAP = 3; + +function remediationAttemptCap(): number { + const env = process.env.REMEDIATION_ATTEMPT_CAP; + if (!env) return DEFAULT_REMEDIATION_ATTEMPT_CAP; + const n = Number.parseInt(env, 10); + return Number.isNaN(n) || n < 1 ? DEFAULT_REMEDIATION_ATTEMPT_CAP : n; +} async function loadAllRunbooks(): Promise { try { @@ -638,6 +690,30 @@ async function loadAllRunbooks(): Promise { } } +async function loadRunbook(service: string): Promise { + try { + const files = await readdir(RUNBOOKS_DIR); + const file = files.find(f => f === `${service}.yaml` || f === `${service}.yml`); + if (!file) return null; + const content = await readFile(join(RUNBOOKS_DIR, file), "utf-8"); + return parseYaml(content) as unknown as Runbook; + } catch { + return null; + } +} + +function getRunbookStep(runbook: Runbook, stepId: number): RunbookStep | null { + if (!Array.isArray(runbook.steps)) return null; + return runbook.steps.find((s: RunbookStep) => s.id === stepId) || null; +} + +// Resolve the base URL of the remediation executor. Native modes leave this unset; +// the tool refuses to execute rather than pretending it can restart containers. +function executorBaseUrl(): string | null { + const raw = process.env.EXECUTOR_URL?.trim(); + return raw || null; +} + // Poll a service's /metrics until it reads healthy, or the bounded timeout elapses. // Used by execute_runbook_step to confirm a recovery actually took before clearing alerts. async function pollUntilHealthy(svc: ServiceRow, timeoutMs: number, intervalMs: number): Promise { diff --git a/docker-compose.yml b/docker-compose.yml index 4b4d843..6388146 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -26,6 +26,7 @@ services: gateway: build: ./demo-services image: argus-demo-services + container_name: gateway command: ["bun", "demo-services/gateway.ts"] ports: ["8081:8081"] restart: unless-stopped @@ -33,6 +34,7 @@ services: app-ui: build: ./demo-services image: argus-demo-services + container_name: app-ui command: ["bun", "demo-services/app-ui.ts"] ports: ["8082:8082"] restart: unless-stopped @@ -40,6 +42,7 @@ services: app-api: build: ./demo-services image: argus-demo-services + container_name: app-api command: ["bun", "demo-services/app-api.ts"] ports: ["8083:8083"] restart: unless-stopped @@ -47,6 +50,7 @@ services: auth: build: ./demo-services image: argus-demo-services + container_name: auth command: ["bun", "demo-services/auth.ts"] ports: ["8084:8084"] restart: unless-stopped @@ -54,6 +58,7 @@ services: pgsql: build: ./demo-services image: argus-demo-services + container_name: pgsql command: ["bun", "demo-services/pgsql.ts"] ports: ["8085:8085"] restart: unless-stopped @@ -61,6 +66,7 @@ services: redis: build: ./demo-services image: argus-demo-services + container_name: redis command: ["bun", "demo-services/redis.ts"] ports: ["8086:8086"] restart: unless-stopped @@ -75,6 +81,18 @@ services: ports: ["8090:8090"] restart: unless-stopped + # ── Remediation executor (the only container with Docker socket access) ── + argus-executor: + build: ./argus-executor + image: argus-executor + ports: ["3001:3001"] + environment: + EXECUTOR_PORT: "3001" + EXECUTOR_ALLOWLIST: "gateway,app-ui,app-api,auth,pgsql,redis" + volumes: + - /var/run/docker.sock:/var/run/docker.sock + restart: unless-stopped + # ── Observability stack ── victoriametrics: image: victoriametrics/victoria-metrics:latest @@ -141,6 +159,11 @@ services: # How long to wait for an operator to explain an external/auto resolution before # Argus writes a research-only RCA (human-in-the-loop gate on the RCA). RCA_OPERATOR_TIMEOUT_SEC: "${RCA_OPERATOR_TIMEOUT_SEC:-180}" + # URL of the dedicated remediation executor. Native modes leave this unset; the tool + # refuses to execute rather than pretending it can restart non-existent containers. + EXECUTOR_URL: "http://argus-executor:3001" + # Maximum remediation attempts per incident before the control loop gives up. + REMEDIATION_ATTEMPT_CAP: "${REMEDIATION_ATTEMPT_CAP:-3}" # The one cross-boundary link: Ollama runs on the host, not in compose. OLLAMA_BASE_URL: "http://host.docker.internal:11434" OLLAMA_MODEL: "${OLLAMA_MODEL:-}" diff --git a/runbooks/services/app-api.yaml b/runbooks/services/app-api.yaml index a96a800..be19448 100644 --- a/runbooks/services/app-api.yaml +++ b/runbooks/services/app-api.yaml @@ -51,7 +51,7 @@ steps: - id: 5 title: Restart app-api container action: executable - command: "docker compose restart app-api" + target: app-api requires_confirmation: true rollback: "Redeploy from source: docker compose up -d --build --no-deps app-api" diff --git a/runbooks/services/app-ui.yaml b/runbooks/services/app-ui.yaml index af33abf..f33afd3 100644 --- a/runbooks/services/app-ui.yaml +++ b/runbooks/services/app-ui.yaml @@ -43,7 +43,7 @@ steps: - id: 4 title: Restart app-ui container action: executable - command: "docker compose restart app-ui" + target: app-ui requires_confirmation: true rollback: "Redeploy from source: docker compose up -d --build --no-deps app-ui" diff --git a/runbooks/services/auth.yaml b/runbooks/services/auth.yaml index 8e0b907..0cdec19 100644 --- a/runbooks/services/auth.yaml +++ b/runbooks/services/auth.yaml @@ -44,7 +44,7 @@ steps: - id: 4 title: Restart auth container action: executable - command: "docker compose restart auth" + target: auth requires_confirmation: true rollback: "Redeploy from source: docker compose up -d --build --no-deps auth" diff --git a/runbooks/services/gateway.yaml b/runbooks/services/gateway.yaml index ece726f..109d614 100644 --- a/runbooks/services/gateway.yaml +++ b/runbooks/services/gateway.yaml @@ -36,7 +36,7 @@ steps: - id: 3 title: Restart gateway container action: executable - command: "docker compose restart gateway" + target: gateway requires_confirmation: true rollback: "Redeploy from source: docker compose up -d --build --no-deps gateway" diff --git a/runbooks/services/pgsql.yaml b/runbooks/services/pgsql.yaml index 69a4712..3d6a5d9 100644 --- a/runbooks/services/pgsql.yaml +++ b/runbooks/services/pgsql.yaml @@ -32,7 +32,7 @@ steps: - id: 2 title: Restart pgsql container action: executable - command: "docker compose restart pgsql" + target: pgsql requires_confirmation: true rollback: "Redeploy from source: docker compose up -d --build --no-deps pgsql" diff --git a/runbooks/services/redis.yaml b/runbooks/services/redis.yaml index 4bf1887..9709e10 100644 --- a/runbooks/services/redis.yaml +++ b/runbooks/services/redis.yaml @@ -40,15 +40,18 @@ steps: - id: 3 title: Flush stale keys - action: executable - command: "docker compose exec redis sh -c 'redis-cli --scan --pattern \"stale:*\" | xargs redis-cli del'" - requires_confirmation: true - rollback: "No rollback — keys are already stale" + action: manual + instruction: > + If redis_memory_used_ratio > 0.90, Redis is under memory + pressure. Check for large keys or stale cache entries: + docker compose exec redis redis-cli INFO memory + To flush stale keys manually: + docker compose exec redis sh -c 'redis-cli --scan --pattern "stale:*" | xargs redis-cli del' - id: 4 title: Restart Redis container action: executable - command: "docker compose restart redis" + target: redis requires_confirmation: true rollback: "Redeploy from source: docker compose up -d --build --no-deps redis" From 12ef224905a474b764637495dfb6006118e4623b Mon Sep 17 00:00:00 2001 From: Sandipan Das Date: Sun, 20 Sep 2026 21:08:22 -0400 Subject: [PATCH 2/2] fix: mitigated incident with firing alerts shows verifying, not executing After execute_runbook_step reports back (incident.status = mitigated), the UI should display 'verifying' while it waits for the detection path to clear the remaining firing alerts, instead of staying in 'executing'. Update the workflow state machine test to match. --- argus-server/db/workflow.ts | 8 ++++---- argus-server/tests/workflow-step.test.ts | 11 ++++++----- 2 files changed, 10 insertions(+), 9 deletions(-) diff --git a/argus-server/db/workflow.ts b/argus-server/db/workflow.ts index 5a383d1..08d2d14 100644 --- a/argus-server/db/workflow.ts +++ b/argus-server/db/workflow.ts @@ -19,11 +19,11 @@ export function computeWorkflowStep( if (firingAlerts.length === 0) return "resolved"; if (incident && incident.status === "resolved") return "resolved"; - // Approved confirmation gate → executing (takes priority over mitigated) - if (approved.length > 0) return "executing"; - // Incident mitigated but firing alerts remain → still executing (cooldown: metrics settling). + // Approved confirmation gate, and execution has not yet reported back → executing. + if (approved.length > 0 && incident?.status !== "mitigated") return "executing"; + // Incident mitigated but firing alerts remain → verifying (cooldown: metrics settling). // If all alerts had resolved, we'd have returned "resolved" above. - if (incident && incident.status === "mitigated") return "executing"; + if (incident && incident.status === "mitigated") return "verifying"; if (pending.length > 0) return "confirmation"; if (incident) { diff --git a/argus-server/tests/workflow-step.test.ts b/argus-server/tests/workflow-step.test.ts index d8a4940..c353b7d 100644 --- a/argus-server/tests/workflow-step.test.ts +++ b/argus-server/tests/workflow-step.test.ts @@ -43,8 +43,8 @@ describe("computeWorkflowStep — pipeline state machine", () => { expect(computeWorkflowStep([firing], incident({ status: "resolved" }), [])).toBe("resolved"); }); - test("incident mitigated + firing alerts → executing (cooldown: metrics still settling)", () => { - expect(computeWorkflowStep([firing], incident({ status: "mitigated" }), [])).toBe("executing"); + test("incident mitigated + firing alerts → verifying (cooldown: metrics still settling)", () => { + expect(computeWorkflowStep([firing], incident({ status: "mitigated" }), [])).toBe("verifying"); }); test("approved confirmation → executing (approval gate cleared)", () => { @@ -90,10 +90,11 @@ describe("computeWorkflowStep — pipeline state machine", () => { expect(computeWorkflowStep([firing], null, [])).toBe("detected"); }); - test("approved confirmation takes priority over mitigated status", () => { - // Both paths → "executing"; this confirms the order doesn't matter for that outcome. + test("approved confirmation on mitigated incident with firing alerts → verifying", () => { + // Once execution has reported back (status = mitigated), the presence of an approved + // confirmation no longer matters; we are waiting for alerts to clear. const r1 = computeWorkflowStep([firing], incident({ status: "mitigated" }), [confirmation("approved")]); - expect(r1).toBe("executing"); + expect(r1).toBe("verifying"); }); test("one alert still firing = still active (partial recovery)", () => {