sdOps · Sep 21, 2026 · Sep 21, 2026 · Sep 21, 2026
@@ -5,11 +5,10 @@ demo microservices stack, and when services degrade it investigates, opens an in
 proposes a fix, waits for a human to approve, executes the remediation, and writes a
 root-cause analysis, mostly on its own.
 
-It is a **demo / reference system, not a product**. The failure scenarios are scripted, the
-runbook "commands" hit mock `/recover` endpoints, and a single sequential agent session
-won't scale to a real org's alert volume. What it _is_ meant to be is an honest, working
-illustration of how to wire an LLM agent into an operational control loop, and more specifically
-of **where such a system needs determinism and where it needs judgment.**
+It is a **demo / reference system, not a product**. The failure scenarios are scripted, and a
+single sequential agent session won't scale to a real org's alert volume. What it _is_ meant to
+be is an honest, working illustration of how to wire an LLM agent into an operational control
+loop, and more specifically of **where such a system needs determinism and where it needs judgment.**
 
 That boundary is the thesis of this document.
 
@@ -35,6 +34,10 @@ graph TB
     db[(SQLite)]
   end
 
+  subgraph executor["argus-executor — Docker only"]
+    ex[POST /execute]
+  end
+
   ui[argus-ui — React/Vite, :5173]
 
   demo -- GET /metrics --> scr
@@ -44,7 +47,8 @@ graph TB
   rec -- reads desired state --> db
   rec -- enqueues tasks --> ag
   ag -- tool calls --> db
-  ag -- POST /recover --> demo
+  ag -- POST /execute --> ex
+  ex -- docker restart --> demo
   rest --> db
   ui -- poll /api/workflows, /api/chat --> rest
   rest -- WS broadcast (tagged by workflowId) --> ui
@@ -54,6 +58,7 @@ graph TB
 |---|---|---|---|
 | **argus-server** | Bun | 3000 | Headless agent, reconciler, REST API, alert scraper, WebSocket |
 | **argus-ui** | React + Vite | 5173 | Display-only triage console (pipeline, incidents, chat) |
+| **argus-executor** | Bun + Docker CLI | 3001 | Dedicated remediation worker: the only container with Docker socket access |
 | **demo-services** | Bun | 8081–8086 | Six mock services with `GET /metrics`, `POST /fail`, `POST /recover` |
 | **infra** | Docker Compose | — | VictoriaMetrics + vmalert + Alertmanager (optional alert source) |
 | **runbooks/** | YAML | — | Per-service triggers, steps, known failure patterns |
@@ -226,8 +231,10 @@ without a human approval on record.** The flow:
    row to `approved`. That's all the endpoint does; the approval is now durable.
 3. The reconciler sees `approved + not mitigated + alerts firing` and enqueues execution.
 4. The agent calls `execute_runbook_step`, which **re-checks the gate** (refuses unless the
-   confirmation is `approved`), then performs the real recovery: `POST /recover` on the
-   affected services, resolves their alerts, marks the incident mitigated.
+   confirmation is `approved`), looks up the typed runbook step (`action: executable`, `target`),
+   validates the target against the executor allowlist, and routes the real container restart
+   through `argus-executor`. It polls the service until metrics verify healthy, then marks the
+   incident `mitigated`. Alert resolution is left to the detection path.
 
 The gate is enforced in two independent places: the UI flow and the tool itself. The agent cannot talk itself past it.
 
@@ -326,15 +333,21 @@ The line between deterministic code and the LLM is the thing to take from this a
 | Deciding *when* to act (reconciler) | Diagnosing the root cause |
 | Computing workflow / pipeline state | Proposing a remediation step |
 | Enforcing the approval gate | Writing the incident + RCA narrative |
-| Performing the DB writes & `/recover` call | Asking the operator the right question |
+| Performing the typed container restart via `argus-executor` | Asking the operator the right question |
 | Persisting transcript & aliasing threads | Judgment under ambiguity |
 | Shipping & indexing logs (future) | Reading a log line and deciding what it means |
 
-Detection, scheduling, state, and safety are **mechanism**: they must be predictable, so they
-are code. Correlation, diagnosis, and explanation are **judgment**: they benefit from a model.
-Keeping the model out of the control loop's mechanics is what makes an autonomous agent safe to
-run unattended. Not everything splits that cleanly; the last row in the table straddles the
-boundary, and that's where the interesting design questions live.
+The remediation row is the important one for the safety argument: the model names a
+pre-approved runbook step, but deterministic code decides what that step maps to, validates
+the target against an allowlist, and routes the real container restart through a dedicated
+executor that owns the Docker socket. The model never supplies a raw command string, and the
+executor validates independently even of `argus-server`.
+
+The three primed scenarios (`db`, `cache`, `auth`) exercise the control loop against
+runbooks that pre-map symptoms to root causes. The `api-oom` scenario is different:
+`app-api` fails directly, with no upstream cause and no matching `known_patterns` entry,
+so the agent must reason from live metrics and runbook steps without the shortcut.
+That is the unprimed case, and it is what tests the judgment side of the boundary.
 
 ### 7.1 Instrumenting the agentic side
 
@@ -400,13 +413,19 @@ All access goes through `argus-server/db/api.ts`; tools and handlers never embed
 
 - Failure scenarios are **scripted** (`POST /fail` / `POST /recover`); this is not chaos
   engineering.
-- Remediation is **simulated** against mock endpoints. No real infrastructure is touched.
+- The full **container restart** remediation path only runs under `docker compose up` /
+  `mise run infra:up`, where an `argus-executor` container owns the Docker socket. In native
+  modes (`mise run dev`, `mise run infra:start`) there are no containers to restart, so
+  `execute_runbook_step` refuses rather than simulating.
 - A single session won't handle **production alert volume**; this is an architecture demo.
 - Autonomy **requires the model to be reachable** (Ollama in dev); if it's down, detection and
   state still work, but triage pauses (the reconciler no-ops until the agent is up).
 - Narrow restart edge: if the server restarts in the seconds between an investigation finishing
   and the incident being aliased, that one transcript can stay under the pending thread.
 
+The `POST /recover` endpoints remain on the demo services for manual resets and the `recover`
+demo scenario; the agent no longer calls them.
+
 ---
 
 ## 11. Repository layout

@@ -82,11 +82,13 @@ mise run dev              # server + UI + services, with bun --watch hot-reload
 ## Demo scenarios
 
 ```bash
-mise run infra:demo -- db       # database cascade
-mise run infra:demo -- cache    # redis eviction storm
-mise run infra:demo -- auth     # auth outage
-mise run infra:demo -- all      # everything fails
-mise run infra:demo -- recover  # restore all services
+mise run infra:demo -- db         # database cascade
+mise run infra:demo -- cache      # redis eviction storm
+mise run infra:demo -- auth       # auth outage
+mise run infra:demo -- frontend   # app-ui failure
+mise run infra:demo -- api-oom    # app-api crash, no runbook match
+mise run infra:demo -- all        # everything fails
+mise run infra:demo -- recover    # restore all services
 ```
 
 ## URLs
@@ -119,6 +121,12 @@ Each service exposes `GET /metrics` (Prometheus format), `POST /fail`, and `POST
 2. **Auth outage:** auth fails → token validation errors → app-api 401s → gateway errors. Root cause: auth
 3. **Cache eviction storm:** redis memory pressure → cache misses → app-api latency → pgsql pool exhaustion. Root cause: redis
 
+## The unprimed scenario (`api-oom`)
+
+`api-oom` fails the `app-api` process directly. The agent can read live metrics, check that `auth`, `pgsql`, and `redis` are healthy, and follow the `app-api` runbook steps, but the runbook's `known_patterns` all point to upstream dependencies. There is no named pattern for a direct app-api failure, so the agent has to reason without being handed the answer. Watch whether it proposes a sensible restart anyway, or whether it gets stuck looking for an upstream cause that does not exist.
+
+This is the scenario that tests the judgment side of the determinism boundary, not just the control loop.
+
 ## Prerequisites
 
 - [mise](https://mise.jdx.dev) — **required**: it runs every `mise run …` command, pins the
@@ -172,8 +180,10 @@ Each monitored service has a YAML runbook in [`runbooks/services/`](runbooks/ser
 the operational knowledge an on-call engineer would reach for. A runbook declares:
 
 - **`triggers`** — the metrics and thresholds that signal trouble (e.g. `pgsql_connection_pool_used > 0.90`).
-- **`steps`** — ordered remediation steps, each typed `manual`, `executable` (with a `command`,
-  `requires_confirmation`, and a `rollback`), or `notify` (escalate to a channel).
+- **`steps`** — ordered remediation steps, each typed `manual`, `executable` (with a `target`
+  container, `requires_confirmation`, and a `rollback`), or `notify` (escalate to a channel).
+  The agent names a pre-approved step; deterministic code maps that to a typed action and
+  runs it through the executor. The model never supplies a raw command string.
 - **`known_patterns`** — named failure patterns that map a cluster of symptoms to a **root cause**.
   This is what lets the agent resolve a multi-service cascade to the one service actually at fault.
 - **`related_runbooks`** — links to dependent services' runbooks.
@@ -218,7 +228,7 @@ These runbooks are curated, human-authored knowledge the agent consults. The mod
 | `get_recent_deployments` | Check recent deployments |
 | `notify_channel` | Send notification to a Slack channel |
 | `request_confirmation` | Request human approval before remediation |
-| `execute_runbook_step` | Execute a remediation step after confirmation |
+| `execute_runbook_step` | Execute an approved, typed runbook step via the executor after verification |
 
 ## All mise tasks
 
@@ -237,4 +247,17 @@ These runbooks are curated, human-authored knowledge the agent consults. The mod
 
 > `infra:up`/`infra:down` wrap `docker compose` on the top-level `docker-compose.yml`.
 > The infra-only observability stack (for use with native services) remains
-> `infra/docker-compose.yml`, driven by `mise run infra:start`/`infra:stop`.
+> `infra/docker-compose.yml`, driven by `mise run infra:start`/`infra:stop`.
+
+## Remediation and the executor
+
+In the full Docker topology (`docker compose up` / `mise run infra:up`), approved
+remediation runs through a dedicated `argus-executor` container. The executor is the only
+component with access to `/var/run/docker.sock`; it accepts a typed `{ action, target }`
+request, validates the target against an allowlist of the six demo services, and restarts the
+real container. The agent never supplies a raw command string.
+
+In native modes (`mise run dev` and `mise run infra:start`) there are no containers to
+restart, so `execute_runbook_step` refuses with a clear message rather than pretending. The
+`/recover` endpoints on the demo services still work for manual resets and the `recover`
+demo scenario.
@@ -0,0 +1,22 @@
+# argus-executor — the only component that touches /var/run/docker.sock.
+# It exposes POST /execute and runs only pre-approved actions against an allowlist.
+#
+# Build context is the argus-executor/ directory:
+#   docker build -f argus-executor/Dockerfile ./argus-executor
+
+# Copy a static Docker CLI binary so the Bun image does not need Docker-in-Docker.
+FROM docker:27-cli AS docker
+
+FROM oven/bun:1.3.6
+
+COPY --from=docker /usr/local/bin/docker /usr/local/bin/docker
+
+WORKDIR /app/argus-executor
+COPY package.json ./
+RUN bun install
+
+COPY . ./
+
+EXPOSE 3001
+ENV EXECUTOR_PORT=3001
+CMD ["bun", "run", "main.ts"]
@@ -0,0 +1,125 @@
+// Argus executor — tiny, hard-coded remediation worker.
+//
+// This service is the ONLY component that touches /var/run/docker.sock. It exposes a
+// single POST /execute endpoint that accepts a typed { action, target } and runs only
+// pre-approved actions against an allowlist of demo service containers. It validates on
+// its own; it does not trust argus-server to decide what is safe.
+
+const PORT = Number.parseInt(process.env.EXECUTOR_PORT || "3001", 10);
+const DEFAULT_ALLOWLIST = ["gateway", "app-ui", "app-api", "auth", "pgsql", "redis"];
+const ALLOWLIST = process.env.EXECUTOR_ALLOWLIST
+  ? process.env.EXECUTOR_ALLOWLIST.split(",").map(s => s.trim()).filter(Boolean)
+  : DEFAULT_ALLOWLIST;
+const ALLOWSET = new Set(ALLOWLIST);
+
+const DENYLIST = new Set(["argus-server", "argus-executor", "victoriametrics", "vmalert", "alertmanager", "mock-slack"]);
+
+const ACTIONS = new Set(["restart"]);
+
+type LogLevel = "info" | "warn" | "error";
+function log(level: LogLevel, msg: string, extra?: Record<string, unknown>) {
+  const line = { ts: new Date().toISOString(), level, msg, ...extra };
+  console[level === "error" ? "error" : level === "warn" ? "warn" : "log"](JSON.stringify(line));
+}
+
+interface ExecuteRequest {
+  action: string;
+  target: string;
+}
+
+function isExecuteRequest(body: unknown): body is ExecuteRequest {
+  if (typeof body !== "object" || body === null) return false;
+  const b = body as Record<string, unknown>;
+  return typeof b.action === "string" && typeof b.target === "string";
+}
+
+function jsonResponse(body: unknown, status = 200) {
+  return new Response(JSON.stringify(body, null, 2), {
+    status,
+    headers: { "Content-Type": "application/json" },
+  });
+}
+
+async function runDocker(args: string[], timeoutMs = 60000): Promise<{ ok: boolean; stdout: string; stderr: string }> {
+  const proc = Bun.spawn(["docker", ...args], {
+    stdout: "pipe",
+    stderr: "pipe",
+  });
+  const timeout = setTimeout(() => proc.kill("SIGTERM"), timeoutMs);
+  try {
+    const [stdoutBuf, stderrBuf] = await Promise.all([new Response(proc.stdout).arrayBuffer(), new Response(proc.stderr).arrayBuffer()]);
+    await proc.exited;
+    return {
+      ok: proc.exitCode === 0,
+      stdout: new TextDecoder().decode(stdoutBuf).trim(),
+      stderr: new TextDecoder().decode(stderrBuf).trim(),
+    };
+  } finally {
+    clearTimeout(timeout);
+  }
+}
+
+async function handleExecute(body: ExecuteRequest): Promise<Response> {
+  const { action, target } = body;
+
+  if (!ACTIONS.has(action)) {
+    log("warn", "rejected action", { action, target, reason: "action_not_allowed" });
+    return jsonResponse({ error: `Action "${action}" is not allowed. Allowed actions: ${[...ACTIONS].join(", ")}` }, 400);
+  }
+
+  if (DENYLIST.has(target)) {
+    log("warn", "rejected denylisted target", { action, target, reason: "target_denied" });
+    return jsonResponse({ error: `Target "${target}" is explicitly denied.` }, 403);
+  }
+
+  if (!ALLOWSET.has(target)) {
+    log("warn", "rejected target", { action, target, reason: "target_not_in_allowlist", allowlist: ALLOWLIST });
+    return jsonResponse({ error: `Target "${target}" is not in the executor allowlist.` }, 403);
+  }
+
+  log("info", "executing remediation", { action, target });
+
+  if (action === "restart") {
+    const { ok, stdout, stderr } = await runDocker(["restart", target]);
+    if (!ok) {
+      log("error", "docker restart failed", { target, stderr, stdout });
+      return jsonResponse({ error: `docker restart ${target} failed`, details: stderr || stdout }, 502);
+    }
+    log("info", "docker restart succeeded", { target, stdout });
+    return jsonResponse({ action, target, status: "restarted", output: stdout });
+  }
+
+  // Unreachable — action was validated above.
+  return jsonResponse({ error: "unexpected action" }, 500);
+}
+
+const server = Bun.serve({
+  port: PORT,
+  hostname: "0.0.0.0",
+  async fetch(req) {
+    const url = new URL(req.url);
+
+    if (req.method === "GET" && url.pathname === "/health") {
+      return jsonResponse({ status: "ok", allowlist: ALLOWLIST });
+    }
+
+    if (req.method !== "POST" || url.pathname !== "/execute") {
+      return jsonResponse({ error: "Not found" }, 404);
+    }
+
+    let body: unknown;
+    try {
+      body = await req.json();
+    } catch {
+      return jsonResponse({ error: "Invalid JSON" }, 400);
+    }
+
+    if (!isExecuteRequest(body)) {
+      return jsonResponse({ error: "Request must be { action: string, target: string }" }, 400);
+    }
+
+    return handleExecute(body);
+  },
+});
+
+log("info", "argus-executor listening", { port: PORT, allowlist: ALLOWLIST });
@@ -0,0 +1,6 @@
+{
+  "name": "argus-executor",
+  "private": true,
+  "version": "0.1.0",
+  "type": "module"
+}
@@ -33,7 +33,7 @@ When an alert fires you ALWAYS follow this sequence:
 6. create_incident — record your findings with likely_cause
 7. notify_channel — structured summary with root cause theory
 8. request_confirmation — propose the remediation step and STOP. The operator will approve in the UI.
-9. execute_runbook_step — call this to perform the remediation ONLY after the operator has approved. When you are told the approval was granted (you'll be handed the incident_id and confirmation_id), call execute_runbook_step with those plus the step_id, service, and command you proposed. The tool enforces the approval gate and performs the real recovery. After request_confirmation but before approval, end your turn with a brief summary of what you proposed and wait.
+9. execute_runbook_step — call this to perform the remediation ONLY after the operator has approved. When you are told the approval was granted (you'll be handed the incident_id and confirmation_id), call execute_runbook_step with those plus the service and step_id you proposed. The tool looks up the actual command in the runbook by deterministic code, enforces the approval gate, restarts the real container through the executor, and verifies the service returns to healthy. After request_confirmation but before approval, end your turn with a brief summary of what you proposed and wait.
 
 AUTONOMOUS OPERATION:
 - When you are told a new alert has fired for a service, begin investigating immediately and create the incident yourself — do NOT wait to be asked. Run the full sequence (steps 1–8) on your own, then STOP at request_confirmation for the operator's approval.

@@ -198,6 +198,16 @@ export function createDeployment(db: Database, { service_id, version, commit_sha
   return db.query("SELECT * FROM deployments WHERE id = ?").get(Number(result.lastInsertRowid)) as DeploymentRow;
 }
 
+export function getIncidentRemediationAttempts(db: Database, incidentId: number): number {
+  const row = db.query("SELECT remediation_attempts FROM incidents WHERE id = ?").get(incidentId) as { remediation_attempts: number } | null;
+  return row?.remediation_attempts ?? 0;
+}
+
+export function incrementIncidentRemediationAttempts(db: Database, incidentId: number): number {
+  db.query("UPDATE incidents SET remediation_attempts = remediation_attempts + 1, updated_at = datetime('now') WHERE id = ?").run(incidentId);
+  return getIncidentRemediationAttempts(db, incidentId);
+}
+
 // ── Notifications ──
 
 export function createNotification(db: Database, { incident_id, channel, message }: { incident_id: number; channel: string; message: string }): NotificationRow {