diff --git a/.changeset/agent-run-extension-seams.md b/.changeset/agent-run-extension-seams.md new file mode 100644 index 0000000000..6f20284296 --- /dev/null +++ b/.changeset/agent-run-extension-seams.md @@ -0,0 +1,21 @@ +--- +'@hyperdx/api': minor +'@hyperdx/app': minor +--- + +The managed-agent alert flow now exposes fail-open extension seams +(`onProvisionAgent`, `onSessionStart`, `onBeforeDelivery`, `resolveAnthropicKey`) +with a downstream-owned registration point and an `AgentRun.metadata` field, so +downstream distributions can extend investigations (e.g. notebook tracking) and +swap the agent's system and kickoff prompts wholesale without editing core +files. Claude webhooks now also honour their user-editable `body` template as +the agent's kickoff prompt (previously documented but ignored), falling back to +the built-in enriched payload for empty or broken templates. + +The Anthropic API key for managed agents is now resolved from the environment +(`AI_API_KEY` with `AI_PROVIDER=anthropic`, or the legacy `ANTHROPIC_API_KEY`), +which is the sensible default for self-hosted deployments. The per-team, +UI-managed key (previously stored encrypted in MongoDB) has been removed from +open source and is now a downstream concern, injected through the +`resolveAnthropicKey` extension seam. With no extensions registered and the +default webhook body, behaviour is otherwise unchanged. diff --git a/.changeset/claude-managed-agents-webhook.md b/.changeset/claude-managed-agents-webhook.md new file mode 100644 index 0000000000..94ad35d4af --- /dev/null +++ b/.changeset/claude-managed-agents-webhook.md @@ -0,0 +1,23 @@ +--- +'@hyperdx/common-utils': minor +'@hyperdx/api': minor +'@hyperdx/app': minor +--- + +feat: add Claude Managed Agents webhook template with enriched, agent-ready payload + +Adds a "Claude Managed Agents" webhook service type that posts an enriched, +agent-ready JSON payload (sent identically to a Generic webhook). The pre-built +body carries structured alert context (status, type, comparator, threshold, +current value, group key, source query, team id, time range) and a prompt +instructing the agent to investigate via its pre-configured ClickStack MCP +server and post a root-cause summary. No MCP URL or auth is sent in the +payload — the agent reaches ClickStack through the MCP server declared on the +agent with credentials held in a vault. + +These enriched template variables (`{{status}}`, `{{alertType}}`, +`{{comparator}}`, `{{threshold}}`, `{{value}}`, `{{groupKey}}`, +`{{sourceQuery}}`, `{{teamId}}`, `{{alertId}}`, `{{note}}`) are also available to +Generic webhook bodies for pre-agent routing and dedup. The alert's freeform +`note` is surfaced as `context.runbook` so a runbook link attached to the alert +reaches the agent. diff --git a/.changeset/managed-agents-provisioning.md b/.changeset/managed-agents-provisioning.md new file mode 100644 index 0000000000..cf463038d9 --- /dev/null +++ b/.changeset/managed-agents-provisioning.md @@ -0,0 +1,17 @@ +--- +'@hyperdx/api': minor +'@hyperdx/app': minor +--- + +feat: provision Claude Managed Agents from the HyperDX UI + +Add a Managed Agents section under Team Settings → API & Agents (gated by +`HDX_MANAGED_AGENTS_ENABLED` / `NEXT_PUBLIC_HDX_MANAGED_AGENTS_ENABLED`). The +Anthropic API key is read from the server environment (`AI_API_KEY` with +`AI_PROVIDER=anthropic`, or the legacy `ANTHROPIC_API_KEY`); per-team, +UI-managed key storage is a downstream (EE) concern injected via the +`resolveAnthropicKey` extension seam. A team can provision an opinionated +ClickStack SRE agent in one click — HyperDX creates the Anthropic environment, +vault (with the provisioning user's ClickStack access key as the MCP +credential), and agent with the ClickStack MCP server pre-configured, then +stores the references for management. diff --git a/AGENTS.md b/AGENTS.md index 17fa68b54f..66996751b7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -100,6 +100,9 @@ before stopping. them). Use the shared helpers in `packages/api/src/utils/instrumentation.ts`. See [`agent_docs/observability.md`](agent_docs/observability.md). +8. **EE extensibility**: this repo is upstream of an enterprise fork — build + extension seams, not fork-edited function bodies. See "Designing for + Downstream (EE) Extensibility" below. ## Running Tests @@ -169,6 +172,62 @@ make dev-e2e FILE=navigation REPORT=1 # Open HTML report after run make dev-e2e-clean # Remove test artifacts ``` +## Designing for Downstream (EE) Extensibility + +HyperDX has an enterprise fork (hyperdx-ee) that receives regular upstream +merges from this repo. Every inline edit EE makes to an OSS file becomes a +merge conflict on the next upstream merge — the EE "conflict resolution" +agent exists to clean those up, and we would rather not need it. Design OSS +features so EE can extend them **without editing OSS function bodies, +schemas, or test files**: + +1. **Extension seams over inline edits.** When a feature has lifecycle + points downstream will plausibly hook into (a session starting, a + delivery going out), expose a typed hook registry with no-op defaults — + see `packages/api/src/services/agentRunExtensions.ts` for the reference + pattern. Hook runners must be fail-open (an extension error never breaks + the core flow) and instrumented (a span per hook invocation plus an + outcome counter). +2. **Downstream-owned override files.** Registration happens in designated + files that upstream commits to never editing after creation — see + `packages/api/src/extensions/index.ts`. EE replaces the file's body + wholesale, so upstream merges never conflict on it. +3. **Additive, schema-free extension data.** Give downstream a + `metadata: Mixed` field on models it needs to decorate (see + `AgentRun.metadata`) instead of having it add typed fields to OSS + schemas. +4. **Options objects / optional trailing parameters** on exported functions + downstream feeds data into — adding an optional key never conflicts; + reshaping a signature does. +5. **Swappable defaults.** Operator-visible defaults downstream may want to + replace wholesale — prompts, templates, message copy — should resolve + through a hook (see `onProvisionAgent`'s `systemPrompt` and + `onSessionStart`'s `promptOverride` in `agentRunExtensions.ts`), not sit as + hardcoded constants at the call site. +6. **OSS tests never require EE edits.** Test the hook contract in OSS with + a fake extension; EE tests its own extensions in its own files. + +When building a feature EE is likely to extend, add the seam in the same PR. +Retrofitting a seam after EE has already forked the file is exactly the +conflict this section exists to prevent. + +### Seam exports and `knip` + +A seam contract (the interfaces/types EE implements against — e.g. the +`Agent*Result` types in `agentRunExtensions.ts`) is exported public API that +has **no importer inside this repo by design**: OSS ships no extensions, so +only the downstream fork consumes it. Our `knip` check (run in the pre-commit +hook and CI) would otherwise flag those exports as unused. + +The rule: mark a genuinely-downstream-only export with a JSDoc `@public` tag and +a one-line reason. `knip.json` sets `"tags": ["-public"]`, so `@public`-tagged +exports are treated as intentional public API rather than dead code. This is +scoped per-export — an export that is used nowhere at all (not even in-file, not +tagged) is still reported, so the check keeps its teeth. Do **not** reach for a +blanket `ignoreExportsUsedInFile` category toggle to silence a seam export; tag +the specific export instead. Conversely, an export that only turned out to be +unused (no EE consumer planned) should be **unexported or removed**, not tagged. + ## Important Context - **Authentication**: Passport.js with team-based access control diff --git a/docs/alert-webhook-claude-managed-agents.md b/docs/alert-webhook-claude-managed-agents.md new file mode 100644 index 0000000000..5d368600cd --- /dev/null +++ b/docs/alert-webhook-claude-managed-agents.md @@ -0,0 +1,286 @@ +# Alert Webhook → Claude Managed Agents + +Drive a [Claude Managed Agent](https://platform.claude.com/docs/en/managed-agents/overview) +from a ClickStack/HyperDX alert: when an alert fires, an agent investigates your +telemetry through the ClickStack MCP server, produces a root-cause summary, posts +it to Slack, and leaves a live session your on-call engineer can pick up. + +``` +ClickStack alert ──fires──▶ Generic webhook (Claude template) + │ + ▼ + Thin receiver (verify · dedup · start session) + │ POST /v1/sessions + ▼ + Claude Managed Agent ──MCP──▶ ClickStack /api/mcp + │ session.status_idled (outbound webhook) + ▼ + Receiver fetches result ──▶ Slack + handoff link +``` + +Claude Managed Agents has **no native inbound alert trigger**, so a small +receiver bridges the HyperDX webhook to the Managed Agents API. The receiver is +operationally critical (it fires on real incidents) but small: verify, dedup, +route, start a session. + +--- + +## 1. The webhook payload HyperDX sends + +In HyperDX, create a webhook under **Team Settings → Webhooks**, choose the +**Claude Managed Agents** service type, and point the URL at your receiver. The +body is pre-filled with an enriched, agent-ready JSON payload: + +> **The payload carries no MCP URL or auth.** The agent reaches ClickStack +> through the MCP server pre-configured on the agent (§2), with the Bearer token +> held in a vault (§3) and injected by Anthropic outside the sandbox. Nothing +> secret is ever sent over the wire in the webhook. + +```json +{ + "source": "clickstack", + "schema_version": "1", + "prompt": "A ClickStack alert fired. Investigate the root cause using your pre-configured ClickStack MCP server (logs, traces, metrics, and alert history). Reconstruct and re-run the alert source_query over the time_range, inspect related logs, traces, and metrics, follow the runbook in context.runbook if present, check recent deploys, then post a structured root-cause summary to Slack and leave the session open for the on-call engineer to continue.", + "alert": { + "id": "{{alertId}}", + "event_id": "{{eventId}}", + "status": "{{status}}", + "type": "{{alertType}}", + "title": "{{title}}", + "body": "{{body}}", + "link": "{{link}}" + }, + "condition": { + "comparator": "{{comparator}}", + "threshold": "{{threshold}}", + "current_value": "{{value}}" + }, + "context": { + "group_key": "{{groupKey}}", + "source_query": "{{sourceQuery}}", + "runbook": "{{note}}", + "team_id": "{{teamId}}", + "time_range": { "start": "{{startTime}}", "end": "{{endTime}}" } + } +} +``` + +The body is a [Handlebars](https://handlebarsjs.com/) template. You can edit it +freely; the variables below are substituted at send time. + +### Template variables + +| Variable | Example | Notes | +| ----------------- | ----------------------------- | ----- | +| `{{alertId}}` | `663f…` | Stable alert id. | +| `{{eventId}}` | `a1b2…` | Per-firing dedup hash (alert + channel + group). Use as an idempotency key. | +| `{{status}}` | `firing` \| `resolved` \| `no_data` | Mapped from alert state; enables suppress-on-resolve and no-data handling. | +| `{{alertType}}` | `search` \| `dashboard_chart` | Mapped from the alert source. | +| `{{title}}` | `🚨 Alert for "5xx rate"…` | | +| `{{body}}` | rendered alert body | Includes sample log lines for saved-search alerts. | +| `{{link}}` | `https://hdx…/search/…` | Deep link into HyperDX. | +| `{{comparator}}` | `>=` `>` `<=` `<` `=` `!=` `between` `outside` | Threshold comparator. | +| `{{threshold}}` | `5` | Numeric; quoted in the default body to keep JSON valid. | +| `{{value}}` | `42` | The value that triggered/resolved the alert. | +| `{{groupKey}}` | `checkout-service` | Grouped-by value, when present. | +| `{{sourceQuery}}` | `Body: "error"` | The search expression / SQL defining the alert. JSON-escaped. | +| `{{note}}` | `Runbook: https://wiki/…` | The alert's freeform note — use it to attach a runbook link. Surfaced as `context.runbook`. | +| `{{teamId}}` | `663f…` | Owning team. | +| `{{startTime}}` / `{{endTime}}` | epoch ms | Evaluation window — scope MCP queries to it. | +| `{{state}}` | `ALERT` \| `OK` \| `INSUFFICIENT_DATA` | Raw internal state (prefer `{{status}}`). | + +> These variables are also available to the plain **Generic** webhook, so you can +> route/dedup by `service`/`severity`/`status` before spending agent tokens. + +### Security + +Add a shared-secret header in the webhook's **Headers** field and verify it in +the receiver. (A first-class signed-webhook — HMAC over the body with a per- +webhook secret + timestamp — is on the roadmap; until then, use a header secret +over HTTPS.) + +--- + +## 2. Create the agent (once) + +Declare the ClickStack MCP server at `/api/mcp`. This URL +lives only here and in the vault credential (§3) — it must be byte-identical in +both, and it is never sent in the webhook payload. + +```bash +curl -sS https://api.anthropic.com/v1/agents \ + -H "x-api-key: $ANTHROPIC_API_KEY" \ + -H "anthropic-version: 2023-06-01" \ + -H "anthropic-beta: managed-agents-2026-04-01" \ + -H "content-type: application/json" \ + -d '{ + "name": "ClickStack SRE Responder", + "model": "claude-opus-4-8", + "system": "You are an SRE agent. A ClickStack alert fired. Investigate via the clickstack MCP server (logs/traces/metrics + alert history), reconstruct and re-run the alert source_query over the time_range, check recent deploys, then post a structured root-cause summary to Slack.", + "mcp_servers": [ + { "type": "url", "name": "clickstack", "url": "https:///api/mcp" } + ], + "tools": [ + { "type": "agent_toolset_20260401" }, + { "type": "mcp_toolset", "mcp_server_name": "clickstack" } + ] + }' +``` + +Save the returned `agent_id`. + +**Tool permissions** default to `always_ask`. For an unattended loop, auto-allow +the read tools via `default_config`/`configs` on the `mcp_toolset`, and keep +write/remediation tools disabled until precision is validated (read-only first). +See [permission policies](https://platform.claude.com/docs/en/managed-agents/permission-policies). + +Create an environment once and save its `environment_id`: + +```bash +curl -sS https://api.anthropic.com/v1/environments \ + -H "x-api-key: $ANTHROPIC_API_KEY" \ + -H "anthropic-version: 2023-06-01" \ + -H "anthropic-beta: managed-agents-2026-04-01" \ + -H "content-type: application/json" \ + -d '{ "name": "sre-sandbox", "config": { "type": "cloud", "networking": { "type": "unrestricted" } } }' +``` + +--- + +## 3. Store secrets in a vault (once) + +The model never sees the token — Anthropic's credential proxy injects it by +matching `mcp_server_url` to the agent's MCP server `url`. **Same exact URL.** + +```bash +# create the vault → save vault_id +curl -sS https://api.anthropic.com/v1/vaults \ + -H "x-api-key: $ANTHROPIC_API_KEY" \ + -H "anthropic-version: 2023-06-01" \ + -H "anthropic-beta: managed-agents-2026-04-01" \ + -H "content-type: application/json" \ + -d '{ "display_name": "ClickStack SRE Credentials" }' + +# add the ClickStack Personal API Access Key (Team Settings → API Keys) +curl -sS https://api.anthropic.com/v1/vaults/$VAULT_ID/credentials \ + -H "x-api-key: $ANTHROPIC_API_KEY" \ + -H "anthropic-version: 2023-06-01" \ + -H "anthropic-beta: managed-agents-2026-04-01" \ + -H "content-type: application/json" \ + -d '{ + "display_name": "ClickStack Personal API Access Key", + "auth": { + "type": "static_bearer", + "mcp_server_url": "https:///api/mcp", + "token": "" + } + }' +``` + +Add Slack / GitHub credentials to the same vault as needed (least privilege: +read-only ClickStack, Slack write scoped to one channel). + +--- + +## 4. The receiver (per alert) + +A small HTTPS service: verify the shared-secret header, dedup on +`alert.event_id` within the firing window, then start a session and hand it the +payload as the prompt. + +```js +// 1. verify header secret, 2. dedup on payload.alert.event_id, then: +const headers = { + 'x-api-key': process.env.ANTHROPIC_API_KEY, + 'anthropic-version': '2023-06-01', + 'anthropic-beta': 'managed-agents-2026-04-01', + 'content-type': 'application/json', +}; + +const session = await fetch('https://api.anthropic.com/v1/sessions', { + method: 'POST', headers, + body: JSON.stringify({ + agent: AGENT_ID, + environment_id: ENV_ID, + vault_ids: [VAULT_ID], + title: payload.alert.title, + }), +}).then(r => r.json()); + +await fetch(`https://api.anthropic.com/v1/sessions/${session.id}/events`, { + method: 'POST', headers, + body: JSON.stringify({ + events: [{ type: 'user.message', content: [{ type: 'text', text: JSON.stringify(payload) }] }], + }), +}); +``` + +The enriched body is already agent-ready — `condition` and `context` tell the +agent what to query without a round-trip. It reaches ClickStack through its +pre-configured MCP server, so the payload never carries an MCP URL or token. + +--- + +## 5. Return the result to Slack + +Register an outbound webhook in the Claude console (**Manage → Webhooks**) for +`session.status_idled`. On receipt, verify the signature (HMAC-SHA256 over the +raw body) and fetch the result: + +```python +import anthropic +client = anthropic.Anthropic() # reads ANTHROPIC_WEBHOOK_SIGNING_KEY + +event = client.beta.webhooks.unwrap(request_body, headers) # raises on bad sig / >5 min old +if event.data.type == "session.status_idled": + events = client.beta.sessions.events.list(event.data.id) + summary = next((e.content[0].text for e in events if e.type == "agent.message"), "") + # post `summary` to Slack; include a deep link to the session for handoff +``` + +### Suggested Slack format + +``` + () +Fired: