diff --git a/CHANGELOG.md b/CHANGELOG.md index f3075abf1..a15926a41 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -25,6 +25,20 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). ### Changed +- `specs/010-security` and `specs/008-mcp`: rewrote all seven sub-specs + (`010-1-vault`, `010-2-injection-defense`, `010-3-authorization`, `010-4-audit`, + `008-1-lifecycle`, `008-2-discovery`, `008-3-security`) to describe the actual + shipped implementation in `zeph-vault`, `zeph-sanitizer`, `zeph-tools`, and + `zeph-mcp` instead of invented type/API names flagged by the PR #6629 spec + audit (issues #6631, #6630). `010-4-audit`'s NEVER-rule contradiction was + resolved by clarifying that `CrossToolCorrelator` (within-turn, hard + `InjectionConfirmed` decisions) and `TrajectorySentinel` (advisory, decaying + cross-turn score) are independent mechanisms per the parent spec's scope + note; a genuinely unresolved gap around `TrajectoryRiskAccumulator` making + cross-turn hard-blocking decisions not covered by that carve-out is flagged + with an explicit `[!warning] Architectural gap` callout rather than papered + over. All `[!danger]` callouts added by PR #6629 are removed. Documentation-only, + no code changes. - `specs/031-database-abstraction`: split the 2958-line spec (issue #6632). Section 18 ("Agent Identity in the Shared Data Model") — a fully self-contained multi-tenant data-isolation subsystem with no other cross-references in `/specs/` — was extracted diff --git a/specs/008-mcp/008-1-lifecycle.md b/specs/008-mcp/008-1-lifecycle.md index 6a74772a8..ad89c0bfe 100644 --- a/specs/008-mcp/008-1-lifecycle.md +++ b/specs/008-mcp/008-1-lifecycle.md @@ -19,268 +19,181 @@ related: # Spec: MCP Server Lifecycle -> [!danger] Stale content — 2026-07 audit -> The "Startup Auto-Retry with Exponential Backoff (#3578)" section and most of the pseudocode below -> (`McpConnection`, `JsonRpcTransport`, `Waiter`, `log::warn!`/`log::info!` calls) do not match the -> real implementation and were not corrected in this pass (rewrite, not a text fix). Real code: -> `crates/zeph-mcp/src/manager/{mod.rs,builder.rs,connect.rs,server.rs,retry.rs}`. Real retry config -> fields are `startup_retry_backoff_ms`/`tool_timeout_secs` (see `[[008-mcp/spec]]`), not -> `startup_retry_base_delay_ms`/`startup_retry_max_delay_ms`/`startup_retry_backoff_factor`/ -> `max_startup_retries` as stated below. `log::*` macros are banned project-wide (`tracing` only, -> see `[[constitution]]` II/IV) — do not copy the logging calls in this file's pseudocode. - Server startup/shutdown, connection management, stdio environment isolation, graceful cleanup. ## Overview -MCP servers are subprocess-based tool providers. Zeph manages their complete lifecycle: spawning with environment isolation, maintaining connections, detecting failures, and graceful shutdown. +MCP servers are subprocess-based or HTTP tool providers. Zeph manages their complete lifecycle: spawning with environment isolation, maintaining connections, detecting failures, and graceful shutdown. The `McpManager` (`crates/zeph-mcp/src/manager/`) orchestrates multi-server lifecycle. ## Key Invariants **Always:** -- Each MCP server runs in isolated subprocess with env vars scrubbed -- Server startup failures logged with full context (stderr, exit code, timeout) -- Connections are bidirectional: Zeph sends requests, servers send notifications -- Server shutdown waits for pending requests (configurable timeout: default 5s) +- Each stdio MCP server runs in isolated subprocess with `ZEPH_*` secrets scrubbed from environment +- Server startup failures logged with stderr/exit code +- Server connections are bidirectional: Zeph sends requests, servers send notifications (`tools/list_changed`, etc.) +- Server shutdown waits for pending requests (timeout: configurable via `shutdown_timeout_secs`) +- Env var scrubbing happens unconditionally before subprocess spawn **Never:** -- Pass secrets (API keys, tokens) to MCP server environment -- Leave zombie processes on exit (always await subprocess termination) -- Assume server is healthy without heartbeat validation +- Pass `ZEPH_*` secrets to MCP server environment — they are already scrubbed +- Leave zombie processes on shutdown — always collect process status +- Assume server is healthy without probing — initial `tools/list` serves as health check +- Send requests to a server while it is shutting down ## Startup Sequence ``` -1. Resolve server config (name, command, env, stdio mode) -2. Scrub environment: remove ZEPH_* secrets, keep only safe vars -3. Spawn subprocess with stdio isolation -4. Send initialize request, await response (timeout: 10s) -5. Register tool registry, store connection metadata -6. Mark server as "ready" in MCP client state +1. Validate transport config (stdio command, HTTP URL, or OAuth credentials) +2. Scrub environment: remove ZEPH_*, AWS_*, GITHUB_TOKEN, etc. +3. Spawn subprocess (stdio) or establish HTTP connection (http/oauth) +4. Send initialize request via rmcp crate, await response (timeout: 10s) +5. Fetch tools/list via DefaultMcpProber for pre-connect injection scan +6. Sanitize tool definitions (names, descriptions, input schemas) +7. Register tools in `McpToolRegistry` with trust scores +8. Mark server as "connected" in manager state ``` -Code sketch: +API: ```rust -async fn start_server(&self, config: &ServerConfig) -> Result { - // 1. Sanitize environment - let env = self.scrub_secrets(&config.env); - - // 2. Spawn process - let mut child = Command::new(&config.command) - .env_clear() - .envs(&env) - .stdin(Stdio::piped()) - .stdout(Stdio::piped()) - .stderr(Stdio::piped()) - .spawn() - .context("Failed to spawn MCP server")?; - - let stdin = child.stdin.take().unwrap(); - let stdout = child.stdout.take().unwrap(); - - // 3. Initialize protocol (exchange capabilities) - let conn = JsonRpcTransport::new(stdin, stdout); - let caps = conn.initialize( - &InitializeRequest { - protocol_version: "2024-11-05", - capabilities: /* client capabilities */, - }, - Duration::from_secs(10), - ).await?; - - // 4. Store connection - let server_id = self.store_connection(config.name.clone(), conn).await?; - - Ok(server_id) +pub struct ServerEntry { + pub id: String, // Unique server ID + pub transport: McpTransport, // Stdio/Http/OAuth variant + pub timeout: Duration, // Request timeout + pub trust_level: McpTrustLevel, // Trusted/Untrusted/Sandboxed + pub tool_allowlist: Option>, // Optional tool allowlist + pub allow_untrusted_without_allowlist: bool, // Secure default: false + pub expected_tools: Vec, // Attestation list + pub roots: Vec, // Root resources for file servers + pub tool_metadata: HashMap, // Pre-defined tool metadata + pub elicitation_enabled: bool, // Tool parameter probing + pub elicitation_timeout_secs: u64, // Probing timeout + pub env_isolation: bool, // Scrub env vars (default: true) + pub media_passthrough: bool, // Allow media types in responses } -``` -## Connection Management - -Maintain bidirectional messaging: - -```rust -pub struct McpConnection { - id: String, - name: String, - transport: JsonRpcTransport, - pending_requests: Arc>>, - server_notifications: Arc>>, - health_check_interval: Duration, +pub enum McpTransport { + Stdio { + command: String, + args: Vec, + env: HashMap, + }, + Http { + url: String, + headers: HashMap, + }, + OAuth { + auth_uri: String, + token_uri: String, + client_id: String, + client_secret: Secret, + }, } -impl McpConnection { - async fn send_request(&self, method: &str, params: Value) -> Result { - let id = self.next_request_id(); - let waiter = Waiter::new(); - - self.pending_requests.lock().insert(id, waiter.clone()); - self.transport.send_request(id, method, params).await?; - - // Wait for response with timeout - waiter.wait(Duration::from_secs(30)).await - } +impl McpManager { + /// Connect all configured servers in parallel. + pub async fn connect_all(&self) -> (Vec, Vec); - async fn handle_notification(&self, notif: Notification) { - // Server can send unsolicited notifications (e.g., resource changed) - self.server_notifications.lock().push_back(notif); - } + /// Call a tool on a specific server. + pub async fn call_tool( + &self, + server_id: &str, + tool_name: &str, + arguments: serde_json::Value, + ) -> Result; } ``` -## Failure Detection & Reconnection +## Connection Maintenance -Monitor server health: +`McpClient` (via rmcp crate) maintains bidirectional messaging: -```rust -async fn health_check_loop(&self, conn: &McpConnection) { - let mut interval = tokio::time::interval( - conn.health_check_interval - ); - - loop { - interval.tick().await; - - match conn.send_request("ping", json!({})).await { - Ok(_) => { - conn.mark_healthy(); - } - Err(e) => { - log::warn!("Server {} health check failed: {}", conn.name, e); - self.mark_unhealthy(&conn.id).await; - - // Trigger reconnect logic - self.attempt_reconnect(&conn.id).await; - } - } - } -} -``` - -## Graceful Shutdown +- **Requests**: Zeph sends tool calls, resource reads, prompt queries +- **Notifications**: Server sends `tools/list_changed`, `resources/list_changed`, etc. +- **Responses**: Async correlation by request ID; timeout triggers error +- **Errors**: JSON-RPC errors forwarded as `McpError` -Cleanup on agent termination: +**Configuration example** (`[[mcp.servers]]`): -```rust -async fn shutdown_server(&self, server_id: &str) -> Result<()> { - let conn = self.get_connection(server_id)?; - - // 1. Reject new requests - conn.mark_shutdown(); - - // 2. Wait for pending requests (timeout: 5s) - let timeout = Duration::from_secs(5); - match timeout_at( - Instant::now() + timeout, - self.wait_pending_requests(server_id), - ).await { - Ok(Ok(())) => log::info!("Server {} graceful shutdown", server_id), - Ok(Err(e)) => log::warn!("Server {} shutdown error: {}", server_id, e), - Err(_) => log::warn!("Server {} shutdown timeout", server_id), - } - - // 3. Terminate subprocess - self.kill_subprocess(server_id).await?; - - // 4. Remove from registry - self.remove_connection(server_id).await; - - Ok(()) -} +For a Stdio transport (subprocess): +```toml +[[mcp.servers]] +id = "filesystem" +command = "npx" +args = ["-y", "@modelcontextprotocol/server-filesystem"] +# env = { CUSTOM_VAR = "value" } # Optional: env vars for the subprocess +timeout = 30 # Timeout in seconds for requests +trust_level = "untrusted" # "trusted", "untrusted", or "sandboxed" +tool_allowlist = ["read_file", "list_directory"] ``` -## Startup Auto-Retry with Exponential Backoff (#3578) - -MCP server startup is unreliable in practice: a server process may crash before -completing the `initialize` handshake, or a network MCP server may be temporarily -unavailable at agent start time. Without retry, a single failed server blocks agent -startup or silently reduces the tool catalog. - -### Retry Contract - -`McpManager::start_with_retry(config)` wraps `start_server()` in an exponential -backoff loop: - -``` -attempt 1: immediate -attempt 2: base_delay_ms (default 200 ms) -attempt 3: base_delay_ms × backoff_factor (default 2.0) -... -attempt N: min(base_delay_ms × backoff_factor^(N-2), max_delay_ms) +For HTTP transport (remote service): +```toml +[[mcp.servers]] +id = "remote-api" +url = "http://localhost:8000/mcp" +headers = { Authorization = "Bearer ${VAULT_API_TOKEN}" } # Supports vault refs +timeout = 60 +trust_level = "untrusted" ``` -On exhaustion (all `max_startup_retries` attempts failed): +**Field reference:** +- `id`: Unique server identifier +- `command` (Stdio): executable to spawn +- `args` (Stdio): command-line arguments +- `url` (HTTP): remote server URL +- `env` (Stdio): environment variables (supports `${VAULT_KEY}` references) +- `headers` (HTTP): static HTTP headers +- `timeout`: request timeout in seconds +- `trust_level`: `"trusted"` | `"untrusted"` | `"sandboxed"` +- `tool_allowlist`: optional array of tool names to expose +- `expected_tools`: optional array for attestation/schema-drift detection +- `media_passthrough`: enable image attachment (default: false) +- `env_isolation`: isolate subprocess environment (default: false) -- **`critical = false` servers**: log `ERROR`, skip server, agent starts without it. - The missing server's tools are absent from the catalog until a `/mcp reconnect` command. -- **`critical = true` servers**: return `Err(McpError::CriticalServerStartFailed)`, - aborting agent startup. - -### Jitter +## Failure Detection & Reconnection -Each backoff delay is jittered by `±25%` (uniform random) to prevent thundering herds -when multiple MCP servers restart simultaneously after a crash. +Initial connection failures are logged with stderr; transient failures during tool calls trigger retry logic (via rmcp's error handling and `McpToolExecutor` retry). Long-lived servers are not actively health-checked (only re-validated if a notification arrives). -### Tracing +Failed servers are marked `Error` in `ServerConnectOutcome` and their tools are unavailable for dispatch. -Each retry attempt emits a `tracing::warn!` with attempt number, server name, and -error. The initial failure emits `tracing::info!` (not warn — first attempt failure is -expected in slow-start environments). +## Graceful Shutdown -### Config +Cleanup on agent termination — `McpManager` is dropped after the agent loop exits: -```toml -[[mcp.servers]] -name = "local-tools" -command = "python3 /path/to/server.py" -stdio = "pipe" # or "pty" for terminal emulation -timeout_init_s = 10 -timeout_request_s = 30 -healthcheck_interval_s = 60 -critical = false # if true, startup failure aborts the agent -max_startup_retries = 3 # total attempts (1 initial + N-1 retries); 0 = no retry -startup_retry_base_delay_ms = 200 # base delay before first retry -startup_retry_max_delay_ms = 5000 # cap on exponential backoff -startup_retry_backoff_factor = 2.0 # multiplier applied per attempt - -# Environment scrubbing: keep only these vars -allow_env_vars = ["PATH", "HOME", "RUST_LOG"] +```rust +impl McpManager { + /// Shutdown all servers gracefully (by-value self, no timeout param). + /// + /// Consumes the manager; servers are shut down as part of the Drop impl. + pub async fn shutdown_all(self) { + // Cancel pending requests for all servers + // Send shutdown signal if server supports it + // Close stdio handles or HTTP connections + // Wait for graceful close or force-terminate + } +} ``` -### Key Invariants - -- Retry delay is bounded by `startup_retry_max_delay_ms` — backoff cannot grow unbounded -- `critical = true` servers abort startup on first failure (no retry is attempted before aborting) - — override: set `max_startup_retries > 0` to retry even critical servers before aborting -- NEVER silently swallow a critical server failure — `Err` must propagate to `McpManager::start_all` -- Jitter is applied on retries only, not on the initial attempt -- The TUI startup spinner shows per-server retry status when `max_startup_retries > 0` +**Timeout behavior**: shutdown is not time-bounded in the public API. The agent loop exit triggers a shutdown cascade; slow servers do not block agent termination since the manager is dropped. -## Configuration (Legacy) +## Tool Registry Coordination -```toml -[[mcp.servers]] -name = "local-tools" -command = "python3 /path/to/server.py" -stdio = "pipe" # or "pty" for terminal emulation -timeout_init_s = 10 -timeout_request_s = 30 -healthcheck_interval_s = 60 - -# Environment scrubbing: keep only these vars -allow_env_vars = ["PATH", "HOME", "RUST_LOG"] -``` +`McpToolRegistry` (`crates/zeph-mcp/src/registry.rs`) indexes tools by server. When a server sends the `tools/list_changed` notification, the manager re-fetches and updates the registry: +- Fetch updated `tools/list` from the server +- Sanitize tool definitions (scrub injection patterns) +- Update the registry index +- Invalidate any per-message tool pruning caches (since tool set changed) -## Integration Points +## Transport Security -- [[002-agent-loop/spec]] — MCP servers initialized during agent startup -- [[008-2-discovery]] — Server capabilities discovered after initialization -- [[008-3-security]] — Environment scrubbing occurs before spawn -- [[010-security/spec]] — Subprocess isolation enforced here +- **Stdio**: inherited from parent environment (scrubbed of secrets via `env_blocklist`); runs with same UID as agent +- **HTTP**: URL resolved via `validate_url()` (crates/zeph-tools/src/net.rs) — private IPs and non-HTTPS schemes blocked +- **OAuth**: authorization endpoints validated against SSRF blocklist ## See Also - [[008-mcp/spec]] — Parent -- [[008-2-discovery]] — Tool discovery after lifecycle setup -- [[008-3-security]] — Security constraints on subprocess spawning +- [[008-2-discovery]] — Tool discovery and semantic indexing +- [[008-3-security]] — Tool sanitization and policy enforcement +- [[010-3-authorization]] — SSRF protection diff --git a/specs/008-mcp/008-2-discovery.md b/specs/008-mcp/008-2-discovery.md index d5202d441..e32551641 100644 --- a/specs/008-mcp/008-2-discovery.md +++ b/specs/008-mcp/008-2-discovery.md @@ -21,220 +21,216 @@ related: # Spec: MCP Tool Discovery & Pruning -> [!danger] Stale content — 2026-07 audit -> The "Per-Message Tool Pruning" (`ToolPruningCache`) and "Tool Embedding Precomputation" sections -> below do not match the real implementation and were not corrected in this pass (rewrite, not a -> text fix). Real pruning cache is `PruningCache` (`crates/zeph-mcp/src/pruning.rs`) — a single-slot -> cache keyed on `(message_content_hash, tool_list_hash)` that selects tools via an **LLM call**, not -> embedding/dot-product ranking as described below. Real collision detection is -> `detect_collisions::` / `ToolCollision` (`crates/zeph-mcp/src/tool.rs`) plus -> `log_tool_collisions` (`crates/zeph-mcp/src/manager/connect.rs`). `log::warn!` calls in the -> pseudocode below are banned project-wide (`tracing` only, see `[[constitution]]` II/IV). -> The `[mcp.discovery]` config block (`pruning_enabled`/`top_k_tools`/`cache_size`/`collision_mode`) -> is unverified and likely stale given the cache design mismatch above. - Semantic tool discovery, per-message pruning cache, collision detection, tool filtering. ## Overview -MCP servers expose hundreds of tools across multiple categories. Zeph discovers these at startup and applies semantic filtering per-message to reduce token overhead and prevent tool confusion. +MCP servers expose hundreds of tools across multiple categories. Zeph discovers these at startup via `tools/list`, applies sanitization, and selects relevant tools per-message to reduce token overhead and prevent tool confusion. ## Key Invariants **Always:** - All server tools registered with full schema (name, description, input_schema) -- Tool collision detection triggers on registration: same name from multiple servers -- Per-message tool set pruned based on context relevance (embedding similarity) -- Tool discovery errors are non-fatal: server remains active even if tool listing fails +- Tool collision detection triggers on registration: log and disambiguate same names from multiple servers +- Per-message tool set pruned via LLM call to reduce token overhead +- Tool list cache invalidated on server reconnection +- Pruning cache is single-slot: keyed on `(message_hash, tool_list_hash)` **Never:** -- Include duplicate tool names (same name from multiple servers) without disambiguation -- Pass full tool registry to LLM (always apply semantic pruning) -- Cache tool descriptions without versioning (invalidate on server restart) +- Pass full tool registry to LLM without semantic pruning +- Serve tools from collided names without disambiguation (e.g., "tool_x" from two servers ambiguous) +- Cache tool definitions after server restart without re-fetching ## Tool Registration & Collision Detection -At server startup, fetch and register tools: +At server startup, fetch and register tools via `tools/list` and scan for name collisions: ```rust -async fn discover_tools(&self, server: &McpServer) -> Result> { - // 1. Request tool list from server - let tools = server.connection.list_tools().await?; - - // 2. Validate and enrich schemas - let mut discovered = Vec::new(); - for tool in tools { - match validate_json_schema(&tool.input_schema) { - Ok(schema) => { - let enriched = ToolDefinition { - id: format!("{}:{}", server.name, tool.name), - name: tool.name.clone(), - description: tool.description, - input_schema: schema, - server_id: server.id.clone(), - }; - discovered.push(enriched); - } - Err(e) => { - log::warn!("Invalid tool schema for {}.{}: {}", server.name, tool.name, e); - // Continue; don't fail entire discovery - } - } - } - - // 3. Detect collisions - self.detect_collisions(&discovered)?; - - Ok(discovered) +pub struct ToolCollision { + pub sanitized_id: String, // Collision identifier + pub server_a: String, // First server ID + pub qualified_a: String, // Qualified name: "server_a:tool_name" + pub trust_a: McpTrustLevel, // Trust level of first server + pub server_b: String, // Second server ID + pub qualified_b: String, // Qualified name: "server_b:tool_name" + pub trust_b: McpTrustLevel, // Trust level of second server } -fn detect_collisions(&self, tools: &[ToolDefinition]) -> Result<()> { - let mut by_name: HashMap<&str, Vec<&ToolDefinition>> = HashMap::new(); - - for tool in tools { - by_name.entry(&tool.name) - .or_insert_with(Vec::new) - .push(tool); - } - - for (name, defs) in by_name { - if defs.len() > 1 { - log::warn!( - "Tool name collision: '{}' defined in {}", - name, - defs.iter() - .map(|d| d.server_id.as_str()) - .collect::>() - .join(", ") - ); - - // Auto-rename to disambiguate: "tool_name" → "server1:tool_name" - // (Handled at invocation time) - } - } - - Ok(()) +/// Detect tool name collisions across all registered tools. +/// +/// The `trust_map` provides trust levels for each server (missing servers default to `Untrusted`). +pub fn detect_collisions( + tools: &[McpTool], + trust_map: &HashMap, +) -> Vec { + // Scan all tools for matching sanitized IDs + // Report each collision with both servers' trust levels + // Used to assess risk of ambiguous tool dispatch } ``` -## Per-Message Tool Pruning +Collision handling: +- Collisions are logged at registration but do NOT block tool loading +- Disambiguation happens at tool invocation time via server prefix (e.g., `filesystem:read_file` vs `http:read_file`) +- Both `trust_a` and `trust_b` are recorded to assess data-flow risk via policy enforcement -Semantic filtering reduces LLM token overhead: +## Per-Message Tool Pruning Cache + +`PruningCache` (`crates/zeph-mcp/src/pruning.rs`) caches LLM-selected relevant tools per turn: ```rust -pub struct ToolPruningCache { - // Query embedding → recommended tool IDs (cached) - cache: Arc>>>, - embedding_provider: Arc, - tool_embeddings: Arc>>, // cached at startup - top_k: usize, // default: 10 +pub struct PruningCache { + // Fields are private; external interface is only new() and reset() } -impl ToolPruningCache { - async fn prune_tools_for_context( - &self, - context: &str, // user query + recent history - all_tools: &[ToolDefinition], - ) -> Result> { - // 1. Generate query embedding - let query_embedding = self.embedding_provider - .embed_text(context) - .await?; - - // 2. Check cache - let cache_key = format!("{:?}", &query_embedding[..10]); // simple hash - if let Some(cached) = self.cache.lock().get(&cache_key) { - let pruned: Vec<_> = cached.iter() - .filter_map(|id| all_tools.iter().find(|t| &t.id == id)) - .collect(); - return Ok(pruned); - } - - // 3. Semantic similarity: dot product with tool embeddings - let mut scores: Vec<(usize, f32)> = all_tools - .iter() - .enumerate() - .map(|(i, tool)| { - let embedding = self.tool_embeddings.get(&tool.id) - .map(|e| dot_product(&query_embedding, e)) - .unwrap_or(0.0); - (i, embedding) - }) - .collect(); - - // 4. Select top-K by score - scores.sort_by(|a, b| b.1.partial_cmp(&a.1).unwrap()); - let selected: Vec = scores - .into_iter() - .take(self.top_k) - .map(|(i, _)| all_tools[i].id.clone()) - .collect(); - - // 5. Cache result - self.cache.lock().put(cache_key, selected.clone()); - - let pruned: Vec<_> = selected.iter() - .filter_map(|id| all_tools.iter().find(|t| &t.id == id)) - .collect(); - - Ok(pruned) - } +pub struct PruningParams { + /// Maximum number of MCP tools to include after pruning + pub max_tools: usize, + /// Minimum number of tools below which pruning is skipped (use all) + pub min_tools_to_prune: usize, + /// Tool names that are always included regardless of relevance ranking + pub always_include: Vec, +} + +/// Prune tools using LLM-based ranking (single-slot cache). +/// +/// Call signature varies by discovery strategy; for LLM pruning: +pub async fn prune_tools_cached( + cache: &mut PruningCache, + all_tools: &[McpTool], + task_context: &str, + params: &PruningParams, + provider: &P, +) -> Result, PruningError> { + // Single-slot cache: check (message_hash, tool_list_hash) key + // On miss: call LLM "which of these tools are relevant to: {task_context}?" + // Reduces token overhead: ~100 tools down to ~10 relevant ones } ``` -## Tool Embedding Precomputation +Cache semantics: +- **Single-slot**: only one `(message_hash, tool_list_hash)` pair cached at a time +- **Keyed on content+catalog**: if either changes, cache invalidates +- **LLM-populated**: invokes a model-provider to rank tools by relevance +- **Always-include pinned**: `always_include` tool names bypass relevance filtering + +**Configuration**: +```toml +[mcp.pruning] +enabled = false # Enable per-message tool pruning (default: false) +max_tools = 15 # Max tools returned after pruning +min_tools_to_prune = 10 # Skip pruning if fewer than this many available +pruning_provider = "" # LLM provider for pruning (empty = use default) +# always_include = ["critical_tool"] # Tools always present + +[mcp.tool_discovery] +strategy = "none" # "none" (default, no pruning), "llm" (LLM ranking), "embedding" (semantic) +top_k = 10 # Number of top tools to include per query (embedding strategy only) +min_similarity = 0.2 # Minimum cosine similarity threshold (embedding strategy only) +embedding_provider = "" # LLM provider for embeddings (empty = use default) +``` -Cache at startup for fast lookup: +## Semantic Tool Indexing (Optional) + +`SemanticToolIndex` (`crates/zeph-mcp/src/semantic_index.rs`) provides embedding-based ranking as an alternative to LLM-based pruning: ```rust -async fn precompute_tool_embeddings( - tools: &[ToolDefinition], - provider: &LlmProvider, -) -> Result>> { - let mut embeddings = HashMap::new(); - - // Embed tool description + input schema summary - for tool in tools { - let text = format!( - "Tool: {}\nDescription: {}\nInputs: {:?}", - tool.name, - tool.description, - tool.input_schema.properties.keys().collect::>(), - ); - - let embedding = provider.embed_text(&text).await?; - embeddings.insert(tool.id.clone(), embedding); +pub enum ToolDiscoveryStrategy { + /// No pruning; all tools provided to the LLM + None, + /// Ask LLM which tools are relevant (via pruning_provider) + Llm, + /// Embedding-based semantic ranking (requires embedding model) + Embedding, +} + +pub struct SemanticToolIndex { /* fields private */ } + +impl SemanticToolIndex { + /// Build an embedding-based tool index. + pub async fn build( + tools: &[McpTool], + embed_fn: &F, + ) -> Result + where + F: Fn(&str) -> zeph_llm::provider::EmbedFuture + Send + Sync, + { + // Compute and cache embeddings for all tool descriptions + } + + /// Select relevant tools by semantic similarity to a query embedding. + pub fn select( + &self, + query_embedding: &[f32], + top_k: usize, + min_similarity: f32, + always_include: &[String], // Tool names always included + ) -> Vec { + // Rank by cosine similarity; return top-k + always_include } - - Ok(embeddings) } ``` -## Configuration +**Embeddings** are computed once at startup and cached in memory. Query embedding is computed on-demand from the task context. -```toml -[mcp.discovery] -enabled = true +## Tool Attestation + +When `expected_tools` list is declared in server config, the registry validates against schema drift: + +```rust +pub type ToolFingerprint = String; // Blake3 hex digest + +#[non_exhaustive] +pub enum AttestationResult { + /// All actual tools match the operator-declared expected set. + Verified { + fingerprints: HashMap, + }, + /// Server returned tools not in `expected_tools`. + Unexpected { + unexpected_tools: Vec, + fingerprints: HashMap, + }, + /// No `expected_tools` declared — attestation skipped. + Unconfigured, +} -# Per-message pruning -pruning_enabled = true -top_k_tools = 10 # keep top 10 most relevant -cache_size = 1000 # LRU cache entries +/// Attestation at tool registration time (sync, no Result). +pub fn attest_tools( + tools: &[McpTool], + expected_tools: &[String], + previous_fingerprints: Option<&HashMap>, +) -> AttestationResult { + // Compare actual tool set against expected_tools + // Compute fingerprints; detect schema drift from previous session +} -# Collision handling -collision_mode = "disambiguate" # or "error", "first" +/// Compute Blake3 fingerprint of a tool's schema (name + description + input_schema) +pub fn fingerprint_tool(tool: &McpTool) -> ToolFingerprint { + // Returns hex digest string +} ``` +**Attestation outcomes**: +- **Verified**: all actual tools in expected set; fingerprints match previous → no warnings +- **Unexpected**: actual tools not in expected set → warning logged (tool still usable) +- **Unconfigured**: no expected_tools declared → attestation skipped +- **Schema drift**: fingerprints differ from previous session → warning logged + +Attestation does NOT block tool loading; it aids operators in detecting catalog changes between deployments. + +## Tool Sanitization + +All tool definitions (names, descriptions, input schemas) pass through `sanitize_tools()` before registration to scrub injection patterns — see [[008-3-security]]. + ## Integration Points -- [[008-1-lifecycle]] — Tool discovery runs after server startup -- [[008-3-security]] — Pruning cache prevents tool injection via descriptions -- [[006-tools/spec]] — Pruned tool subset sent to ToolExecutor -- [[003-llm-providers/spec]] — Embedding provider used for semantic filtering +- [[008-1-lifecycle]] — Tools fetched during startup; cache invalidated on reconnection +- [[008-3-security]] — Sanitization applied to tool descriptions and parameter schemas +- [[006-tools/spec]] — Tool dispatch via `ToolExecutor` trait ## See Also - [[008-mcp/spec]] — Parent -- [[008-1-lifecycle]] — Server initialization -- [[008-3-security]] — Injection defense during pruning -- [[006-tools/spec]] — Tool execution after pruning +- [[008-1-lifecycle]] — Server connection lifecycle +- [[008-3-security]] — Tool sanitization and security checks diff --git a/specs/008-mcp/008-3-security.md b/specs/008-mcp/008-3-security.md index 6a9af9112..759b2a136 100644 --- a/specs/008-mcp/008-3-security.md +++ b/specs/008-mcp/008-3-security.md @@ -1,8 +1,9 @@ --- aliases: - MCP Security - - OAP Authorization - - SMCP Secure MCP + - Tool Sanitization + - Trust Scoring + - Data Flow Policy tags: - sdd - spec @@ -20,289 +21,305 @@ related: - "[[010-3-authorization]]" --- -# Spec: MCP Security & OAP Authorization +# Spec: MCP Security & Policy Enforcement -> [!danger] Stale content — 2026-07 audit -> Most of this file (Elicitation Phases pseudocode, `InjectionDetector`/`DeBERTaClassifier`, -> `OAPPolicy`, `SmcpEnvelope`) does not match the real implementation and was not corrected in this -> pass (rewrite, not a text fix) — none of `OAPPolicy`, `SmcpEnvelope`, `InjectionDetector`, -> `DeBERTaClassifier` exist in the codebase. Real analogues: `ContentSanitizer`, -> `ExfiltrationGuard`, `RiskChainAccumulator` (`crates/zeph-sanitizer/`), `CandleClassifier`/ -> `CandlePiiClassifier` (`crates/zeph-llm/src/classifier/`). Elicitation itself is accurately -> documented in [[008-4-elicitation]] — treat that file as authoritative for elicitation, not the -> "Elicitation Phases" section below. The top `## Key Invariants` block (Untrusted/tool_allowlist/ -> allow_untrusted_without_allowlist fail-closed behavior) reads as accurate and should be preserved -> when this file is rewritten. - -Elicitation phases, injection detection, OAP (Org-Aware Permission) authorization, SMCP secure protocol. +Tool sanitization, trust scoring, data-flow policy, elicitation gating, embedding anomaly detection. ## Overview -MCP servers are untrusted code running in subprocesses. Zeph enforces multiple layers of defense: input sanitization, output injection detection, and capability-based authorization via OAP. +MCP servers are untrusted code running in subprocesses or remote HTTP services. Zeph enforces multiple layers of defense before tool calls reach the agent and after results return: + +1. **Pre-connect**: Probing (`DefaultMcpProber`) scans `resources/list` and `prompts/list` for injection +2. **At registration**: Tool definitions sanitized; collision detection runs +3. **Input phase**: Schema validation + policy checks + elicitation probing (optional) +4. **Output phase**: Injection scanning + embedding anomaly detection + PII redaction +5. **Trust scoring**: Per-server risk track with decay ## Key Invariants **Always:** -- All tool inputs sanitized before passing to server (schema validation, injection checks) -- All tool outputs scanned for injection patterns before returning to agent -- Server responses that fail validation are rejected with logging -- OAP authorization policies applied: which tools can be called by which agents -- `Untrusted` servers fail closed on a missing `tool_allowlist` — zero tools exposed unless - an allowlist is declared or `allow_untrusted_without_allowlist` is explicitly set. Only - `Trusted` exposes all tools without an allowlist. Trust levels not explicitly handled by - the allowlist policy (including future variants) also fail closed. +- All tool definitions sanitized (names, descriptions, input schema parameter descriptions) before registration +- Tool inputs validated against `input_schema` JSON schema before passing to server +- Tool outputs scanned for injection patterns before returning to agent +- `Untrusted` servers fail closed: zero tools exposed unless `tool_allowlist` is declared or `allow_untrusted_without_allowlist = true` +- Trust level enforcement: `Trusted` → all tools; `Untrusted` → allowlist only; `Sandboxed` → allowlist only, no elicitation +- Anomalous tool-call sequences flagged by `EmbeddingAnomalyGuard` post-execution **Never:** -- Pass unsanitized user input to MCP tools -- Trust server output without injection detection (DeBERTa + regex) -- Allow cross-agent tool access without explicit capability delegation -- Expose all tools for an `Untrusted` server with no declared `tool_allowlist` unless the - operator has explicitly opted in via `allow_untrusted_without_allowlist` +- Pass unsanitized user input to MCP tools — always validate against schema +- Trust server output without scanning — all responses pass through injection/PII detection +- Expose all tools for `Untrusted` servers without explicit operator opt-in +- Run elicitation on `Sandboxed` servers even if `elicitation_enabled = true` -## Elicitation Phases +## Tool Sanitization Pipeline -Tool invocation has three security phases: +`sanitize_tools()` (`crates/zeph-mcp/src/sanitize.rs`) mutates tool definitions in-place, scrubbing injection patterns: +```rust +pub struct SanitizeResult { + /// Number of individual fields (description, schema strings) replaced with "[sanitized]" + pub injection_count: usize, + /// Names of tools that had at least one injected field + pub flagged_tools: Vec, + /// (tool_name, pattern_name) pairs for audit and logging + pub flagged_patterns: Vec<(String, String)>, +} + +/// Sanitize tool definitions in-place: scrub injection patterns from descriptions/schemas. +/// +/// Modifies `tools` directly; does not wrap or return them (sync, no Result). +pub fn sanitize_tools( + tools: &mut [McpTool], + server_id: &str, + max_description_bytes: usize, +) -> SanitizeResult { + // 1. Scan tool.name for injection patterns + // 2. Scan tool.description; truncate/replace matched patterns + // 3. Recursively scan tool.input_schema.properties[*].description + // 4. Truncate schemas exceeding max_description_bytes (threat: decompression bomb) + // 5. Record all flagged fields for audit + // 6. Return metadata; tools are mutated in place +} ``` -1. INPUT PHASE - ├─ Schema validation (input_schema) - ├─ Injection detection (SQL, shell, prompt injection) - └─ Rate limiting check - -2. EXECUTION PHASE - ├─ RPC call to MCP server - ├─ Timeout enforcement - └─ Error handling - -3. OUTPUT PHASE - ├─ Response format validation - ├─ Injection detection (DeBERTa on output) - └─ PII detection (NER + redaction) -``` -Code: +**Patterns scanned**: +- Prompt injection: "ignore this instruction", "pretend you are", "SYSTEM:", etc. +- SQL injection: `SELECT`, `DROP`, `; --`, etc. +- Shell injection: `$(...)`, `` `...` ``, `; rm -rf`, etc. + +**Sanitization behavior**: matched patterns are replaced with `[sanitized]`. Tools with flagged descriptions remain callable; only the suspicious text is replaced. Trust score is updated based on `injection_count`. + +## Pre-Connect Probing + +Before loading tools via `tools/list`, `DefaultMcpProber::probe()` scans `resources/list` and `prompts/list` for pre-connection risk assessment: ```rust -async fn invoke_tool_secure( - &self, - tool_name: &str, - input: Value, - caller_context: &AgentContext, -) -> Result { - // PHASE 1: Input validation - let tool = self.get_tool(tool_name)?; - - // Schema validation - jsonschema::validate(&input, &tool.input_schema) - .map_err(|e| anyhow!("Schema validation failed: {}", e))?; - - // Injection detection - if self.detect_injection(&input)? { - return Err(anyhow!("Injection detected in tool input")); - } - - // Rate limiting - self.check_rate_limit(tool_name, caller_context)?; - - // PHASE 2: Execution - let result = self.server.call_tool( - tool_name, - input, - Duration::from_secs(30), // timeout - ).await?; - - // PHASE 3: Output validation - - // Format check - if !result.is_object() && !result.is_string() { - return Err(anyhow!("Unexpected tool output format")); - } - - // Injection detection on output - if self.detect_output_injection(&result)? { - log::warn!("Injection detected in tool output from {}", tool_name); - return Err(anyhow!("Tool output validation failed")); +pub struct ProbeResult { + /// Trust score delta from probing (negative = risk detected) + pub score_delta: f64, + /// Probing summary (injection pattern count, resources/prompts found) + pub summary: String, + /// If true, server failed probing and is marked dangerous + pub block: bool, +} + +impl DefaultMcpProber { + /// Scan resources and prompts for injection patterns; update trust scoring. + /// + /// No Result wrapper — probing is advisory. Always returns ProbeResult. + pub async fn probe( + &self, + server_id: &str, + client: &McpClient, + ) -> ProbeResult { + // Fetches resources/list, prompts/list (if available) + // Scans descriptions for injection patterns + // Computes score delta; returns risk assessment } - - // PII redaction - let redacted = self.redact_pii(&result)?; - - Ok(redacted) } ``` -## Injection Detection +**Probing semantics**: always runs before `tools/list`, even for trusted servers. Results are logged but do NOT block connection. Trust score is adjusted based on pattern counts detected. + +## Trust Score System -Multi-layer detection using DeBERTa + regex patterns: +`ServerTrustScore` (`crates/zeph-mcp/src/trust_score.rs`) tracks per-server cumulative risk with asymmetric decay: ```rust -pub struct InjectionDetector { - deberta: Arc, // "is this injection?" binary classifier - regex_patterns: Vec, // SQL, shell, prompt injection patterns +pub struct ServerTrustScore { + pub server_id: String, // Unique server identifier + pub score: f64, // [0.0, 1.0]; 0.5 = neutral (initial) + pub success_count: u64, // Successful tool executions + pub failure_count: u64, // Failed or injection-detected calls + pub updated_at_secs: u64, // Timestamp of last update } -impl InjectionDetector { - fn detect_injection(&self, value: &Value) -> Result { - let text = match value { - Value::String(s) => s.clone(), - Value::Object(o) => serde_json::to_string(o)?, - _ => return Ok(false), - }; - - // 1. Regex check (fast) - for pattern in &self.regex_patterns { - if pattern.is_match(&text) { - log::warn!("Regex injection pattern matched"); - return Ok(true); - } - } - - // 2. DeBERTa check (slow, only if suspicious) - if text.len() > 100 && text.contains("SELECT") || text.contains("$(") { - let score = self.deberta.classify(&text).await?; - if score > 0.8 { - return Ok(true); - } - } - - Ok(false) - } +impl ServerTrustScore { + /// Record a successful tool execution; boost score. + pub fn record_success(&mut self); + + /// Record a failure or injection detection; penalize score. + pub fn record_failure(&mut self); } ``` -## OAP Authorization +**Score updates**: +- **Success**: `+0.02` per call (capped at 1.0) +- **Injection penalty**: `-0.25` per injection pattern detected +- **Failure penalty**: `-0.10` per call failure +- **Exponential decay**: scores **above** 0.5 decay toward 0.5 at ~0.01/day; **below or at** 0.5 stay put (asymmetric: no recovery from decay alone) -Capability-based access control: +**Semantics**: score below 0.5 indicates the server is untrusted and requires explicit `tool_allowlist` to expose tools. An attacker cannot regain trust by going quiet (decay doesn't raise scores below neutral). + +## Data-Flow Policy Enforcement + +`check_data_flow()` restricts sensitive tools based on trust level — sensitivity levels are ordered `None < Low < Medium < High`: ```rust -pub struct OAPPolicy { - // agent_id → allowed tool names - permissions: HashMap>, - // tool_id → required capability - capabilities: HashMap, +#[non_exhaustive] +#[derive(Debug, thiserror::Error)] +pub enum DataFlowViolation { + #[error( + "tool '{tool_name}' (sensitivity={sensitivity:?}) on server '{server_id}' \ + (trust={trust:?}) violates data-flow policy" + )] + SensitivityTrustMismatch { + server_id: String, + tool_name: ToolName, + sensitivity: DataSensitivity, + trust: McpTrustLevel, + }, } -impl OAPPolicy { - fn check_authorization( - &self, - agent_id: &str, - tool_name: &str, - ) -> Result<()> { - // 1. Check if agent has tool permission - let allowed = self.permissions - .get(agent_id) - .ok_or_else(|| anyhow!("Agent {} not in ACL", agent_id))?; - - if !allowed.contains(tool_name) { - return Err(anyhow!( - "Agent {} not authorized for tool '{}'", - agent_id, - tool_name - )); - } - - // 2. Check capability delegation - let required = self.capabilities - .get(tool_name) - .copied() - .unwrap_or(Capability::PublicRead); - - if !self.has_capability(agent_id, required) { - return Err(anyhow!( - "Agent {} lacks capability {:?} for tool '{}'", - agent_id, - required, - tool_name - )); - } - - Ok(()) - } +/// Check data-flow policy: tool sensitivity must not exceed server trust. +/// +/// Sensitivity is read from `tool.security_meta.data_sensitivity`. +pub fn check_data_flow( + tool: &McpTool, + server_trust: McpTrustLevel, +) -> Result<(), DataFlowViolation> { + let sensitivity = tool.security_meta.data_sensitivity; + + // Examples of violations (errors): + // - High-sensitivity tool on Sandboxed server → block + // - High-sensitivity tool on Untrusted server → block + + // Allowed (no error): + // - Any tool on Trusted server + // - Medium-sensitivity tool on Untrusted/Sandboxed (Sandboxed logs WARN) + // - Low-sensitivity tool on Untrusted/Sandboxed +} +``` + +**Policy enforcement**: high-sensitivity tools (e.g., credential management, user deletion) require `Trusted` servers only; medium-sensitivity requires `Trusted` or `Untrusted` (Sandboxed is allowed with warning); low/none are available everywhere. + +## Output Validation: Injection Scanning + +Tool output sanitization is a cross-cutting concern at the agent loop layer (`crates/zeph-core/src/agent/tool_execution/sanitize.rs`), not per-executor. All tool outputs — MCP, web scrape, memory retrieval, and native — flow through a unified sanitization pipeline: + +```rust +/// Sanitize tool output body before inserting into LLM message history. +pub(super) async fn sanitize_tool_output( + &mut self, + body: &str, + tool_name: &str, +) -> ( + String, // Sanitized body + bool, // Injection detected flag + ContentSourceKind, // Source classification + zeph_sanitizer::ContentTrustLevel, // Trust level +) { + // Classify output source: MCP response, web scrape, memory retrieval, or generic tool result + let source = build_tool_output_source(tool_name); + + // MCP responses are identified by tool_name containing ':' (server:tool format) + // and tagged as ContentSourceKind::McpResponse + + // Injection detection via regex spotlighting and optional ML classifiers + let sanitized = self.sanitizer.sanitize(&body, source); + + (sanitized.body, !sanitized.injection_flags.is_empty(), source.kind, source.trust_level) } -#[derive(Debug, Clone, Copy)] -pub enum Capability { - PublicRead, - MutableState, - Execute, - Privileged, +/// Route tool outputs to their correct source classification. +fn build_tool_output_source(tool_name: &str) -> ContentSource { + if tool_name.contains(':') || tool_name == "mcp" { + // MCP responses identified by server:tool naming convention + ContentSource::new(ContentSourceKind::McpResponse).with_identifier(tool_name) + } else if tool_name.starts_with("web") { + ContentSource::new(ContentSourceKind::WebScrape).with_identifier(tool_name) + } else if tool_name == "memory_search" { + ContentSource::new(ContentSourceKind::MemoryRetrieval).with_identifier(tool_name) + } else { + ContentSource::new(ContentSourceKind::ToolResult).with_identifier(tool_name) + } } ``` -## SMCP (Secure MCP) +**Flow**: `McpToolExecutor::execute_tool()` returns raw output → agent loop classifies via `build_tool_output_source()` → sanitization applies injection detection and trust wrapping via `sanitize_tool_output()` → clean body enters LLM context. + +## Output Validation: Embedding Anomaly Detection -Optional enhanced security mode for critical tools: +`EmbeddingAnomalyGuard` (`crates/zeph-mcp/src/embedding_guard.rs`) detects anomalous per-(server,tool) output patterns asynchronously via centroid drift: ```rust -pub struct SmcpEnvelope { - // HMAC-signed request/response - request_id: String, - nonce: [u8; 32], - signature: Vec, - timestamp: i64, +pub struct EmbeddingGuardEvent { + pub server_id: String, + pub tool_name: ToolName, + pub result: EmbeddingGuardResult, } -impl SmcpEnvelope { - fn sign_request( - request: &ToolRequest, - shared_secret: &[u8], - ) -> Result { - let mut hasher = HmacSha256::new_from_slice(shared_secret)?; - let payload = serde_json::to_vec(request)?; - hasher.update(&payload); - - let signature = hasher.finalize().into_bytes().to_vec(); - - Ok(SmcpEnvelope { - request_id: uuid::Uuid::new_v4().to_string(), - nonce: rand::random(), - signature, - timestamp: now(), - }) +#[non_exhaustive] +#[derive(Debug, Clone)] +pub enum EmbeddingGuardResult { + /// Output is within the expected distribution for this (server,tool) pair + Normal { distance: f64 }, + /// Output is anomalous — possible injection or unexpected content + Anomalous { distance: f64, threshold: f64 }, + /// Cold-start: insufficient clean samples; regex fallback used instead + RegexFallback { injection_detected: bool }, +} + +pub struct EmbeddingAnomalyGuard { /* ... */ } + +impl EmbeddingAnomalyGuard { + /// Asynchronous fire-and-forget check of tool output for anomalies. + /// + /// Results are sent to a channel; no return value. + pub fn check_async( + &self, + server_id: &str, + tool_name: ToolName, + tool_output: &str, + ) { + // Background task: embed output + // Compute distance from running centroid of this (server,tool)'s historical outputs + // Send EmbeddingGuardEvent to the result channel } } ``` -## Configuration - -```toml -[mcp.security] -injection_detection_enabled = true -deberta_enabled = false # expensive; enable for high-risk tools -output_validation_enabled = true -oap_authorization_enabled = true -rate_limit_per_minute = 60 - -# Injection patterns (regex) -injection_patterns = [ - "(?i)(DROP|DELETE|UPDATE|INSERT).*FROM", # SQL - "\\$\\(.*\\)|`.*`", # Shell - "prompt\\s*injection|jailbreak|ignore", # Prompt injection -] - -# Tool capabilities -[[mcp.security.tool_capabilities]] -tool = "shell_execute" -capability = "Execute" -rate_limit_per_minute = 10 - -[[mcp.security.tool_capabilities]] -tool = "list_files" -capability = "PublicRead" -``` +**Semantics**: the guard tracks per-(server,tool) centroid of **that tool's own** historical outputs, not cross-tool pattern matching. Anomalies (outputs far from the tool's historical pattern) trigger trust-score penalties. + +## Allowlist & Fail-Closed Semantics + +Trust level behavior: + +| Trust Level | Tool Allowlist Provided | Behavior | +|---|---|---| +| `Trusted` | No | Expose all tools | +| `Trusted` | Yes | Expose allowlisted tools only | +| `Untrusted` | No | **Expose zero tools** (fail closed) unless `allow_untrusted_without_allowlist = true` | +| `Untrusted` | Yes | Expose allowlisted tools only | +| `Sandboxed` | No | Expose zero tools | +| `Sandboxed` | Yes | Expose allowlisted tools only; disable elicitation | + +Default: `allow_untrusted_without_allowlist = false` (secure default). An operator must explicitly opt in to expose tools from an untrusted server without an allowlist. + +## Elicitation Gating + +When `elicitation_enabled = true`, Zeph probes tool parameters for hidden capabilities: + +- `Sandboxed` servers: elicitation disabled regardless of config +- `Untrusted` servers: elicitation allowed unless allowlist says otherwise +- `Trusted` servers: elicitation allowed + +See [[008-4-elicitation]] for elicitation details. + +## Attestation + +When `expected_tools` list is provided, compare actual tools against expected list at startup (see [[008-2-discovery]]). Schema mismatches logged but do NOT block usage. ## Integration Points -- [[008-1-lifecycle]] — Security checks during subprocess spawning -- [[008-2-discovery]] — Tool descriptions scanned for injection before caching -- [[010-security/spec]] — Parent security spec; refers to injection defense -- [[010-3-authorization]] — OAP policy enforcement -- [[025-classifiers/spec]] — DeBERTa-backed injection detection +- [[008-1-lifecycle]] — Sanitization runs after tools/list fetch +- [[008-2-discovery]] — Collision detection after sanitization +- [[008-4-elicitation]] — Parameter probing (optional) +- [[010-2-injection-defense]] — Output scanning via ContentSanitizer +- [[010-3-authorization]] — Policy enforcement ## See Also - [[008-mcp/spec]] — Parent -- [[008-1-lifecycle]] — Server lifecycle where security starts -- [[008-2-discovery]] — Tool discovery with injection scanning -- [[010-security/spec]] — Cross-cutting security constraints -- [[010-3-authorization]] — OAP authorization policies +- [[010-security/spec]] — Security architecture +- [[008-4-elicitation]] — Optional tool parameter discovery diff --git a/specs/010-security/010-1-vault.md b/specs/010-security/010-1-vault.md index 1314cd4a0..4f09c4397 100644 --- a/specs/010-security/010-1-vault.md +++ b/specs/010-security/010-1-vault.md @@ -20,166 +20,133 @@ related: # Spec: Secret Vault & Credential Resolution -> [!danger] Stale content — 2026-07 audit -> This file's code samples and type names do not match the real implementation and were not -> corrected in this pass (rewrite, not a text fix): it uses `serde_yaml` (banned project-wide, see -> `[[constitution]]` II — use `serde_norway`), `log::info!`/`log::warn!` (banned, `tracing` only), -> invented types `VaultBackend`/`AgeVault` (real trait is `VaultProvider`, -> `crates/zeph-vault/src/lib.rs`), and lists `age`/`kms`/`custom` backends (only `age`/`env` exist, -> per the correct listing in `[[010-security/spec]]` and `[[038-vault/spec]]`). **Treat -> [[038-vault/spec]] as authoritative for vault behavior**, not this file. - Age-encrypted secret storage, credential resolution, ZEPH_* environment key mapping, vault access control. ## Overview -Zeph stores all secrets (API keys, tokens, passwords) in an encrypted age vault, not in environment variables or `.env` files. The vault is automatically decrypted at startup, and credentials are resolved on-demand by subsystems. +Zeph stores all secrets (API keys, tokens, passwords) in an encrypted age vault, not in environment variables or `.env` files. The vault is automatically decrypted at startup, and credentials are resolved on-demand by subsystems using the pluggable `VaultProvider` trait. + +Two backends ship out of the box: `AgeVaultProvider` (recommended, age encryption) and `EnvVaultProvider` (development/testing, reads `ZEPH_SECRET_*` env vars). ## Key Invariants **Always:** -- All API keys, tokens, and passwords stored in age vault only -- Vault backend (age/kms/custom) configured once at startup -- Credentials resolved via `vault.get("KEY_NAME")` at runtime, never from env vars -- ZEPH_* environment variables automatically resolved from vault at startup +- All API keys, tokens, and passwords stored in encrypted age vault (production) or env vars (testing only) +- Vault backend configured once at startup via `VaultProvider` trait implementation +- Credentials resolved via `vault.get_secret("KEY_NAME").await` at runtime, never from raw env vars +- Secret values kept in `zeroize::Zeroizing` buffers — automatically zeroed on drop +- Vault file permissions set to `0o600` (owner-read/write only) on Unix **Never:** - Store secrets in `.env`, config files, or command-line arguments -- Pass API keys through shell pipelines or logs +- Pass API keys through logging, error messages, or debug output - Hardcode credentials in source code -- Use plaintext file backend in production - -## Age Vault Structure - -YAML format for quick editing: - -```yaml -# ~/.age/zeph.age (encrypted) ---- -# OpenAI -openai_api_key: "sk-proj-..." -openai_org_id: "org-..." - -# Anthropic -claude_api_key: "sk-ant-..." - -# Local APIs -ollama_api_base: "http://localhost:11434" +- Use synchronous blocking I/O inside async contexts for vault access -# Database -postgres_url: "postgresql://user:pass@host/db" +## Vault Provider Trait -# External Services -telegram_bot_token: "123456:ABCdef..." -github_token: "ghp_..." +```rust +pub trait VaultProvider: Send + Sync { + /// Retrieve a secret by key. + /// + /// Returns `Ok(None)` when the key does not exist. + /// Returns `Err(VaultError)` on backend failures (I/O, decryption, network). + fn get_secret( + &self, + key: &str, + ) -> Pin, VaultError>> + Send + '_>>; + + /// Return all known secret keys (optional). + /// + /// Default implementation returns an empty `Vec`. + fn list_keys(&self) -> Vec { + Vec::new() + } +} ``` -Encrypted with user's age public key: +## Age Backend -```bash -age --encrypt --recipient age1xxx... ~/.age/zeph.age.plaintext > ~/.age/zeph.age -rm ~/.age/zeph.age.plaintext +**File layout:** +``` +~/.config/zeph/ +├── vault-key.txt # age identity (private key), mode 0600 +└── secrets.age # age-encrypted JSON: {"KEY": "value", ...} ``` -## Vault Access Interface - +**API:** ```rust -pub trait VaultBackend: Send + Sync { - async fn get(&self, key: &str) -> Result; - async fn set(&self, key: &str, value: &str) -> Result<()>; - async fn list(&self) -> Result>; - async fn delete(&self, key: &str) -> Result<()>; -} +pub struct AgeVaultProvider { /* private */ } -pub struct AgeVault { - // Decrypted in-memory map after startup - secrets: Arc>>, - vault_path: PathBuf, - identity_path: PathBuf, +impl AgeVaultProvider { + /// Load vault from encrypted files. + pub fn new(key_path: &Path, vault_path: &Path) -> Result; + + /// Initialize a new vault with a fresh age keypair. + pub fn init_vault(dir: &Path) -> Result<(), AgeVaultError>; + + /// Synchronous getter for convenience (non-async). + pub fn get(&self, key: &str) -> Option<&str>; + + /// Set or update a secret (requires mutable access). + pub fn set_secret_mut(&mut self, key: String, value: String, is_new: bool) + -> Result<(), AgeVaultError>; + + /// Remove a secret. + pub fn remove_secret_mut(&mut self, key: &str) -> bool; + + /// Save encrypted vault to disk (atomic write via `.tmp` suffix). + pub fn save(&self) -> Result<(), AgeVaultError>; + + /// Async variant that offloads I/O to a background thread. + pub async fn load_async(key_path: &Path, vault_path: &Path) + -> Result; + + pub async fn save_async(&self) -> Result<(), AgeVaultError>; } -impl AgeVault { - async fn load(&mut self) -> Result<()> { - // 1. Read encrypted file - let encrypted = fs::read_to_string(&self.vault_path)?; - - // 2. Decrypt using age identity - let decrypted = age_decrypt(&encrypted, &self.identity_path)?; - - // 3. Parse YAML - let parsed: HashMap = serde_yaml::from_str(&decrypted)?; - - // 4. Store in memory - *self.secrets.write().await = parsed; - - Ok(()) - } - - async fn get(&self, key: &str) -> Result { - self.secrets - .read() - .await - .get(key) - .cloned() - .ok_or_else(|| anyhow!("Secret '{}' not found in vault", key)) - } +impl VaultProvider for AgeVaultProvider { /* ... */ } +``` + +Secrets stored as JSON (not YAML) for forward compatibility: +```json +{ + "ZEPH_CLAUDE_API_KEY": "sk-ant-...", + "ZEPH_OPENAI_API_KEY": "sk-proj-...", + "OLLAMA_API_BASE": "http://localhost:11434" } ``` -## Startup Resolution +## Env Backend -At agent initialization, populate required ZEPH_* vars: +Development/testing backend that reads `ZEPH_SECRET_*` prefixed environment variables: ```rust -async fn resolve_vault_at_startup(vault: &AgeVault) -> Result<()> { - let required_keys = vec![ - "openai_api_key", - "claude_api_key", - "ollama_api_base", - ]; - - for key in required_keys { - if let Ok(value) = vault.get(key).await { - // Don't set env var; store in config instead - // ZEPH_OPENAI_API_KEY is resolved lazily from vault - log::info!("Resolved credential: {}", key); - } else { - log::warn!("Missing vault key: {}", key); - } +pub struct EnvVaultProvider; + +impl VaultProvider for EnvVaultProvider { + fn get_secret(&self, key: &str) -> Pin>> { + // Looks for environment variable with `ZEPH_SECRET_` prefix + // E.g., key "API_KEY" reads env var "ZEPH_SECRET_API_KEY" } - - Ok(()) } ``` -## Lazy Resolution in LLM Providers +Used for testing and CI pipelines where file-based vaults are impractical. -Providers request credentials on-demand: +## Arc Wrapper + +`ArcAgeVaultProvider` wraps `Arc>` to allow sharing the vault as a trait object while still supporting mutable operations (e.g., OAuth credential persistence): ```rust -pub struct ClaudeProvider { - vault: Arc, -} +pub struct ArcAgeVaultProvider { /* ... */ } -impl ClaudeProvider { - async fn get_api_key(&self) -> Result { - self.vault.get("claude_api_key").await - } - - async fn invoke(&self, req: &Request) -> Result { - let api_key = self.get_api_key().await?; - - // Make request with credential - let client = reqwest::Client::new(); - client - .post("https://api.anthropic.com/v1/messages") - .bearer_auth(&api_key) - .json(req) - .send() - .await? - .json() - .await - } +impl VaultProvider for ArcAgeVaultProvider { /* ... */ } + +// Mutable methods available via downcasting: +impl ArcAgeVaultProvider { + pub fn set_secret_mut(&self, key: String, value: String, is_new: bool) + -> Result<(), AgeVaultError>; } ``` @@ -192,10 +159,10 @@ Management interface: cargo run -- vault list # Get a secret (for external scripts) -cargo run -- vault get openai_api_key +cargo run -- vault get ZEPH_OPENAI_API_KEY -# Set a secret (interactive prompt) -cargo run -- vault set new_key_name +# Initialize a new vault with fresh keypair +cargo run -- vault init # Validate vault integrity cargo run -- vault check @@ -205,12 +172,25 @@ cargo run -- vault check ```toml [vault] -backend = "age" # or "kms", "custom" -vault_path = "~/.age/zeph.age" -identity_path = "~/.age/identity" -fallback_env = false # don't read from env vars as fallback +backend = "age" # "age" (default, recommended), "env" (dev/testing only), or "keyring" (OS keyring) ``` +**Vault file path resolution** (for `Age` backend): + +Vault files are stored with hardcoded names in a platform-specific config directory, resolved in order: +1. `$XDG_CONFIG_HOME/zeph` (Linux/BSD) +2. `$APPDATA\zeph` (Windows) +3. `$HOME/.config/zeph` (macOS and fallback) + +Files within the resolved directory: +- `vault-key.txt` — age private key (created with `0o600` permissions on Unix) +- `secrets.age` — age-encrypted JSON of secrets + +**Backend variants:** +- `age` — Recommended for production. Encrypted age vault with private key. +- `env` — Development/testing only. Reads `ZEPH_SECRET_*` environment variables (explicitly set by the user or via `.env` in testing). +- `keyring` — OS-native keyring (macOS Keychain, Windows Credential Manager, Linux Secret Service). + ## Integration Points - [[003-llm-providers/spec]] — All providers resolve API keys from vault diff --git a/specs/010-security/010-2-injection-defense.md b/specs/010-security/010-2-injection-defense.md index bf9199359..42cc3dc41 100644 --- a/specs/010-security/010-2-injection-defense.md +++ b/specs/010-security/010-2-injection-defense.md @@ -23,262 +23,211 @@ related: # Spec: Injection Defense & Content Isolation -> [!danger] Stale content — 2026-07 audit -> The `IpiDetector`/`PiiDetector`/`DeBERTaClassifier` code blocks below are fictional — none of -> these type names exist in the codebase — and were not corrected in this pass (rewrite, not a text -> fix). Real code: `crates/zeph-llm/src/classifier/` (`CandleClassifier`, `CandlePiiClassifier`) and -> `crates/zeph-sanitizer/src/causal_ipi.rs` (`TurnCausalAnalyzer`). The "User Verification Gate" -> section's direct `println!(...)` also contradicts [[001-system-invariants/spec#1. Channel Contract]] -> ("the `Channel` trait is the only I/O boundary") — do not copy that pattern. - -Indirect Prompt Injection (IPI) defense, DeBERTa-backed detection, AlignSentinel confidence scoring, PII NER detection, content isolation. +Indirect Prompt Injection (IPI) defense, regex-based detection, ML classifier soft signals, TurnCausalAnalyzer, PII NER detection, content spotlighting. ## Overview -Zeph processes untrusted content from web scraping, MCP tool outputs, and user uploads. IPI attacks attempt to override agent behavior by embedding instructions in observed content. Zeph defends via multi-layer detection: regex, DeBERTa binary classifier, confidence scoring, and explicit user verification gates. +Zeph processes untrusted content from web scraping, MCP tool outputs, A2A calls, and memory retrieval. IPI attacks attempt to override agent behavior by embedding instructions in observed content. Zeph defends via multi-layer detection: regex spotlighting with content source attribution, optional ML-backed classifiers, turn-level causal analysis for tool-chain anomalies, and PII NER redaction. ## Key Invariants **Always:** -- All web-fetched content scanned for IPI patterns before returning to agent -- MCP tool outputs validated before reaching context/LLM -- Suspicious content flagged with confidence score; high scores require user verification -- PII (SSN, credit card, email) detected via NER and redacted or blocked +- All untrusted content (`WebScrape`, `MemoryRetrieval`, `A2A`, `McpToolResult`) wrapped in spotlighting delimiters and source attribution +- External content injected at `ContentTrustLevel::ExternalUntrusted` or `LocalUntrusted` depending on source +- Regex-based injection pattern detection runs unconditionally on external content (`flag_injection_patterns = true`) +- PII entities truncated before ML model inference (max 4096 chars) to prevent OOM +- Secrets masked via `SecretMaskRegistry` at LLM boundary to prevent leakage **Never:** -- Trust content from web pages, emails, or tool outputs without scanning -- Execute instructions found in observed content without explicit user approval -- Log secrets or PII in debug dumps -- Suppress IPI warnings when confidence is high +- Run IPI ML classifiers on agent-generated content — only on external/tool-sourced content +- Accumulate cross-turn injection signals for confirmation decisions — `CrossToolCorrelator` clears state at turn boundaries +- Log or expose redacted PII values or secret shapes in debug output +- Skip spotlight wrapping for untrusted content even if regex confidence is low -## IPI Detection Layers +## Content Sanitizer: Spotlighting & Injection Detection +`ContentSanitizer` (`crates/zeph-sanitizer/src/sanitizer.rs`) wraps untrusted content with source attribution and scans for regex injection patterns: + +```rust +pub struct ContentSanitizer { /* ... */ } + +impl ContentSanitizer { + /// Sanitize external content: spot-light, truncate, flag injection patterns. + pub fn sanitize( + &self, + text: &str, + source: ContentSource, + ) -> SanitizedContent { + // 1. Apply content trust level based on source.kind + // 2. Truncate if > max_content_size + // 3. If external + untrusted: spotlight with ... + // 4. Scan for regex injection patterns; populate injection_flags + // 5. Return wrapped content ready for LLM context + } +} + +pub struct SanitizedContent { + pub source: ContentSource, // Source of the content (web, tool, MCP, etc.) + pub body: String, // Spotlighted, truncated content + pub injection_flags: Vec, // Detected pattern names + pub was_truncated: bool, // True if exceeded max size +} ``` -Layer 1: Regex Patterns (Fast) -├─ "ignore this instruction" -├─ "prompt injection|jailbreak" -├─ SYSTEM: embedded in content -└─ Confidence: 0.95 (high) - -Layer 2: DeBERTa Binary Classifier (Slow) -├─ Fine-tuned on IPI examples -├─ Outputs confidence [0.0–1.0] -└─ Run only if Layer 1 suspicious - -Layer 3: AlignSentinel Confidence Scoring -├─ Combines regex + DeBERTa scores -├─ Context-aware adjustments -└─ > 0.8 = require user verification -``` -Code: +**Trust levels** (`ContentTrustLevel`): +- `Trusted` — passes unchanged (system prompt, validated user input) +- `LocalUntrusted` — tool results from local executors; wrapped in `` with NOTE header +- `ExternalUntrusted` — web, MCP, A2A, memory; wrapped in `` with IMPORTANT warning + +## Optional ML-Backed Soft Signals + +When feature `classifiers` is enabled and a backend is attached, `ContentSanitizer::classify_injection` provides DeBERTa-backed injection probability (advisory only): ```rust -pub struct IpiDetector { - regex_patterns: Vec<(Regex, f32)>, // (pattern, confidence) - deberta: Arc, - align_sentinel: Arc, +impl ContentSanitizer { + /// Optional: classify content with ML classifier (requires backend). + pub async fn classify_injection(&self, text: &str) + -> Result { + // DeBERTa model (via candle): returns continuous [0.0–1.0] probability + // Policy-blocked outputs are skipped; ML classification only on advisory path + } } +``` + +**Note**: IPI detection is part of the `[security.content_isolation]` configuration and controlled via the `flag_injection_patterns` flag. There is no separate `[security.ipi]` section; injection detection runs unconditionally on external content. + +## Turn-Level Causal Analysis + +`TurnCausalAnalyzer` (`crates/zeph-sanitizer/src/causal_ipi.rs`) detects anomalous patterns in tool-call sequences within a turn: -impl IpiDetector { - async fn scan_content(&self, text: &str) -> Result { - // Layer 1: Regex (fast) - let mut max_regex_score = 0.0; - for (pattern, confidence) in &self.regex_patterns { - if pattern.is_match(text) { - max_regex_score = max_regex_score.max(*confidence); - } - } - - // Short-circuit if regex very confident - if max_regex_score > 0.9 { - return Ok(ScanResult { - is_injection: true, - confidence: max_regex_score, - detected_by: "regex", - content_preview: truncate(text, 100), - }); - } - - // Layer 2: DeBERTa (expensive; only if regex moderately suspicious) - let deberta_score = if max_regex_score > 0.5 || text.len() > 500 { - self.deberta.classify(text).await? - } else { - 0.0 - }; - - // Layer 3: AlignSentinel combines scores - let final_score = self.align_sentinel.combine_scores( - max_regex_score, - deberta_score, - text.len(), - ); - - Ok(ScanResult { - is_injection: final_score > 0.75, - confidence: final_score, - detected_by: if deberta_score > max_regex_score { - "deberta" - } else { - "regex" - }, - content_preview: truncate(text, 100), - }) +```rust +pub struct TurnCausalAnalyzer { /* ... */ } + +impl TurnCausalAnalyzer { + /// Analyze whether a pair of (prior_response, current_response) shows causal anomalies. + /// + /// Synchronous local computation (no LLM call). Returns a value with .is_flagged / .deviation_score fields. + pub fn analyze(&self, pre_response: &str, post_response: &str) -> CausalAnalysis { + // Local embedding-based comparison of prior response vs current response + // Returns anomaly score if deviation is high } } ``` +This check runs **within the turn only** — state is cleared at turn boundaries per the parent spec's NEVER rule. A separate async LLM-backed method exists for probe generation but isn't what's described here. + ## PII Detection & Redaction -Named Entity Recognition for sensitive data: +`CandlePiiClassifier` (feature: `classifiers`) performs Named Entity Recognition (NER) on tool inputs/outputs when the feature is enabled: ```rust -pub struct PiiDetector { - ner_model: Arc, // Detects SSN, email, credit card, etc. +impl ContentSanitizer { + /// Optional: detect PII via NER model (requires feature + backend). + pub async fn detect_pii(&self, text: &str) + -> Result> { + // Truncate to pii_max_input_chars (default 4096) before model inference + // NER inference: returns PII entities (SSN, credit card, email, phone, etc.) + // Detected entities are redacted from final content + } } -impl PiiDetector { - async fn detect_pii(&self, text: &str) -> Result> { - // NER inference - let entities = self.ner_model.extract_entities(text).await?; - - Ok(entities - .into_iter() - .filter(|e| matches!( - e.entity_type, - EntityType::SSN - | EntityType::CreditCard - | EntityType::Email - | EntityType::PhoneNumber - )) - .collect()) - } - - async fn redact_pii(&self, text: &str) -> Result { - let entities = self.detect_pii(text).await?; - - let mut redacted = text.to_string(); - for entity in entities.iter().rev() { - // Replace in reverse order to preserve offsets - let replacement = match entity.entity_type { - EntityType::SSN => "[REDACTED_SSN]", - EntityType::CreditCard => "[REDACTED_CC]", - EntityType::Email => "[REDACTED_EMAIL]", - EntityType::PhoneNumber => "[REDACTED_PHONE]", - _ => "[REDACTED]", - }; - - redacted.replace_range(entity.start..entity.end, replacement); - } - - Ok(redacted) +pub struct PiiFilter { + regex_patterns: Vec, // Fallback: regex-based PII detection +} + +impl PiiFilter { + /// Regex-based PII scrubbing (email, phone, SSN, credit card). + /// + /// Returns a `Cow<'a, str>` — if no patterns matched, returns a borrow of the original; + /// if patterns matched, returns an owned String with scrubbed content. + pub fn scrub<'a>(&self, text: &'a str) -> Cow<'a, str> { + // Regex-based PII scrubbing (email, phone, SSN, credit card) + // Runs unconditionally; always precedes ML classification } } ``` -## Content Isolation Boundary +Config: +```toml +[security.pii_filter] +enabled = true # Master switch for PII redaction (default: true) +filter_email = true # Scrub email addresses +filter_phone = true # Scrub US phone numbers +filter_ssn = true # Scrub US Social Security Numbers +filter_credit_card = true # Scrub credit card numbers +filter_names = false # Scrub personal names via heuristic (opt-in, default: false) +# custom_patterns = [] # Custom regex patterns on top of built-ins +``` + +## Secret Shape Masking -Web-fetched and tool-output content lives in isolated context: +`SecretMaskRegistry` (`crates/zeph-sanitizer/src/secret_mask.rs`) masks vault-secret placeholders at the LLM boundary to prevent leakage via side-channel analysis: ```rust -pub struct IsolatedContent { - // Content is NOT directly visible to LLM prompts - original: String, - scanned: bool, - ipi_confidence: f32, - pii_redacted: bool, - source: ContentSource, +pub enum SecretCategory { + ApiKey, + Token, + Password, + Certificate, + Webhook, + Generic, } -impl IsolatedContent { - async fn release_to_agent( - &self, - detector: &IpiDetector, - pii_detector: &PiiDetector, - ) -> Result { - if !self.scanned { - // Scan - let scan = detector.scan_content(&self.original).await?; - if scan.confidence > 0.75 { - return Err(anyhow!( - "Content flagged as likely IPI (confidence: {:.0}%)", - scan.confidence * 100.0 - )); - } - } - - // Redact PII - let redacted = pii_detector.redact_pii(&self.original).await?; - - Ok(redacted) - } -} +pub struct SecretMaskRegistry { /* ... */ } -#[derive(Debug, Clone, Copy)] -pub enum ContentSource { - WebFetch, - McpToolOutput, - UserUpload, - Email, +impl SecretMaskRegistry { + /// Mask all vault secret references in text before LLM inference. + pub fn mask(&self, text: &str) -> String { + // Replaces secret values with [MASKED_] placeholders + // Preserves schema/structure for debugging + } } ``` -## User Verification Gate +## Guardrail Filter (Optional Gating) -High-confidence IPI blocks agent until user approves: +`GuardrailFilter` provides an optional LLM-based pre-screener at the input boundary. When external content passes through, it is classified as SAFE/UNSAFE via Llama Guard classifier before being added to context: ```rust -async fn handle_suspicious_content( - detector: &IpiDetector, - content: &str, -) -> Result { - let scan = detector.scan_content(content).await?; - - if scan.confidence > 0.75 { - // Block and ask user - println!( - "⚠️ Suspicious content detected (confidence: {:.0}%)", - scan.confidence * 100.0 - ); - println!("Preview: {}", scan.content_preview); - println!("Source: {:?}", scan.detected_by); - println!("Should I use this content?"); - - let user_approval = /* await user input */; - if !user_approval { - return Err(anyhow!("User rejected suspicious content")); - } +pub struct GuardrailFilter { /* ... */ } + +impl GuardrailFilter { + /// Screen content via LLM guardrail classifier before adding to context. + pub async fn screen(&self, text: &str) + -> Result { + // Returns SAFE, UNSAFE, or UNKNOWN with confidence + // High-confidence UNSAFE blocks content from context } - - Ok(content.to_string()) } ``` -## Configuration - +Config: ```toml -[security.injection_defense] -enabled = true -regex_patterns_enabled = true -deberta_enabled = true -confidence_threshold = 0.75 # require verification above this - -# PII Detection -pii_detection_enabled = true -sensitive_types = ["ssn", "credit_card", "email", "phone"] -redact_mode = "mask" # or "block" +[security.guardrail] +enabled = false # Enable LLM-based guardrail classifier (default: false) +# provider = "ollama" # LLM provider for guardrail calls +# model = "llama-guard-3:1b" # Model to use for classification +timeout_ms = 500 # Timeout for each guardrail call (milliseconds) +action = "block" # Action on flagged content: "block" or "warn" +fail_strategy = "closed" # On timeout/LLM error: "open" (allow) or "closed" (block) +scan_tool_output = false # Scan tool outputs before context (default: false) +max_input_chars = 4096 # Max chars sent to guard model (default: 4096) ``` +## Quarantine & Dual-LLM Extraction + +`QuarantinedSummarizer` (`crates/zeph-sanitizer/src/quarantine.rs`) applies a Dual-LLM approach: one LLM processes untrusted content in isolation; a second LLM summarizes into trusted context. This prevents the agent from seeing potentially malicious instructions directly. + ## Integration Points -- [[008-mcp/spec]] — MCP tool outputs scanned before reaching agent -- [[025-classifiers/spec]] — DeBERTa inference infrastructure -- [[010-4-audit]] — IPI detection logged for compliance -- WebFetch tool — Content scanned before returning +- [[008-mcp/spec]] — MCP tool sanitization via `sanitize_tools()` +- [[025-classifiers/spec]] — DeBERTa (IPI), Candle PII NER (optional classifiers feature) +- [[010-4-audit]] — Audit signals (`AuditSignal`, `AuditSignalType`) ingested by `TrajectoryRiskAccumulator` +- WebScrape tool — Content sanitized before returning +- Context assembly — Spotlight wrapping applied per source trust level ## See Also -- [[010-security/spec]] — Parent -- [[010-1-vault]] — Prevent secret leakage from redaction -- [[025-classifiers/spec]] — DeBERTa and NER models -- [[010-4-audit]] — Audit log of IPI detections +- [[010-security/spec]] — Parent; cross-turn NEVER rule for `CrossToolCorrelator` +- [[010-1-vault]] — Prevent secret leakage via masking +- [[025-classifiers/spec]] — Classifier infrastructure (DeBERTa, PII NER) diff --git a/specs/010-security/010-3-authorization.md b/specs/010-security/010-3-authorization.md index ff4301fb3..f5e4b6658 100644 --- a/specs/010-security/010-3-authorization.md +++ b/specs/010-security/010-3-authorization.md @@ -23,16 +23,6 @@ related: # Spec: Authorization & Capability-Based Access Control -> [!danger] Stale content — 2026-07 audit -> `AuthPolicy`, `ShellSandbox`, `SsrfValidator` below are fictional — zero matches in the codebase — -> and were not corrected in this pass (rewrite, not a text fix). The code samples also chain -> `.unwrap()` on `Regex::new(...)`, which directly violates -> [[001-system-invariants/spec#11. Error Handling Contract]] ("NEVER use `panic!()`... no `unwrap()` -> in production paths") — do not copy that pattern. Real mechanisms live in the shell blocklist -> (`crates/zeph-tools/src/filter/security.rs` and the shell executor's `PermissionPolicy` gate) and -> the SSRF `validate_url` path referenced correctly in [[010-security/spec]] and -> [[010-5-egress-logging]] — treat those as authoritative over this file. - Permission policy enforcement, shell sandbox blocklist, SSRF protection, tool authorization. ## Overview @@ -42,234 +32,150 @@ Zeph's authorization layer enforces what operations the agent is allowed to perf ## Key Invariants **Always:** -- All tool execution requires authorization check against policy -- Shell commands checked against blocklist before execution -- HTTP requests checked for SSRF patterns (localhost, private ranges) -- Authorization failures logged with full context +- All tool execution checked against `PolicyEnforcer` deny/allow rules before execution +- Shell commands checked against the blocklist unconditionally — **before** `PermissionPolicy` evaluation +- HTTP requests validated via `validate_url()` — private IP ranges blocked by default +- Authorization failures logged to audit trail with full context **Never:** -- Bypass authorization checks for "trusted" tools +- Bypass blocklist checks for "trusted" tools — blocklist is unconditional - Allow shell execution without sandbox validation -- Make HTTP requests to private IP ranges without explicit allow-list +- Make HTTP requests to private IP ranges without explicit allowlist -## Capability-Based Access Control +## Declarative Policy Compiler -Policies define which agents can execute which tools: +`PolicyEnforcer` (`crates/zeph-tools/src/policy.rs`) evaluates TOML-based access-control rules with deny-first semantics: ```rust -#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)] -pub enum Capability { - ToolRead, // read-only tools - ToolWrite, // file write, API modify - ShellExecute, // shell commands - HttpRequest, // HTTP/HTTPS requests - NetworkListen, // open ports - ProcessManage, // spawn/kill processes -} +pub struct PolicyEnforcer { /* ... */ } -pub struct AuthPolicy { - // agent_id → capabilities - grants: HashMap>, - // capability → individual tool allows - tool_overrides: HashMap>, +#[non_exhaustive] +pub enum PolicyDecision { + Allow { trace: String }, // Rule matched; execution allowed + Deny { trace: String }, // Rule matched; execution denied } -impl AuthPolicy { - async fn check_authorization( +impl PolicyEnforcer { + /// Evaluate policy rules against tool call context. + /// + /// Deny rules checked first. If a deny rule matches, returns `Deny`. + /// Otherwise, checks allow rules. If no rule matches, uses `default_effect`. + pub fn evaluate( &self, - agent_id: &str, tool_name: &str, - required_capability: Capability, - ) -> Result<()> { - // 1. Check if agent has capability - let capabilities = self.grants - .get(agent_id) - .ok_or_else(|| anyhow!("Agent {} not in policy", agent_id))?; - - if !capabilities.contains(&required_capability) { - return Err(anyhow!( - "Agent {} lacks capability {:?}", - agent_id, - required_capability - )); - } - - // 2. Check tool-level override (e.g., "shell_execute" allowed but "rm -rf" blocked) - if let Some(allowed_tools) = self.tool_overrides.get(tool_name) { - if !allowed_tools.contains(agent_id) { - return Err(anyhow!( - "Tool '{}' not in allow-list for agent {}", - tool_name, - agent_id - )); - } - } - - Ok(()) - } + params: &serde_json::Map, + context: &PolicyContext, + ) -> PolicyDecision { /* ... */ } +} + +pub struct PolicyContext { + pub trust_level: SkillTrustLevel, + pub env: std::collections::HashMap, } ``` -## Shell Sandbox +**Rule matching** (all conditions are AND'd): +- `effect`: `"allow"` or `"deny"` +- `tool`: glob pattern matching tool name (e.g., `"read_*"`, `"rm"`) +- `paths`: glob patterns matched against path-like parameters; rule fires if ANY matches +- `env`: environment variable names that must ALL be present for rule to apply +- `trust_level`: minimum required trust level (`Trusted` > `Neutral` > `Untrusted`) +- `args_match`: regex matched against individual string parameter values +- `capabilities`: named capabilities associated with this rule (for auditing/metadata) -Blocklist of dangerous commands: +**Semantics**: deny rules are evaluated first; matching deny → `Deny`. If no deny matches, evaluate allow rules; matching allow → `Allow`. If no rule matches, use `default_effect` (typically `Deny`). -```rust -pub struct ShellSandbox { - blocklist: Vec, -} +**Config example**: +```toml +[[tools.policy]] +effect = "deny" +tool = "rm" # Glob pattern +# Blocks all rm invocations (handled specially by shell blocklist anyway) + +[[tools.policy]] +effect = "allow" +tool = "read_file" +paths = ["/home/*/documents/*"] # Glob patterns on paths +trust_level = "trusted" # Only for trusted skill callers +``` -#[derive(Clone)] -pub struct ShellPattern { - pattern: Regex, - reason: &'static str, -} +## Shell Sandbox Blocklist -impl ShellSandbox { - fn new() -> Self { - Self { - blocklist: vec![ - // Destructive commands - ShellPattern { - pattern: Regex::new(r"^\s*(rm|rmdir|dd)\s+-rf").unwrap(), - reason: "recursive deletion blocked", - }, - ShellPattern { - pattern: Regex::new(r":(){ :|:|");").unwrap(), - reason: "fork bomb detected", - }, - // Privilege escalation - ShellPattern { - pattern: Regex::new(r"^sudo\s+|/etc/sudoers").unwrap(), - reason: "privilege escalation blocked", - }, - // System modification - ShellPattern { - pattern: Regex::new(r"^\s*(chmod|chown|passwd|usermod)").unwrap(), - reason: "system modification blocked", - }, - ], - } - } - - fn validate_command(&self, cmd: &str) -> Result<()> { - for pattern in &self.blocklist { - if pattern.pattern.is_match(cmd) { - return Err(anyhow!("{}", pattern.reason)); - } - } - Ok(()) - } -} +`ShellExecutor` enforces an unconditional blocklist before spawning shell commands. The blocklist runs **before** PermissionPolicy evaluation: + +**Hardcoded blocklist** (`crates/zeph-tools/src/shell/mod.rs`): ``` +sudo, mkfs, dd if=, curl, wget, nc, ncat, netcat, shutdown, reboot, halt +``` + +Any invocation containing one of these patterns (case-sensitive, as a substring or token) is blocked unconditionally. + +**Special handling for `rm`**: +- `rm -rf` is **allowed** only when all three conditions are met: + - Operating on relative paths (e.g., `rm -rf ./tempdir`) + - NOT on `.git/worktrees`, root, or `$HOME` + - NOT on absolute paths +- Example: `rm -rf /` is blocked; `rm -rf ./temp` is allowed (subject to policy checks) + +**Limitations** (documented in code): +- Bypass via indirect invocation: `bash -c "rm ..."` — the `-c` argument is not scanned for blocked patterns +- Bypass via variable indirection: `cmd=rm; $cmd file` — the shell variable is not expanded for scanning +- The blocklist is a **first-pass defense**, not comprehensive; it mitigates common attacks but does not prevent all privilege escalation attempts + +Blocklist validation is unconditional; all other shell commands then pass through `PolicyEnforcer` for fine-grained access control. ## SSRF Protection -Prevent requests to internal services: +`validate_url()` (`crates/zeph-tools/src/net.rs`) blocks requests to private IP ranges and loopback addresses: ```rust -pub struct SsrfValidator { - blocked_ranges: Vec, - allow_list: Vec, // explicit allowed domains -} +/// Validate a URL for SSRF attacks. +/// +/// Blocks all private IP ranges, localhost, link-local, and non-HTTPS schemes: +/// - IPv4: 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16, 169.254.0.0/16, 127.0.0.0/8, 0.0.0.0, 255.255.255.255 +/// - IPv6: ::1, fc00::/7, fe80::/10 +/// - Schemes: only `https://` allowed; `http://`, `file://`, `ftp://`, `data://`, `javascript://` blocked +pub fn validate_url(raw: &str) -> Result; +``` + +**Redirect handling**: each redirect target is validated independently — a redirect chain where any hop points to a private IP fails at that hop. + +**Applied to**: +- `WebScrapeExecutor` (`scrape` tool) — validates URLs before fetching; validates each redirect target +- Redirect chains: all hops must pass validation; a 302 to `http://localhost/` is caught and fails + +**Allowlist**: no allowlist exists for SSRF validation; the deny-all private-IP policy is not configurable. -impl SsrfValidator { - fn new() -> Self { - Self { - blocked_ranges: vec![ - // Private IPv4 - "127.0.0.0/8".parse().unwrap(), // loopback - "169.254.0.0/16".parse().unwrap(), // link-local - "10.0.0.0/8".parse().unwrap(), // private - "172.16.0.0/12".parse().unwrap(), // private - "192.168.0.0/16".parse().unwrap(), // private - // IPv6 - "::1/128".parse().unwrap(), // loopback - "fc00::/7".parse().unwrap(), // private - "fe80::/10".parse().unwrap(), // link-local - ], - allow_list: vec![], - } - } - - async fn validate_url(&self, url: &str) -> Result<()> { - let parsed = url::Url::parse(url)?; - - // 1. Check allow-list first - if let Some(domain) = parsed.domain() { - if self.allow_list.contains(&domain.to_string()) { - return Ok(()); - } - } - - // 2. Resolve hostname - let addr = tokio::net::lookup_host( - format!("{}:{}", parsed.host_str().ok_or("no host")?, - parsed.port().unwrap_or(443)) - ).await? - .next() - .ok_or("hostname resolution failed")?; - - // 3. Check IP against blocked ranges - for range in &self.blocked_ranges { - if range.contains(addr.ip()) { - return Err(anyhow!( - "SSRF blocked: {} resolves to private IP {}", - parsed.host_str().unwrap_or("?"), - addr.ip() - )); - } - } - - Ok(()) - } +## Credential Environment Variable Scrubbing + +`ShellExecutor` filters environment variables via a configurable `env_blocklist` before spawning subprocess commands: + +```rust +pub struct ShellExecutor { + env_blocklist: Vec, // Env var names to strip from subprocess (prefix match) } ``` -## Configuration +The blocklist is applied inline during command construction: +1. User provides extra env vars (e.g., API keys for a script) +2. ShellExecutor constructs subprocess env by filtering the parent's env + extra vars +3. Any env var matching a prefix in `env_blocklist` is stripped -```toml -[security.authorization] -# Capability grants per agent -[[security.authorization.agents]] -id = "primary_agent" -capabilities = ["ToolRead", "ToolWrite", "HttpRequest"] - -[[security.authorization.agents]] -id = "sandbox_agent" -capabilities = ["ToolRead"] - -# Tool allow-lists (tool → allowed agents) -[security.authorization.tool_overrides] -shell_execute = ["primary_agent"] -file_delete = ["primary_agent"] -network_listen = [] - -# Shell sandbox -[security.sandbox] -enabled = true -blocklist = [ - "^\\s*(rm|rmdir)\\s+-rf", - ":(){ :|:|;", - "^sudo\\s+", -] - -# SSRF protection -[security.ssrf] -enabled = true -blocked_ranges = ["127.0.0.0/8", "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"] -allow_list = ["api.example.com", "internal.trusted.service"] -``` +**Default blocklist** (from config `[tools.shell].env_blocklist`): +- Typically includes: `ZEPH_*`, `AWS_*`, `GITHUB_*`, `OPENAI_*`, `ANTHROPIC_*`, etc. + +Blocklist filtering is unconditional and applied at subprocess spawn time, not at config time. MCP stdio servers inherit the same env filtering policy. ## Integration Points -- [[006-tools/spec]] — Tool execution calls authorization check -- [[008-3-security]] — OAP authorization for MCP tools -- [[010-4-audit]] — Authorization failures logged +- [[008-mcp/spec]] — MCP tool allowlist + policy enforcement +- [[006-tools/spec]] — tool registry + authorization binding +- [[010-4-audit]] — authorization violations logged to audit trail +- Web executor — SSRF validation on all HTTP requests +- MCP OAuth — OAuth flows validated via `validate_oauth_metadata_urls()` ## See Also -- [[010-security/spec]] — Parent -- [[006-tools/spec]] — ToolExecutor enforces authorization -- [[008-3-security]] — OAP authorization for MCP -- [[010-4-audit]] — Audit trail of authorization checks +- [[010-security/spec]] — Parent; shell blocklist and SSRF noted as unconditional +- [[010-2-injection-defense]] — guardrail filtering runs before tool dispatch +- [[008-3-security]] — MCP tool allowlist enforcement diff --git a/specs/010-security/010-4-audit.md b/specs/010-security/010-4-audit.md index 7e00aa570..347973a11 100644 --- a/specs/010-security/010-4-audit.md +++ b/specs/010-security/010-4-audit.md @@ -2,8 +2,8 @@ aliases: - Audit Trail - Security Logging - - Cross-Tool Injection Correlation - - Env-Var Scrubbing + - Tool Execution Audit + - Authorization Logging tags: - sdd - spec @@ -16,365 +16,164 @@ related: - "[[010-1-vault]]" - "[[010-2-injection-defense]]" - "[[010-3-authorization]]" + - "[[010-5-egress-logging]]" --- # Spec: Audit Trail & Security Logging -> [!danger] Stale content — 2026-07 audit -> The `AuditEntry`/`AuditEventType`/`AuditStatus`/`AuditLogger` schema below is fictional — the real -> `AuditEntry` (`crates/zeph-tools/src/audit.rs`) is tool-execution-centric (`tool`, `command`, -> `result`, `duration_ms`, `error_category`, `caller_id`, `correlation_id`, `vigil_risk`, ...), not -> the generic `id`/`agent_id`/`event_type`/`resource`/`action`/`status` shape shown here. See -> [[010-5-egress-logging]] for the accurate, current `AuditEntry`/`AuditLogger` description. The -> "Cross-Tool Injection Correlation" section (`InjectionCorrelator`, time-windowed accumulation) -> directly **contradicts** [[010-security/spec]]'s authoritative `CrossToolCorrelator`, which states -> cross-turn signal accumulation is NEVER performed and is cleared every turn — do not implement -> against this section without reconciling that contradiction first. Not corrected in this pass -> (needs a mechanism-level rewrite, not a text fix). - -AgentRFC protocol audit, cross-tool injection correlation, environment variable scrubbing, compliance logging. +Tool execution auditing, authorization logging, security event correlation, compliance logging. ## Overview -Zeph maintains an immutable audit trail of all security-relevant events: tool invocations, authorization decisions, IPI detections, vault accesses. This log is used for compliance, incident investigation, and pattern detection (e.g., correlated injection attempts across multiple tools). +Zeph maintains an immutable audit trail of security-relevant events: tool invocations, authorization decisions, IPI detections, shell command executions. This log is used for compliance, incident investigation, and pattern detection. ## Key Invariants **Always:** -- All tool invocations logged with: tool name, input (sanitized), output (truncated), status, latency -- All authorization failures logged with: agent, tool, capability, policy check details -- All IPI detections logged with: confidence, source, content preview, user action -- All vault accesses logged with: key name (not value), action (get/set), success/failure -- Audit log persists to disk (SQLite or JSON lines format) +- All tool invocations logged with: tool name, input (sanitized), output (preview), status, duration +- All authorization denials logged with: agent/skill, tool, reason, policy rule matched +- All shell command execution logged with: command (sanitized), exit code, duration +- All secrets redacted from logs — never log API keys, vault key names, PII +- Audit log persists immutably to disk **Never:** -- Log secret values, API keys, or PII (always sanitize) -- Truncate or modify audit entries after creation (immutable log) -- Disable audit logging even in debug mode +- Log secret values, API keys, or PII directly — always sanitize +- Truncate or modify audit entries after creation +- Disable audit logging, even in debug/testing modes + +## Tool Execution Audit -## Audit Entry Schema +Tool executor logs every call to an `AuditEntry`: ```rust -#[derive(Serialize, Deserialize, Debug)] pub struct AuditEntry { - id: String, // UUID v4 - timestamp: i64, // unix epoch seconds - agent_id: String, - event_type: AuditEventType, - resource: String, // tool name, endpoint, etc. - action: String, // "invoke", "deny", "detect", "access" - status: AuditStatus, // "success", "denied", "error" - details: serde_json::Value, // event-specific details - correlation_id: String, // trace across related events -} - -#[derive(Serialize, Deserialize, Debug)] -pub enum AuditEventType { - ToolInvocation, - AuthorizationCheck, - IpiDetection, - VaultAccess, - ProtocolError, -} - -#[derive(Serialize, Deserialize, Debug)] -pub enum AuditStatus { - #[serde(rename = "success")] - Success, - #[serde(rename = "denied")] - Denied, - #[serde(rename = "error")] - Error, + pub timestamp: String, // Unix timestamp (seconds) when invocation started + pub tool: ToolName, // Tool identifier (e.g., "shell", "web_scrape") + pub command: String, // Human-readable command or URL being invoked + pub result: AuditResult, // Outcome of the invocation (success/failure) + pub duration_ms: u64, // Wall-clock duration in milliseconds + pub error_category: Option, // Fine-grained error category label (if failed) + pub error_domain: Option, // High-level error domain for recovery + pub error_phase: Option, // Invocation phase where error occurred + pub claim_source: Option, // Provenance of tool result + pub mcp_server_id: Option, // MCP server ID (if routed through McpToolExecutor) + pub injection_flagged: bool, // Tool output flagged by regex injection detection + pub embedding_anomalous: bool, // Tool output flagged by embedding guard anomaly + pub cross_boundary_mcp_to_acp: bool, // Tool result crossed MCP-to-ACP trust boundary + pub adversarial_policy_decision: Option, // Adversarial policy decision + pub exit_code: Option, // Process exit code for shell executions + pub truncated: bool, // Whether output was truncated before storage + pub caller_id: Option, // Caller identity that initiated this call + pub policy_match: Option, // Policy rule trace that matched + pub correlation_id: Option, // Correlation ID shared with associated EgressEvent + pub vigil_risk: Option, // VIGIL risk level when gate flagged output } ``` -## Tool Invocation Audit +Sanitization before logging: +- Input/output truncated before storage if oversized +- Known secret keys redacted: `api_key`, `token`, `password`, `secret`, `auth`, `key` +- Vault-resolved secrets never logged +- PII scrubbed via `[security.pii_filter]` regex patterns (SSN, credit card, email) -Log every tool execution: - -```rust -async fn audit_tool_invocation( - logger: &AuditLogger, - tool_name: &str, - input: &Value, - output: &Value, - status: ExecutionStatus, - latency_ms: u64, -) -> Result<()> { - let entry = AuditEntry { - id: uuid::Uuid::new_v4().to_string(), - timestamp: now(), - agent_id: "primary".to_string(), - event_type: AuditEventType::ToolInvocation, - resource: tool_name.to_string(), - action: "invoke".to_string(), - status: match status { - ExecutionStatus::Success => AuditStatus::Success, - ExecutionStatus::Error => AuditStatus::Error, - }, - details: json!({ - "input": sanitize_for_logging(input), - "output_preview": truncate_output(output, 200), - "latency_ms": latency_ms, - "status_code": 200, // if HTTP - }), - correlation_id: correlation_context::get_trace_id(), - }; - - logger.log(entry).await?; - Ok(()) -} - -fn sanitize_for_logging(value: &Value) -> Value { - // Redact keys known to contain secrets - match value { - Value::Object(map) => { - let mut sanitized = map.clone(); - for key in ["api_key", "password", "token", "secret", "auth"] { - if sanitized.contains_key(key) { - sanitized.insert( - key.to_string(), - Value::String("[REDACTED]".to_string()), - ); - } - } - Value::Object(sanitized) - } - _ => value.clone(), - } -} +**Configuration** (`[tools.audit]`): +```toml +[tools.audit] +enabled = true # Enable audit logging (default: true) +destination = "stdout" # Log destination: "stdout", "stderr", or file path +# tool_risk_summary = false # Log per-tool risk summary at startup (default: false) ``` -## Authorization Audit +## Authorization & Shell Execution Details -Log all permission checks: +Authorization denials and shell execution details are captured within the unified `AuditEntry` structure: +- **Authorization**: `AuditEntry.policy_match` contains the matched policy rule; `AuditEntry.adversarial_policy_decision` records the decision (`allow`/`deny:`/`error:`) +- **Shell execution**: `AuditEntry.command` is the sanitized shell command; `AuditEntry.exit_code` is the process exit code; `AuditEntry.duration_ms` is wall-clock execution time -```rust -async fn audit_authorization( - logger: &AuditLogger, - agent_id: &str, - tool_name: &str, - capability: &str, - allowed: bool, -) -> Result<()> { - let entry = AuditEntry { - id: uuid::Uuid::new_v4().to_string(), - timestamp: now(), - agent_id: agent_id.to_string(), - event_type: AuditEventType::AuthorizationCheck, - resource: tool_name.to_string(), - action: "check".to_string(), - status: if allowed { - AuditStatus::Success - } else { - AuditStatus::Denied - }, - details: json!({ - "required_capability": capability, - "decision": if allowed { "allow" } else { "deny" }, - }), - correlation_id: correlation_context::get_trace_id(), - }; - - logger.log(entry).await?; - Ok(()) -} -``` +All command text is sanitized before logging: `sudo`, passwords, and secret env vars are redacted from the `command` field. ## IPI Detection Audit -Log all injection attempts: +Injection detection results logged via `AuditSignal`: ```rust -async fn audit_ipi_detection( - logger: &AuditLogger, - source: &str, // "web_fetch", "mcp_output", etc. - confidence: f32, - pattern_matched: &str, - user_action: &str, // "approved", "rejected", "blocked" -) -> Result<()> { - let entry = AuditEntry { - id: uuid::Uuid::new_v4().to_string(), - timestamp: now(), - agent_id: "security".to_string(), - event_type: AuditEventType::IpiDetection, - resource: source.to_string(), - action: "detect".to_string(), - status: match user_action { - "approved" => AuditStatus::Success, - _ => AuditStatus::Denied, - }, - details: json!({ - "confidence": format!("{:.2}%", confidence * 100.0), - "pattern": pattern_matched, - "user_action": user_action, - }), - correlation_id: correlation_context::get_trace_id(), - }; - - logger.log(entry).await?; - Ok(()) +pub enum AuditSignalType { + PolicyViolation, // Tool blocked by policy + PromptInjectionPattern, // Regex pattern matched + ToolChainAnomaly, // Unexpected tool sequence + ConfidenceDrop, // LLM confidence dropped } -``` - -## Cross-Tool Injection Correlation -Detect patterns across multiple tool invocations: - -```rust -pub struct InjectionCorrelator { - recent_detections: Arc>>, - correlation_window: Duration, +pub enum Severity { + Low, // Minor concern + Medium, // Warrants tracking + High, // Strong indicator } -impl InjectionCorrelator { - async fn check_correlation( - &self, - new_detection: &IpiDetection, - ) -> Result> { - let mut detections = self.recent_detections.lock().await; - - // Remove stale detections - let cutoff = now() - self.correlation_window.as_secs(); - while !detections.is_empty() && detections.front().unwrap().timestamp < cutoff { - detections.pop_front(); - } - - // Check for patterns - let same_pattern_count = detections - .iter() - .filter(|d| d.pattern_matched == new_detection.pattern_matched) - .count(); - - if same_pattern_count >= 3 { - // Same injection pattern detected 3+ times recently - return Ok(Some(InjectionCluster { - pattern: new_detection.pattern_matched.clone(), - count: same_pattern_count + 1, - sources: detections - .iter() - .map(|d| d.source.clone()) - .collect(), - })); - } - - detections.push_back(new_detection.clone()); - Ok(None) - } +pub struct AuditSignal { + pub signal_type: AuditSignalType, + pub severity: Severity, } ``` -## Environment Variable Scrubbing +> [!warning] Architectural gap +> `TrajectoryRiskAccumulator` maintains a **per-session, cross-turn risk score with exponential decay** and makes **hard-blocking tool-execution decisions** when risk exceeds a threshold. This appears to violate or at least is not addressed by the parent spec's NEVER rule, which forbids cross-turn accumulation "for injection-confirmation decisions." The spec carves out an exception for `TrajectorySentinel` (advisory-only, reversible decay), but does not discuss `TrajectoryRiskAccumulator` at all. Whether this system should be governed by the NEVER rule or falls outside its scope (because its decisions are general safety-gating, not injection-confirmation) is an open architectural question that needs resolution. -Remove secrets from subprocess environment: +## Turn Boundary Isolation & Signal Accumulation + +One signal-processing system operates at the session scope: + +**`TrajectoryRiskAccumulator`** — accumulates signals **per-session, across turn boundaries** with exponential decay: ```rust -fn scrub_environment(env: &HashMap) -> HashMap { - let secret_prefixes = vec![ - "ZEPH_", - "OPENAI_API", - "ANTHROPIC_API", - "AWS_", - "GCP_", - "GITHUB_TOKEN", - "SLACK_", - "TELEGRAM_", - "DATABASE_PASSWORD", - ]; - - let mut scrubbed = HashMap::new(); - - for (key, value) in env { - let is_secret = secret_prefixes - .iter() - .any(|prefix| key.to_uppercase().starts_with(prefix)); - - if is_secret { - log::debug!("Scrubbing env var: {}", key); - // Don't include in subprocess environment - continue; - } - - scrubbed.insert(key.clone(), value.clone()); - } - - scrubbed +pub struct TrajectoryRiskAccumulator { + // Maintains trajectory_risk: [0.0, 1.0] + // Decays exponentially between turns + // Makes hard-blocking decisions when risk >= threshold +} + +impl TrajectoryRiskAccumulator { + pub fn is_blocked(&self) -> bool; // Hard block: risk >= threshold + pub fn ingest(&mut self, signal: &AuditSignal); // Cross-turn accumulation + pub fn advance_turn(&mut self); // Apply exponential decay at turn boundary } ``` -## Audit Log Storage +**Status**: Not explicitly addressed in parent spec's NEVER rule. Whether this system's cross-turn, hard-blocking behavior is intended carve-out (like `TrajectorySentinel`) or an overlooked violation remains unresolved. + +## Audit Log Storage & Serialization -Immutable persistence: +Immutable persistence via `AuditLogger` (`crates/zeph-tools/src/audit.rs:110`), which serializes entries as flat JSON objects (newline-terminated JSONL format): ```rust -pub struct AuditLogger { - db: Arc, // immutable log table -} +pub struct AuditLogger { /* destination: AuditDestination */ } impl AuditLogger { - async fn log(&self, entry: AuditEntry) -> Result<()> { - sqlx::query( - "INSERT INTO audit_log ( - id, timestamp, agent_id, event_type, resource, - action, status, details, correlation_id - ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)" - ) - .bind(&entry.id) - .bind(entry.timestamp) - .bind(&entry.agent_id) - .bind(format!("{:?}", entry.event_type)) - .bind(&entry.resource) - .bind(&entry.action) - .bind(format!("{:?}", entry.status)) - .bind(serde_json::to_string(&entry.details)?) - .bind(&entry.correlation_id) - .execute(&*self.db) - .await?; - - Ok(()) - } - - async fn query_by_correlation( - &self, - correlation_id: &str, - ) -> Result> { - sqlx::query_as::<_, AuditEntry>( - "SELECT * FROM audit_log WHERE correlation_id = ? ORDER BY timestamp" - ) - .bind(correlation_id) - .fetch_all(&*self.db) - .await - .context("audit query failed") - } + /// Log a single audit entry asynchronously. + pub async fn log(&self, entry: &AuditEntry); } ``` -## Configuration +**Destination types** (`AuditDestination`): +- **Stdout** (default): entries written to standard output, one JSON line per entry +- **Stderr**: entries written to standard error +- **File**: entries written to a file path (created with `0o600` permissions on Unix) -```toml -[security.audit] -enabled = true -log_level = "info" # "debug", "info", "warn" -backend = "sqlite" # or "jsonl" -path = ".local/audit.db" - -# Retention -retention_days = 90 # after which entries can be archived -archive_path = ".local/audit.archive.jsonl" - -# Sampling (reduce volume in production) -sample_rate = 1.0 # log all events; set < 1.0 to sample -``` +All entries are serialized as compact JSON objects (newline-terminated JSONL). Optional fields are omitted to keep entries compact. + +## Vault Access Logging + +Vault reads/writes are NOT logged to audit trail (to prevent metadata leakage), but are tracked via in-memory metrics. Failed vault lookups are logged with redacted key names. ## Integration Points -- [[006-tools/spec]] — Tool execution audited here -- [[010-1-vault]] — Vault accesses audited -- [[010-2-injection-defense]] — IPI detections audited -- [[010-3-authorization]] — Authorization checks audited +- [[006-tools/spec]] — Tool execution audited +- [[010-1-vault]] — Credential access tracked (metadata-safe) +- [[010-2-injection-defense]] — IPI `AuditSignal` ingested by trajectory accumulator +- [[010-3-authorization]] — Authorization violations logged +- [[010-5-egress-logging]] — HTTP/webhook egress events logged ## See Also -- [[010-security/spec]] — Parent -- [[010-2-injection-defense]] — IPI detection events -- [[010-3-authorization]] — Authorization check events +- [[010-security/spec]] — Parent; cross-turn NEVER rule for hard decisions +- [[010-5-egress-logging]] — HTTP egress audit (`EgressEvent`)