diff --git a/CHANGELOG.md b/CHANGELOG.md index 98f44af..6ce05be 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -16,6 +16,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - All built-in adapters and public fake runtimes now validate returned values against `output_schema` locally, including textual JSON fallbacks. - `AgentResult.is_success` provides one canonical success predicate. +- `TaskSupportReport`, `TaskSupportProvider`, and `validate_task()` provide + side-effect-free task preflight without adding a requirement to the + `AgentRuntime` protocol. Registry and `AgentKit` helpers preserve custom + runtime validation and capability-based fallback for older runtimes. +- `COMPATIBILITY_MANIFEST` records each built-in adapter's install extra, + import module, accepted SDK range, tested lockfile version, and runtime binary + versions such as `openai-codex-cli-bin`. ### Changed @@ -26,6 +33,10 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 duplicate MCP names, and conflicting tool filters. - BREAKING: unknown `Usage` values, including cost, are `None` rather than zero; an explicit provider-reported zero remains `0`/`0.0`. +- Capability declarations now distinguish budget, reasoning-effort, network, + tool-filter, and per-MCP-server environment support. Built-in `run()` methods + consume the same support report used by preflight, eliminating divergent + rejection paths. ### Fixed @@ -39,6 +50,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Permission- and budget-critical vendor options must now be explicit, introspectable SDK parameters; opaque `**kwargs`, positional-only options, and uninspectable callables fail closed. +- Configured model allow-lists, Antigravity MCP name syntax, disjoint + allow/deny lists, and Antigravity's legacy reasoning-effort alias are rejected + during static preflight instead of surfacing later or being silently ignored. ## 0.4.0 - 2026-07-02 diff --git a/README.md b/README.md index a2c0d63..a0e1960 100644 --- a/README.md +++ b/README.md @@ -141,6 +141,14 @@ a field (for example only Claude maps `budget_usd`; Codex and Antigravity reject it with a typed `UnsupportedTaskInputError`) the adapter raises rather than silently dropping it. +Call `validate_task(runtime, task)` (or `kit.validate_task("codex", task)`) to +inspect every statically detectable incompatibility before dispatch. The +returned `TaskSupportReport` is side-effect-free and lists source fields such as +`budget_usd` and `permissions.network`; `run()` still fails closed on the first +issue. Third-party runtimes can opt into provider-specific checks with the +`TaskSupportProvider` protocol, while older `AgentRuntime` implementations +continue to work through capability-based fallback checks. + `AgentResult` returns output, finish reason (see `FinishReason`), locally validated structured output, usage, cost, session id, tool-call audits, and provider metadata. `is_success` is true only for a natural, error-free diff --git a/docs/api-stability.md b/docs/api-stability.md index 5c10907..e302f29 100644 --- a/docs/api-stability.md +++ b/docs/api-stability.md @@ -44,6 +44,12 @@ import the names from the top-level package instead. a task field raises `UnsupportedTaskInputError`; the one exception is vendor-option drift, which is recorded in `AgentResult.metadata["dropped_options"]` instead. +- **Task support can be inspected without dispatch.** `validate_task(runtime, + task)`, `RuntimeRegistry.validate_task_for(...)`, and `AgentKit.validate_task(...)` + return every statically detectable incompatibility as a `TaskSupportReport`. + `TaskSupportProvider` is an optional extension, not a new requirement on the + `AgentRuntime` protocol, so existing third-party runtimes retain structural + compatibility and use capability-based fallback checks. - **`AgentKit` is sugar, not a second API.** The hub assembles the same frozen `AgentTask` and returns the same `AgentResult` the runtimes produce (`ParsedResult` is a runtime-identical subclass adding only the typed @@ -72,6 +78,13 @@ it reaches installed users; a release above a cap is by design invisible to that lane until the cap is raised. A separate lane installs every direct dependency at its declared floor so a stale minimum cannot sit undetected in the metadata. +`COMPATIBILITY_MANIFEST` is the machine-readable form of that policy. Each entry +records the install extra, import module, accepted SDK range, exact lockfile +version exercised by the repository, and any separately versioned runtime binary +(currently `openai-codex-cli-bin`). The manifest is tested against both +`pyproject.toml` and `uv.lock`; it is evidence of the committed test baseline, +not a claim that every version in the accepted range was exhaustively tested. + ## Deprecation When a public name is slated for removal it will be kept working for at least one diff --git a/docs/capability-matrix.md b/docs/capability-matrix.md index 90a49b9..98bf1bc 100644 --- a/docs/capability-matrix.md +++ b/docs/capability-matrix.md @@ -20,6 +20,13 @@ The matrix is intentionally not a lowest-common-denominator contract. Adapters reject unsupported inputs (see below) when silently dropping them would be misleading. +The public `AgentCapabilities` value also advertises the granular task controls +used by preflight: `budget`, `reasoning_effort`, `network_control`, +`tool_filters`, and `mcp_server_env`. Use `validate_task(runtime, task)` when a +concrete task is available; unlike a raw capability flag, its +`TaskSupportReport` preserves instance-specific rules such as a configured model +allow-list and reports all detectable issues at once. + For every provider, malformed schemas are rejected locally before dispatch and returned structured values are validated locally after the SDK completes. `parsed_output_available` distinguishes valid JSON `null` from no parsed value. @@ -63,6 +70,13 @@ tool, uses the SDK's `disabled_tools` route). Each adapter raises `UnsupportedTaskInputError` for task fields it has no SDK surface to honor, rather than dropping them silently. +The same fields can be checked before dispatch through `validate_task`, +`RuntimeRegistry.validate_task_for`, or `AgentKit.validate_task`. Built-in +adapters additionally preflight configured model allow-lists. Antigravity also +checks its MCP server-name syntax and the SDK's prohibition on combining an +allow-list with a deny-list. SDK-version-dependent Antigravity tool vocabulary +is still validated at dispatch, after the installed SDK supplies its enum. + | Field | Claude | Codex | Antigravity | |-------|--------|-------|-------------| | `budget_usd` | Mapped (`max_budget_usd`, fails closed under SDK drift) | Rejected | Rejected | diff --git a/docs/providers.md b/docs/providers.md index 3683f9a..df6793f 100644 --- a/docs/providers.md +++ b/docs/providers.md @@ -31,8 +31,8 @@ All three adapters map the task's system prompt (Claude `system_prompt`, Codex field (falling back to the `metadata` aliases of the same names). `reasoning_effort` maps to the Claude and Codex `effort` options; Antigravity has no reasoning-effort control and rejects the first-class field with a typed -error (its legacy `metadata["reasoning_effort"]` alias stays ignored, as it -always has been). Effort values are passed through to the vendor SDK rather +error. Its legacy `metadata["reasoning_effort"]` alias is rejected too, rather +than being silently ignored. Effort values are passed through to the vendor SDK rather than validated by this library — each vendor defines its own accepted vocabulary (for example `claude-agent-sdk` 0.2.x accepts `low`/`medium`/`high`/`xhigh`/`max`), and an SDK too old to accept `effort` at @@ -42,6 +42,12 @@ Usage fields are `None` when a provider omits its usage breakdown or cost. Provider-reported zeros remain numeric zero, so callers can distinguish a free or zero-token run from missing telemetry. +Every built-in adapter exposes a pure `validate_task()` extension. The public +`validate_task(runtime, task)`, registry, and `AgentKit` helpers call that richer +validator when present and otherwise fall back to declared capabilities for +older third-party runtimes. A report is a static preflight, not an availability +or credential probe; installed-SDK drift can still make dispatch fail closed. + Claude uses the `claude-agent-sdk` package and maps working directory, permissions, filesystem access (a `READ_ONLY` filesystem forces `plan` mode), MCP servers, sessions, structured output, tool allow/deny lists, runtime diff --git a/pyproject.toml b/pyproject.toml index c8d0140..07977f4 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -70,6 +70,7 @@ dev = [ "pytest-asyncio>=0.23", "pytest-cov>=5.0", "ruff>=0.8", + "tomli>=2; python_version < '3.11'", "types-jsonschema>=4.18", ] diff --git a/src/agent_runtime_kit/__init__.py b/src/agent_runtime_kit/__init__.py index 8409473..bd7648d 100644 --- a/src/agent_runtime_kit/__init__.py +++ b/src/agent_runtime_kit/__init__.py @@ -27,10 +27,19 @@ PermissionProfile, RuntimeAvailability, SessionResumeState, + TaskSupportIssue, + TaskSupportProvider, + TaskSupportReport, ToolCallAudit, Usage, runtime_kind_value, ) +from agent_runtime_kit.compatibility import ( + COMPATIBILITY_MANIFEST, + PackageVersion, + RuntimeCompatibility, + compatibility_for, +) from agent_runtime_kit.events import ( output_delta_event, safe_emit, @@ -42,6 +51,7 @@ vendor_turn_event, ) from agent_runtime_kit.registry import RuntimeRegistry, create_default_registry +from agent_runtime_kit.support import validate_task __all__ = [ "AgentCapabilities", @@ -59,20 +69,27 @@ "FilesystemAccess", "FinishReason", "KIND_ALIASES", + "COMPATIBILITY_MANIFEST", "McpServerConfig", "OutputSchemaError", "OutputTypeError", + "PackageVersion", "ParsedResult", "PermissionMode", "PermissionProfile", "RuntimeAvailability", + "RuntimeCompatibility", "RuntimeNotRegisteredError", "RuntimeRegistry", "SessionResumeState", + "TaskSupportIssue", + "TaskSupportProvider", + "TaskSupportReport", "ToolCallAudit", "UnsupportedTaskInputError", "Usage", "create_default_registry", + "compatibility_for", "runtime_kind_value", "output_delta_event", "safe_emit", @@ -82,4 +99,5 @@ "tool_completed_event", "tool_requested_event", "vendor_turn_event", + "validate_task", ] diff --git a/src/agent_runtime_kit/_kit.py b/src/agent_runtime_kit/_kit.py index be3de91..2268e7e 100644 --- a/src/agent_runtime_kit/_kit.py +++ b/src/agent_runtime_kit/_kit.py @@ -26,8 +26,10 @@ PermissionProfile, RuntimeAvailability, SessionResumeState, + TaskSupportReport, ) from agent_runtime_kit.registry import RuntimeFactory, RuntimeRegistry, create_default_registry +from agent_runtime_kit.support import validate_task as validate_runtime_task _T = TypeVar("_T") # An event handler receives one normalized event dict; sync or async. @@ -109,6 +111,21 @@ def availability_for(self, kind: AgentRuntimeKind | str) -> RuntimeAvailability: def capabilities_for(self, kind: AgentRuntimeKind | str) -> AgentCapabilities: return self._registry.capabilities_for(self._normalize_kind(kind)) + def validate_task( + self, + runtime: AgentRuntimeKind | str | AgentRuntime, + task: AgentTask, + ) -> TaskSupportReport: + """Report unsupported task fields without starting a run.""" + + if isinstance(runtime, str): + kind = self._normalize_kind(runtime) + cached = self._runtimes.get(kind) + if cached is None: + return self._registry.validate_task_for(kind, task) + runtime = cached + return validate_runtime_task(runtime, task) + def on(self, event: str = "*") -> Callable[[_HandlerT], _HandlerT]: """Register an event handler for tasks assembled by this hub. diff --git a/src/agent_runtime_kit/_runtime.py b/src/agent_runtime_kit/_runtime.py index 5daafc9..fde1ccd 100644 --- a/src/agent_runtime_kit/_runtime.py +++ b/src/agent_runtime_kit/_runtime.py @@ -5,7 +5,6 @@ from collections.abc import Mapping from typing import Any -from agent_runtime_kit._errors import UnsupportedTaskInputError from agent_runtime_kit._schema import resolve_structured_output from agent_runtime_kit._types import ( AgentCapabilities, @@ -13,6 +12,7 @@ AgentRuntimeKind, AgentTask, RuntimeAvailability, + TaskSupportReport, ToolCallAudit, ) from agent_runtime_kit.events import ( @@ -25,6 +25,7 @@ tool_requested_event, vendor_turn_event, ) +from agent_runtime_kit.support import _validate_declared_task_support, require_task_support class FakeAgentRuntime: @@ -47,6 +48,7 @@ def __init__( streaming=False, tool_audit=True, cancellation=True, + mcp_server_env=True, ) self._output = output self._metadata = dict(metadata or {}) @@ -57,12 +59,17 @@ def availability(self) -> RuntimeAvailability: return RuntimeAvailability.ok(self.kind, package="agent-runtime-kit") + def validate_task(self, task: AgentTask) -> TaskSupportReport: + """Report unsupported fields without side effects.""" + + return _validate_declared_task_support(self.kind, self.capabilities, task) + async def run(self, task: AgentTask) -> AgentResult: """Return a deterministic result after validating capabilities.""" await safe_emit(task, task_started_event(task, self.kind)) try: - _ensure_supported(self.kind, self.capabilities, task) + require_task_support(self.validate_task(task)) output = self._output if self._output is not None else f"Fake result for: {task.goal}" parsed = {"output": output} if task.output_schema is not None else None parsed_available = parsed is not None @@ -139,30 +146,3 @@ async def __aenter__(self) -> FakeAgentRuntime: async def __aexit__(self, _exc_type: object, _exc: object, _tb: object) -> None: await self.aclose() - - -def _ensure_supported( - kind: AgentRuntimeKind, - capabilities: AgentCapabilities, - task: AgentTask, -) -> None: - if task.mcp_servers and not capabilities.mcp_support: - raise UnsupportedTaskInputError(kind, "mcp_servers", "runtime does not support MCP") - if task.working_directory is not None and not capabilities.working_directory: - raise UnsupportedTaskInputError( - kind, - "working_directory", - "runtime does not support per-task working directories", - ) - if (task.session_id or task.resume_from) and not capabilities.session_resume: - raise UnsupportedTaskInputError( - kind, - "session_id", - "runtime does not support session resume", - ) - if task.output_schema is not None and not capabilities.structured_output: - raise UnsupportedTaskInputError( - kind, - "output_schema", - "runtime does not support structured output", - ) diff --git a/src/agent_runtime_kit/_types.py b/src/agent_runtime_kit/_types.py index 5d45eb3..ea37c8a 100644 --- a/src/agent_runtime_kit/_types.py +++ b/src/agent_runtime_kit/_types.py @@ -226,6 +226,56 @@ class AgentCapabilities: streaming: bool = False tool_audit: bool = False cancellation: bool = False + budget: bool = False + reasoning_effort: bool = False + network_control: bool = False + tool_filters: bool = False + mcp_server_env: bool = False + + +@dataclass(frozen=True) +class TaskSupportIssue: + """One task field a runtime cannot honor faithfully.""" + + field: str + message: str + + def __post_init__(self) -> None: + _require_nonblank(self.field, "TaskSupportIssue.field") + _require_nonblank(self.message, "TaskSupportIssue.message") + + +@dataclass(frozen=True) +class TaskSupportReport: + """Pure compatibility result for one task/runtime pair.""" + + kind: AgentRuntimeKind | str + issues: tuple[TaskSupportIssue, ...] = () + + def __post_init__(self) -> None: + issues = _tuple_value(self.issues, "TaskSupportReport.issues") + if not all(isinstance(issue, TaskSupportIssue) for issue in issues): + raise ValueError( + "TaskSupportReport.issues must contain only TaskSupportIssue values" + ) + object.__setattr__(self, "kind", AgentRuntimeKind.coerce(self.kind)) + object.__setattr__(self, "issues", issues) + + @property + def supported(self) -> bool: + return not self.issues + + +@runtime_checkable +class TaskSupportProvider(Protocol): + """Optional extension for runtimes with provider-specific task checks. + + ``AgentRuntime`` deliberately does not require this protocol: third-party + runtimes written before task preflight was introduced remain compatible. + """ + + def validate_task(self, task: AgentTask) -> TaskSupportReport: + """Purely report whether this runtime can honor the task.""" @dataclass(frozen=True) diff --git a/src/agent_runtime_kit/adapters/_common.py b/src/agent_runtime_kit/adapters/_common.py index 05da845..d3cf550 100644 --- a/src/agent_runtime_kit/adapters/_common.py +++ b/src/agent_runtime_kit/adapters/_common.py @@ -22,32 +22,33 @@ AgentTask, AvailabilityReason, RuntimeAvailability, + TaskSupportIssue, ) +from agent_runtime_kit.compatibility import compatibility_for -def package_availability( - kind: AgentRuntimeKind, - *, - module_name: str, - package_name: str, -) -> RuntimeAvailability: +def package_availability(kind: AgentRuntimeKind) -> RuntimeAvailability: """Return import/package availability without importing the package.""" + compatibility = compatibility_for(kind) try: - module_spec = util.find_spec(module_name) + module_spec = util.find_spec(compatibility.module) except ModuleNotFoundError: module_spec = None if module_spec is None: return RuntimeAvailability.unavailable( kind, reason=AvailabilityReason.MISSING_PACKAGE, - message=f"Install the optional dependency: agent-runtime-kit[{_extra_name(kind)}]", - package=package_name, + message=( + "Install the optional dependency: " + f"agent-runtime-kit[{compatibility.extra}]" + ), + package=compatibility.package, ) return RuntimeAvailability.ok( kind, - package=package_name, - version=package_version(package_name), + package=compatibility.package, + version=package_version(compatibility.package), ) @@ -60,20 +61,25 @@ def package_version(package_name: str) -> str | None: return None -def ensure_supported_model( +def model_support_issue( *, - kind: AgentRuntimeKind, + task: AgentTask, model: str, supported_models: tuple[str, ...] | None, -) -> None: - """Raise a typed error when a runtime was configured with an allow-list.""" +) -> TaskSupportIssue | None: + """Report a configured model allow-list mismatch at the source field.""" if supported_models is None or model in supported_models: - return - supported = ", ".join(supported_models) - raise UnsupportedTaskInputError( - kind, - "metadata.model", + return None + supported = ", ".join(supported_models) or "(none)" + if task.model is not None: + field = "model" + elif metadata_str(task.metadata, "model") is not None: + field = "metadata.model" + else: + field = "model" + return TaskSupportIssue( + field, f"model {model!r} is not supported by this runtime; supported: {supported}", ) @@ -118,48 +124,6 @@ def empty_completion_error(sdk_label: str) -> str: return f"{sdk_label} completed with no output, tool calls, or structured output" -def reject_unsupported_inputs( - kind: AgentRuntimeKind, - task: AgentTask, - *, - budget: bool, - network: bool, - tool_filters: bool, -) -> None: - """Raise ``UnsupportedTaskInputError`` for task fields a runtime cannot honor. - - Each flag selects a field whose silent omission would mislead the caller. The - project contract is to reject these inputs rather than drop them quietly, so an - adapter passes ``True`` only for fields it has no SDK surface to honor. - """ - - if budget and task.budget_usd is not None: - raise UnsupportedTaskInputError( - kind, - "budget_usd", - "this runtime does not expose a cost budget; remove budget_usd to proceed", - ) - if network and task.permissions.network is not None: - raise UnsupportedTaskInputError( - kind, - "permissions.network", - "this runtime does not expose network access control", - ) - if tool_filters: - if task.permissions.allowed_tools: - raise UnsupportedTaskInputError( - kind, - "permissions.allowed_tools", - "this runtime does not expose a tool allow-list", - ) - if task.permissions.disallowed_tools: - raise UnsupportedTaskInputError( - kind, - "permissions.disallowed_tools", - "this runtime does not expose a tool deny-list", - ) - - def filter_supported_kwargs( factory: Any, kwargs: Mapping[str, Any], @@ -359,11 +323,3 @@ async def close_vendor_resource( result = close() if hasattr(result, "__await__"): await result - - -def _extra_name(kind: AgentRuntimeKind) -> str: - return { - AgentRuntimeKind.CLAUDE_AGENT_SDK: "claude", - AgentRuntimeKind.CODEX_AGENT_SDK: "codex", - AgentRuntimeKind.ANTIGRAVITY_AGENT_SDK: "antigravity", - }.get(kind, "all") diff --git a/src/agent_runtime_kit/adapters/antigravity.py b/src/agent_runtime_kit/adapters/antigravity.py index abc8216..898b4cb 100644 --- a/src/agent_runtime_kit/adapters/antigravity.py +++ b/src/agent_runtime_kit/adapters/antigravity.py @@ -7,6 +7,7 @@ import importlib import logging import os +import re from collections.abc import Mapping from dataclasses import dataclass from pathlib import Path @@ -22,21 +23,22 @@ FilesystemAccess, PermissionMode, RuntimeAvailability, + TaskSupportIssue, + TaskSupportReport, ToolCallAudit, Usage, ) from agent_runtime_kit.adapters._common import ( close_vendor_resource, empty_completion_error, - ensure_supported_model, filter_supported_kwargs, fingerprint_item, metadata_str, + model_support_issue, optional_int, optional_str, output_schema_from, package_availability, - reject_unsupported_inputs, resolve_structured_output, ) from agent_runtime_kit.events import ( @@ -49,9 +51,12 @@ tool_requested_event, vendor_turn_event, ) +from agent_runtime_kit.support import _validate_declared_task_support, require_task_support logger = logging.getLogger(__name__) +_MCP_SERVER_NAME_PATTERN = re.compile(r"^[a-zA-Z0-9_-]+$") + class AntigravityAgentRuntime: """Run tasks through Google's ``google-antigravity`` SDK.""" @@ -65,6 +70,7 @@ class AntigravityAgentRuntime: streaming=True, tool_audit=True, cancellation=False, + tool_filters=True, ) def __init__( @@ -119,11 +125,7 @@ def availability(self) -> RuntimeAvailability: if self._agent_cls is not None: return RuntimeAvailability.ok(self.kind, package="google-antigravity") - package = package_availability( - self.kind, - module_name="google.antigravity", - package_name="google-antigravity", - ) + package = package_availability(self.kind) if not package.available: return package auth = self._auth_config() @@ -145,31 +147,44 @@ def availability(self) -> RuntimeAvailability: metadata={**dict(package.metadata), "auth_source": auth.source}, ) + def validate_task(self, task: AgentTask) -> TaskSupportReport: + """Report unsupported fields without loading auth or the vendor SDK.""" + + report = _validate_declared_task_support(self.kind, self.capabilities, task) + issues = list(report.issues) + model_issue = model_support_issue( + task=task, + model=self._model(task), + supported_models=self._supported_models, + ) + if model_issue is not None: + issues.append(model_issue) + if task.permissions.allowed_tools and task.permissions.disallowed_tools: + issues.append( + TaskSupportIssue( + "permissions.allowed_tools", + "Antigravity cannot combine an allow-list with a deny-list; " + "set only one of allowed_tools or disallowed_tools", + ) + ) + for server in task.mcp_servers: + if _MCP_SERVER_NAME_PATTERN.fullmatch(server.name) is None: + issues.append( + TaskSupportIssue( + "mcp_servers.name", + f"Antigravity MCP server name {server.name!r} must contain only " + "letters, digits, hyphens, or underscores", + ) + ) + return TaskSupportReport(kind=self.kind, issues=tuple(issues)) + async def run(self, task: AgentTask) -> AgentResult: """Execute one task with Antigravity.""" await safe_emit(task, task_started_event(task, self.kind)) try: - reject_unsupported_inputs( - self.kind, task, budget=True, network=True, tool_filters=False - ) - if task.reasoning_effort: - # LocalAgentConfig exposes no reasoning/thinking-effort control - # (google-antigravity 0.1.x), so the first-class field must not - # silently no-op. The legacy metadata alias stays ignored, as it - # always has been for this adapter. - raise UnsupportedTaskInputError( - self.kind, - "reasoning_effort", - "google-antigravity exposes no reasoning-effort control; " - "unset reasoning_effort for this runtime", - ) + require_task_support(self.validate_task(task)) model = self._model(task) - ensure_supported_model( - kind=self.kind, - model=model, - supported_models=self._supported_models, - ) sdk = self._load_sdk() # Resolve auth off the event loop: ADC discovery can call # google.auth.default(), which reads files and may hit the GCE metadata diff --git a/src/agent_runtime_kit/adapters/claude.py b/src/agent_runtime_kit/adapters/claude.py index 68959d6..1f236ae 100644 --- a/src/agent_runtime_kit/adapters/claude.py +++ b/src/agent_runtime_kit/adapters/claude.py @@ -20,6 +20,7 @@ PermissionMode, PermissionProfile, RuntimeAvailability, + TaskSupportReport, ToolCallAudit, Usage, ) @@ -27,16 +28,15 @@ STRUCTURED_OUTPUT_MISSING, close_vendor_resource, empty_completion_error, - ensure_supported_model, field_value, filter_supported_kwargs, fingerprint_value, metadata_str, + model_support_issue, optional_int, optional_str, output_schema_from, package_availability, - reject_unsupported_inputs, resolve_structured_output, ) from agent_runtime_kit.events import ( @@ -48,6 +48,7 @@ tool_completed_event, tool_requested_event, ) +from agent_runtime_kit.support import _validate_declared_task_support, require_task_support logger = logging.getLogger(__name__) @@ -64,6 +65,10 @@ class ClaudeAgentRuntime: streaming=True, tool_audit=True, cancellation=False, + budget=True, + reasoning_effort=True, + tool_filters=True, + mcp_server_env=True, ) def __init__( @@ -107,11 +112,7 @@ def availability(self) -> RuntimeAvailability: package="claude-agent-sdk", metadata=auth_metadata, ) - package = package_availability( - self.kind, - module_name="claude_agent_sdk", - package_name="claude-agent-sdk", - ) + package = package_availability(self.kind) if not package.available: return package return RuntimeAvailability.ok( @@ -121,20 +122,27 @@ def availability(self) -> RuntimeAvailability: metadata={**dict(package.metadata), **auth_metadata}, ) + def validate_task(self, task: AgentTask) -> TaskSupportReport: + """Report unsupported fields without loading the vendor SDK.""" + + report = _validate_declared_task_support(self.kind, self.capabilities, task) + issue = model_support_issue( + task=task, + model=self._model(task), + supported_models=self._supported_models, + ) + return TaskSupportReport( + kind=self.kind, + issues=report.issues + ((issue,) if issue is not None else ()), + ) + async def run(self, task: AgentTask) -> AgentResult: """Execute one task with Claude Agent SDK, streaming events as they arrive.""" await safe_emit(task, task_started_event(task, self.kind)) model = self._model(task) try: - reject_unsupported_inputs( - self.kind, task, budget=False, network=True, tool_filters=False - ) - ensure_supported_model( - kind=self.kind, - model=model, - supported_models=self._supported_models, - ) + require_task_support(self.validate_task(task)) query_func, options_cls, client_cls = self._load_sdk() options, dropped = self._build_options(task, model, options_cls) stream = _StreamState(self, task) diff --git a/src/agent_runtime_kit/adapters/codex.py b/src/agent_runtime_kit/adapters/codex.py index 66cb5cb..1772e9b 100644 --- a/src/agent_runtime_kit/adapters/codex.py +++ b/src/agent_runtime_kit/adapters/codex.py @@ -8,7 +8,7 @@ from collections.abc import Mapping from typing import Any -from agent_runtime_kit._errors import AgentRuntimeUnavailableError, UnsupportedTaskInputError +from agent_runtime_kit._errors import AgentRuntimeUnavailableError from agent_runtime_kit._types import ( AgentCapabilities, AgentResult, @@ -17,20 +17,20 @@ FilesystemAccess, PermissionMode, RuntimeAvailability, + TaskSupportReport, ToolCallAudit, Usage, ) from agent_runtime_kit.adapters._common import ( close_vendor_resource, empty_completion_error, - ensure_supported_model, field_value, filter_supported_kwargs, metadata_str, + model_support_issue, optional_int, output_schema_from, package_availability, - reject_unsupported_inputs, resolve_structured_output, ) from agent_runtime_kit.events import ( @@ -42,6 +42,7 @@ tool_completed_event, tool_requested_event, ) +from agent_runtime_kit.support import _validate_declared_task_support, require_task_support # Vendor ``ThreadItem`` discriminator values that carry a tool/command invocation. _COMMAND_ITEM = "commandExecution" @@ -69,6 +70,7 @@ class CodexAgentRuntime: streaming=False, tool_audit=True, cancellation=False, + reasoning_effort=True, ) def __init__( @@ -119,11 +121,7 @@ def availability(self) -> RuntimeAvailability: package="openai-codex", metadata=auth_metadata, ) - package = package_availability( - self.kind, - module_name="openai_codex", - package_name="openai-codex", - ) + package = package_availability(self.kind) if not package.available: return package return RuntimeAvailability.ok( @@ -133,24 +131,27 @@ def availability(self) -> RuntimeAvailability: metadata={**dict(package.metadata), **auth_metadata}, ) + def validate_task(self, task: AgentTask) -> TaskSupportReport: + """Report unsupported fields without loading the vendor SDK.""" + + report = _validate_declared_task_support(self.kind, self.capabilities, task) + issue = model_support_issue( + task=task, + model=self._model(task), + supported_models=self._supported_models, + ) + return TaskSupportReport( + kind=self.kind, + issues=report.issues + ((issue,) if issue is not None else ()), + ) + async def run(self, task: AgentTask) -> AgentResult: """Execute one task with the Codex SDK.""" await safe_emit(task, task_started_event(task, self.kind)) try: - if task.mcp_servers: - raise UnsupportedTaskInputError( - self.kind, - "mcp_servers", - "openai_codex does not expose per-task MCP server configuration", - ) - reject_unsupported_inputs(self.kind, task, budget=True, network=True, tool_filters=True) + require_task_support(self.validate_task(task)) model = self._model(task) - ensure_supported_model( - kind=self.kind, - model=model, - supported_models=self._supported_models, - ) codex_cls, config_cls, sandbox_cls, approval_mode_cls = self._load_sdk() result = await self._run_codex( task, diff --git a/src/agent_runtime_kit/compatibility.py b/src/agent_runtime_kit/compatibility.py new file mode 100644 index 0000000..30422e8 --- /dev/null +++ b/src/agent_runtime_kit/compatibility.py @@ -0,0 +1,100 @@ +"""Declared vendor SDK compatibility ranges for the built-in adapters.""" + +from __future__ import annotations + +from dataclasses import dataclass + +from agent_runtime_kit._types import AgentRuntimeKind + + +def _require_nonblank(value: str, field: str) -> None: + if not isinstance(value, str) or not value.strip(): + raise ValueError(f"{field} must be a non-empty string") + + +@dataclass(frozen=True) +class PackageVersion: + """An exact dependency version exercised by the committed lockfile.""" + + package: str + version: str + + def __post_init__(self) -> None: + _require_nonblank(self.package, "PackageVersion.package") + _require_nonblank(self.version, "PackageVersion.version") + + +@dataclass(frozen=True) +class RuntimeCompatibility: + """One built-in adapter's supported range and tested dependency set.""" + + kind: AgentRuntimeKind + extra: str + package: str + module: str + version_specifier: str + tested_version: str + tested_runtime_dependencies: tuple[PackageVersion, ...] = () + + def __post_init__(self) -> None: + for field in ( + "extra", + "package", + "module", + "version_specifier", + "tested_version", + ): + _require_nonblank(getattr(self, field), f"RuntimeCompatibility.{field}") + dependencies = tuple(self.tested_runtime_dependencies) + if not all(isinstance(item, PackageVersion) for item in dependencies): + raise ValueError( + "RuntimeCompatibility.tested_runtime_dependencies must contain " + "only PackageVersion values" + ) + names = tuple(item.package for item in dependencies) + if len(names) != len(set(names)): + raise ValueError( + "RuntimeCompatibility.tested_runtime_dependencies contains duplicates" + ) + object.__setattr__(self, "tested_runtime_dependencies", dependencies) + + +COMPATIBILITY_MANIFEST: tuple[RuntimeCompatibility, ...] = ( + RuntimeCompatibility( + kind=AgentRuntimeKind.CLAUDE_AGENT_SDK, + extra="claude", + package="claude-agent-sdk", + module="claude_agent_sdk", + version_specifier=">=0.2.87,<0.3", + tested_version="0.2.106", + ), + RuntimeCompatibility( + kind=AgentRuntimeKind.CODEX_AGENT_SDK, + extra="codex", + package="openai-codex", + module="openai_codex", + version_specifier=">=0.1.0b3,<0.2", + tested_version="0.1.0b3", + tested_runtime_dependencies=( + PackageVersion(package="openai-codex-cli-bin", version="0.137.0a4"), + ), + ), + RuntimeCompatibility( + kind=AgentRuntimeKind.ANTIGRAVITY_AGENT_SDK, + extra="antigravity", + package="google-antigravity", + module="google.antigravity", + version_specifier=">=0.1.2,<0.2", + tested_version="0.1.4", + ), +) + + +def compatibility_for(kind: AgentRuntimeKind | str) -> RuntimeCompatibility: + """Return the manifest entry for a built-in runtime kind.""" + + normalized = AgentRuntimeKind.coerce(kind) + for entry in COMPATIBILITY_MANIFEST: + if entry.kind is normalized: + return entry + raise KeyError(f"no built-in compatibility manifest entry for {normalized!r}") diff --git a/src/agent_runtime_kit/registry.py b/src/agent_runtime_kit/registry.py index b5ed6dd..66d1a99 100644 --- a/src/agent_runtime_kit/registry.py +++ b/src/agent_runtime_kit/registry.py @@ -11,9 +11,12 @@ AgentCapabilities, AgentRuntime, AgentRuntimeKind, + AgentTask, RuntimeAvailability, + TaskSupportReport, runtime_kind_value, ) +from agent_runtime_kit.support import validate_task RuntimeFactory = Callable[..., AgentRuntime] @@ -69,6 +72,13 @@ def capabilities_for(self, kind: AgentRuntimeKind | str) -> AgentCapabilities: return self.resolve(kind).capabilities + def validate_task_for( + self, kind: AgentRuntimeKind | str, task: AgentTask + ) -> TaskSupportReport: + """Resolve the runtime and run its pure task-support preflight.""" + + return validate_task(self.resolve(kind), task) + def availability_for(self, kind: AgentRuntimeKind | str) -> RuntimeAvailability: """Construct a runtime and return its availability diagnostic. diff --git a/src/agent_runtime_kit/support.py b/src/agent_runtime_kit/support.py new file mode 100644 index 0000000..1070833 --- /dev/null +++ b/src/agent_runtime_kit/support.py @@ -0,0 +1,138 @@ +"""Pure task-to-runtime compatibility checks.""" + +from __future__ import annotations + +from collections.abc import Mapping + +from agent_runtime_kit._errors import UnsupportedTaskInputError +from agent_runtime_kit._types import ( + AgentCapabilities, + AgentRuntime, + AgentRuntimeKind, + AgentTask, + TaskSupportIssue, + TaskSupportReport, +) + + +def _validate_declared_task_support( + kind: AgentRuntimeKind | str, + capabilities: AgentCapabilities, + task: AgentTask, +) -> TaskSupportReport: + """Report every task field the declared runtime capabilities cannot honor.""" + + issues: list[TaskSupportIssue] = [] + if task.mcp_servers and not capabilities.mcp_support: + issues.append( + TaskSupportIssue( + "mcp_servers", + "runtime does not support per-task MCP server configuration", + ) + ) + elif not capabilities.mcp_server_env: + if any(server.env for server in task.mcp_servers): + issues.append( + TaskSupportIssue( + "mcp_servers.env", + "runtime does not support environment values on MCP server configuration", + ) + ) + if task.working_directory is not None and not capabilities.working_directory: + issues.append( + TaskSupportIssue( + "working_directory", + "runtime does not support per-task working directories", + ) + ) + if task.session_id is not None and not capabilities.session_resume: + issues.append( + TaskSupportIssue("session_id", "runtime does not support session resume") + ) + if task.resume_from is not None and not capabilities.session_resume: + issues.append( + TaskSupportIssue("resume_from", "runtime does not support session resume") + ) + if task.output_schema is not None and not capabilities.structured_output: + issues.append( + TaskSupportIssue("output_schema", "runtime does not support structured output") + ) + elif not capabilities.structured_output: + for key in ("output_schema", "json_schema"): + if isinstance(task.metadata.get(key), Mapping): + issues.append( + TaskSupportIssue( + f"metadata.{key}", + "runtime does not support structured output", + ) + ) + break + if task.budget_usd is not None and not capabilities.budget: + issues.append( + TaskSupportIssue("budget_usd", "runtime does not expose a cost budget") + ) + if task.reasoning_effort is not None and not capabilities.reasoning_effort: + issues.append( + TaskSupportIssue( + "reasoning_effort", + "runtime does not expose reasoning-effort control", + ) + ) + elif not capabilities.reasoning_effort: + legacy_effort = task.metadata.get("reasoning_effort") + if isinstance(legacy_effort, str) and legacy_effort.strip(): + issues.append( + TaskSupportIssue( + "metadata.reasoning_effort", + "runtime does not expose reasoning-effort control", + ) + ) + if task.permissions.network is not None and not capabilities.network_control: + issues.append( + TaskSupportIssue( + "permissions.network", + "runtime does not expose network access control", + ) + ) + if not capabilities.tool_filters: + if task.permissions.allowed_tools: + issues.append( + TaskSupportIssue( + "permissions.allowed_tools", + "runtime does not expose a tool allow-list", + ) + ) + if task.permissions.disallowed_tools: + issues.append( + TaskSupportIssue( + "permissions.disallowed_tools", + "runtime does not expose a tool deny-list", + ) + ) + return TaskSupportReport(kind=kind, issues=tuple(issues)) + + +def validate_task(runtime: AgentRuntime, task: AgentTask) -> TaskSupportReport: + """Report task incompatibilities without starting a run. + + Runtimes may implement the optional :class:`TaskSupportProvider` protocol + for provider- or instance-specific rules. Older third-party runtimes remain + supported and fall back to their declared capabilities. + """ + + validator = getattr(runtime, "validate_task", None) + if callable(validator): + report = validator(task) + if not isinstance(report, TaskSupportReport): + raise TypeError("runtime.validate_task() must return TaskSupportReport") + return report + return _validate_declared_task_support(runtime.kind, runtime.capabilities, task) + + +def require_task_support(report: TaskSupportReport) -> None: + """Raise the established typed error for the report's first issue.""" + + if report.supported: + return + issue = report.issues[0] + raise UnsupportedTaskInputError(report.kind, issue.field, issue.message) diff --git a/src/agent_runtime_kit/testing/fakes.py b/src/agent_runtime_kit/testing/fakes.py index 60156cc..8444fc2 100644 --- a/src/agent_runtime_kit/testing/fakes.py +++ b/src/agent_runtime_kit/testing/fakes.py @@ -15,6 +15,7 @@ AgentTask, AvailabilityReason, RuntimeAvailability, + TaskSupportReport, ToolCallAudit, ) from agent_runtime_kit.events import ( @@ -27,6 +28,7 @@ tool_requested_event, vendor_turn_event, ) +from agent_runtime_kit.support import _validate_declared_task_support, require_task_support @dataclass(frozen=True) @@ -142,11 +144,17 @@ def availability(self) -> RuntimeAvailability: ) return RuntimeAvailability.ok(self.kind, package="fake-sdk", version="0.0.0") + def validate_task(self, task: AgentTask) -> TaskSupportReport: + """Report unsupported fields without invoking the scripted harness.""" + + return _validate_declared_task_support(self.kind, self.capabilities, task) + async def run(self, task: AgentTask) -> AgentResult: """Invoke the fake SDK and emit normalized events.""" await safe_emit(task, task_started_event(task, self.kind)) try: + require_task_support(self.validate_task(task)) result = await self.harness.invoke(task) except Exception as exc: await safe_emit(task, task_failed_event(task, self.kind, error=str(exc))) diff --git a/tests/test_support.py b/tests/test_support.py new file mode 100644 index 0000000..15236c9 --- /dev/null +++ b/tests/test_support.py @@ -0,0 +1,253 @@ +from __future__ import annotations + +from pathlib import Path + +try: + import tomllib +except ModuleNotFoundError: # pragma: no cover - Python 3.10 + import tomli as tomllib + +from agent_runtime_kit import ( + COMPATIBILITY_MANIFEST, + AgentCapabilities, + AgentKit, + AgentResult, + AgentRuntime, + AgentRuntimeKind, + AgentTask, + McpServerConfig, + PermissionProfile, + RuntimeAvailability, + SessionResumeState, + TaskSupportIssue, + TaskSupportReport, + compatibility_for, + validate_task, +) +from agent_runtime_kit.adapters import ( + AntigravityAgentRuntime, + ClaudeAgentRuntime, + CodexAgentRuntime, +) + + +class LegacyRuntime: + """A protocol-complete third-party runtime predating task support reports.""" + + kind = "x-legacy" + capabilities = AgentCapabilities() + + def availability(self) -> RuntimeAvailability: + return RuntimeAvailability.ok(self.kind) + + async def run(self, task: AgentTask) -> AgentResult: + del task + raise NotImplementedError + + async def cancel(self, task_id: str) -> None: + del task_id + + async def aclose(self) -> None: + return None + + async def __aenter__(self) -> LegacyRuntime: + return self + + async def __aexit__(self, *args: object) -> None: + del args + + +class CustomSupportRuntime(LegacyRuntime): + kind = "x-custom-support" + capabilities = AgentCapabilities(working_directory=True) + + def validate_task(self, task: AgentTask) -> TaskSupportReport: + del task + return TaskSupportReport( + self.kind, + (TaskSupportIssue("model", "custom runtime rejected the selected model"),), + ) + + +def test_validate_task_reports_every_unsupported_field_without_raising(tmp_path: Path) -> None: + task = AgentTask( + goal="g", + working_directory=tmp_path, + mcp_servers=(McpServerConfig(name="repo", command="mcp", env={"TOKEN": "x"}),), + permissions=PermissionProfile(allowed_tools=("Read",), network=False), + session_id="session-1", + output_schema={"type": "object"}, + budget_usd=1.0, + reasoning_effort="high", + ) + + report = validate_task(LegacyRuntime(), task) + + assert report.supported is False + assert [issue.field for issue in report.issues] == [ + "mcp_servers", + "working_directory", + "session_id", + "output_schema", + "budget_usd", + "reasoning_effort", + "permissions.network", + "permissions.allowed_tools", + ] + + +def test_validate_task_reports_resume_and_legacy_alias_source_fields() -> None: + task = AgentTask( + goal="g", + resume_from=SessionResumeState("session-1"), + metadata={ + "output_schema": {"type": "object"}, + "reasoning_effort": "high", + }, + ) + + report = validate_task(LegacyRuntime(), task) + + assert [issue.field for issue in report.issues] == [ + "resume_from", + "metadata.output_schema", + "metadata.reasoning_effort", + ] + + +def test_validate_task_reports_mcp_env_separately() -> None: + runtime = LegacyRuntime() + runtime.capabilities = AgentCapabilities(mcp_support=True) + task = AgentTask( + goal="g", + mcp_servers=(McpServerConfig(name="repo", command="mcp", env={"TOKEN": "x"}),), + ) + + report = validate_task(runtime, task) + + assert [issue.field for issue in report.issues] == ["mcp_servers.env"] + + +def test_builtin_runtime_reports_are_pure_and_provider_specific(tmp_path: Path) -> None: + claude = ClaudeAgentRuntime() + codex = CodexAgentRuntime() + antigravity = AntigravityAgentRuntime() + + assert claude.validate_task(AgentTask(goal="g", budget_usd=1.0)).supported is True + assert [ + issue.field + for issue in claude.validate_task( + AgentTask(goal="g", permissions=PermissionProfile(network=False)) + ).issues + ] == ["permissions.network"] + assert [ + issue.field + for issue in codex.validate_task( + AgentTask( + goal="g", + mcp_servers=(McpServerConfig(name="repo", command="mcp"),), + budget_usd=1.0, + reasoning_effort="high", + ) + ).issues + ] == ["mcp_servers", "budget_usd"] + assert [ + issue.field + for issue in antigravity.validate_task( + AgentTask( + goal="g", + working_directory=tmp_path, + mcp_servers=(McpServerConfig(name="repo", command="mcp", env={"X": "1"}),), + reasoning_effort="high", + ) + ).issues + ] == ["mcp_servers.env", "reasoning_effort"] + + +def test_adapter_reports_configured_model_allowlist_at_source_field() -> None: + runtimes = ( + ClaudeAgentRuntime(supported_models=("allowed",)), + CodexAgentRuntime(supported_models=("allowed",)), + AntigravityAgentRuntime(supported_models=("allowed",)), + ) + + for runtime in runtimes: + first_class = runtime.validate_task(AgentTask(goal="g", model="blocked")) + legacy = runtime.validate_task(AgentTask(goal="g", metadata={"model": "blocked"})) + + assert first_class.issues[-1].field == "model" + assert legacy.issues[-1].field == "metadata.model" + + +def test_antigravity_reports_static_provider_constraints() -> None: + runtime = AntigravityAgentRuntime() + task = AgentTask( + goal="g", + mcp_servers=(McpServerConfig(name="not valid!", command="mcp"),), + permissions=PermissionProfile( + allowed_tools=("view_file",), + disallowed_tools=("run_command",), + ), + metadata={"reasoning_effort": "high"}, + ) + + report = runtime.validate_task(task) + + assert [issue.field for issue in report.issues] == [ + "metadata.reasoning_effort", + "permissions.allowed_tools", + "mcp_servers.name", + ] + + +def test_agent_kit_and_registry_preserve_custom_runtime_validation() -> None: + kit = AgentKit(include_fake=False, register_default_adapters=False) + kit.registry.register(CustomSupportRuntime.kind, CustomSupportRuntime) + task = AgentTask(goal="g") + + by_kind = kit.validate_task(CustomSupportRuntime.kind, task) + by_instance = kit.validate_task(CustomSupportRuntime(), task) + by_registry = kit.registry.validate_task_for(CustomSupportRuntime.kind, task) + + assert by_kind == by_instance == by_registry + assert by_kind.issues[0].message.startswith("custom runtime") + + +def test_agent_runtime_protocol_remains_compatible_with_legacy_runtime() -> None: + assert isinstance(LegacyRuntime(), AgentRuntime) + + +def test_task_support_report_rejects_non_issue_values() -> None: + try: + TaskSupportReport("x-bad", ("not-an-issue",)) # type: ignore[arg-type] + except ValueError as exc: + assert "TaskSupportIssue" in str(exc) + else: # pragma: no cover - assertion helper without pytest dependency + raise AssertionError("invalid report value was accepted") + + +def test_compatibility_manifest_matches_project_and_lockfile() -> None: + root = Path(__file__).parents[1] + with (root / "pyproject.toml").open("rb") as stream: + project = tomllib.load(stream) + with (root / "uv.lock").open("rb") as stream: + lock = tomllib.load(stream) + + optional = project["project"]["optional-dependencies"] + locked_versions = {item["name"]: item["version"] for item in lock["package"]} + assert len({entry.kind for entry in COMPATIBILITY_MANIFEST}) == len( + COMPATIBILITY_MANIFEST + ) + assert len({entry.package for entry in COMPATIBILITY_MANIFEST}) == len( + COMPATIBILITY_MANIFEST + ) + + for entry in COMPATIBILITY_MANIFEST: + assert optional[entry.extra] == [f"{entry.package}{entry.version_specifier}"] + assert locked_versions[entry.package] == entry.tested_version + assert compatibility_for(entry.kind) is entry + for dependency in entry.tested_runtime_dependencies: + assert locked_versions[dependency.package] == dependency.version + + codex = compatibility_for(AgentRuntimeKind.CODEX_AGENT_SDK) + assert codex.tested_runtime_dependencies[0].package == "openai-codex-cli-bin" diff --git a/uv.lock b/uv.lock index 0fb792e..1c14ec3 100644 --- a/uv.lock +++ b/uv.lock @@ -60,6 +60,7 @@ dev = [ { name = "pytest-asyncio" }, { name = "pytest-cov" }, { name = "ruff" }, + { name = "tomli", marker = "python_full_version < '3.11'" }, { name = "types-jsonschema" }, ] @@ -84,6 +85,7 @@ dev = [ { name = "pytest-asyncio", specifier = ">=0.23" }, { name = "pytest-cov", specifier = ">=5.0" }, { name = "ruff", specifier = ">=0.8" }, + { name = "tomli", marker = "python_full_version < '3.11'", specifier = ">=2" }, { name = "types-jsonschema", specifier = ">=4.18" }, ]