diff --git a/AGENTS.md b/AGENTS.md index 8adfa1a..9a31cc9 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -1,5 +1,10 @@ # AGENTS.md +> This file is the canonical agent entry point. It lives in the repository at +> https://github.com/ProvablyAI/sourcerykit/blob/main/AGENTS.md and is also mirrored as the +> machine view of https://provably.ai. If you reached it through the landing page, the repo +> copy is the source of truth — read it there when in doubt. + SourceryKit is the Python SDK for [Provably](https://provably.ai): verifiable guardrails for AI agents — it records outbound HTTP calls, enforces endpoint policies, and checks an agent's claims against what those calls actually returned, so a hallucinated value is @@ -38,6 +43,14 @@ so the same tool can be called many times, and separate agents can each own a st binding is framework-specific. Copy the closest cookbook and change just that binding; each cookbook's README covers its own wiring. +**The claim must come from the agent, not your code.** Run a real agent with +`output_type=SourceryKitAgentResponse` (or your framework's equivalent) and pass its +`final_output.claimed_values` straight into the handoff. Do NOT hand-roll raw chat-completion +calls or assemble claims yourself from the fetched data — that skips the agent the whole +system exists to verify. Using any OpenAI-compatible endpoint (e.g. OpenRouter)? Point the +SDK's client at your `MODEL_URL` with `MODEL_API_KEY`, like +[cookbooks/openai_agents](cookbooks/openai_agents) does — don't drop to bare HTTP. + **Single-agent** — one agent fetches, claims, and verifies (weather): | Cookbook | Framework | What you'll find | diff --git a/docs/pillars/handoff.md b/docs/pillars/handoff.md index 8ffbbac..ee161ca 100644 --- a/docs/pillars/handoff.md +++ b/docs/pillars/handoff.md @@ -1,15 +1,81 @@ # Handoff The handoff mechanism structures agent claims—such as the result of an API call—and submits them to an evaluation service. These claims are evaluated deterministically against authoritative records to provide verifiable runtime guardrails. -Agent frameworks (OpenAI Agents SDK, LangChain) should use `SourceryKitAgentResponse` as the structured output type. This enforces a typed contract where the LLM returns a `reasoning` string and a `claimed_values` list—a flat collection of `ClaimedValue` objects, each with a JSONPath-style `path` and an extracted string `value`. These `claimed_values` feed directly into the handoff payload. - - ## Core Flow - **Collect Claims**: The agent records statements about its actions alongside supporting request/response payloads. - **Build Payload**: The claims, trusted endpoint snapshots, and execution metadata are compiled into a structured `HandoffPayload`. - **Evaluate**: The payload is sent to the evaluation service via the SDK, which matches claims against the backend source of truth and returns a verdict. +## The agent's structured output: `SourceryKitAgentResponse` +Claims originate from the agent itself, not from your code. Agent frameworks (OpenAI Agents SDK, LangChain) bind `SourceryKitAgentResponse` as the structured output type — `output_type=` for the OpenAI Agents SDK, `response_format=` for LangChain — so the LLM is forced to return this typed contract instead of free-form text. + +`SourceryKitAgentResponse` has two fields: + +| Field | Type | Description | +|---|---|---| +| `reasoning` | `str` | The agent's explanation of its actions for this execution slice. | +| `claimed_values` | `list[ClaimedValue]` | A **flat list** of the values the agent claims it produced (see below). | + +Each `ClaimedValue` in that list has three string fields — nothing else: + +| Field | Type | Description | +|---|---|---| +| `path` | `str` | JSONPath into the tool output, e.g. `$.base_experience`. | +| `value` | `str` | The extracted value, as a string. | +| `sourcerykit_ref` | `str` | Copied verbatim from the tool call's `sourcerykit_ref` return, so the claim maps to the recorded call. Mandatory. | + +`final_output.claimed_values` feeds directly into the handoff payload — you pass the list straight through, as shown in the payload tables below and the [Example](#example) at the end. + + +## Anatomy of the payload_data +The `build_handoff_payload` function accepts a structured `payload_data` dictionary. Other runtime fields—such as network intercepts, organization IDs, and API keys—are resolved automatically by the SDK during compilation. + +> [!NOTE] +> The fields below represent a complete and exhaustive view of the parameters you can manually configure. Any schema fields omitted from these tables are managed entirely by the SDK lifecycle. + +### Payload Input Fields +| Field | Type | Description | +|---|---|---| +| `reasoning` | `str \| None` | Detailing the agent's logic or intent for the overall execution slice. | +| `claims` | `list[HandoffClaim]` | A complete list of raw claim dictionaries to be resolved into execution claims. | + + +### Claim Input Fields +| Field | Type | Description | +|---|---|---| +| `action_name` | `str` | Logical identifier for the agent action producing the claim. | +| `claimed_value` | `list[ClaimedValue]` | The agent's `claimed_values` — pass `final_output.claimed_values` straight through. See [`SourceryKitAgentResponse`](#the-agents-structured-output-sourcerykitagentresponse) for the shape. Not an arbitrary dict of your own field names. | +| `verification_mode` | `str` | The verification strategy applied to this specific claim (e.g., `field_extraction`). | +| `range_min` | `float | int | None` | Optional inclusive lower bound boundary used for `range_threshold` mode. | +| `range_max` | `float | int | None` | Optional inclusive upper bound boundary used for `range_threshold` mode. | + + +## Verification Modes +The evaluation engine processes claims using one of four specific strategies: + +- **field_extraction**: Isolates a specific element in the backend record using the `json_path` string and compares it directly to the `claimed_value`. +- **range_threshold**: Verifies that the extracted numeric value matches the `claimed_value` and falls inclusively between defined `range_min` and `range_max` boundaries. + + +## Evaluation Logic +When `evaluate_handoff` is invoked, the evaluator validates data integrity through a multi-layered trust gate: + +1. **Pre-Flight Check**: Any claim missing a valid `query_record_id` fails immediately with a CAUGHT status. + +2. **Retrieve Logs & Proof**: The engine uses the payload credentials to fetch the original HTTP query logs, response headers, and the cryptographic proof recorded during the interception phase. + +3. **Verify Cryptographic Proof**: Before running any logical data checks, the engine validates the proof of the retrieved logs. This guarantees that: + - The network response was actually captured by the runtime agent. + - The logged data has not been modified or tampered with since it was written to the database. + +4. **Run Verification Rules**: The engine applies your chosen `verification_mode` (such as an `field_extraction`) against the cryptographically verified records. + +5. **Final Verdict**: The run receives a final PASS verdict only if the cryptographic proof is valid and every individual claim satisfies its verification rules. + +For details on how database logging works, see [architecture](https://provably.ai/docs/pillars/architecture). To learn how HTTP requests are captured in real-time, see [interceptor](https://provably.ai/docs/pillars/interceptor). + + ## Example The following example demonstrates how to construct a claim, bundle it into a HandoffPayload, and submit it for evaluation: @@ -63,50 +129,3 @@ print(f"Evaluation Verdict: {result.get('outcome')}") - `CAUGHT` — at least one claim did not match the recorded data, or used an untrusted endpoint. - `ERROR` — nothing could be verified (for example, no claim matched an intercept record). Verifying zero claims always resolves to `ERROR`, never `PASS`. - -## Anatomy of the payload_data -The `build_handoff_payload` function accepts a structured `payload_data` dictionary. Other runtime fields—such as network intercepts, organization IDs, and API keys—are resolved automatically by the SDK during compilation. - -> [!NOTE] -> The fields below represent a complete and exhaustive view of the parameters you can manually configure. Any schema fields omitted from these tables are managed entirely by the SDK lifecycle. - -### Payload Input Fields -| Field | Type | Description | -|---|---|---| -| `reasoning` | `str \| None` | Detailing the agent's logic or intent for the overall execution slice. | -| `claims` | `list[HandoffClaim]` | A complete list of raw claim dictionaries to be resolved into execution claims. | - - -### Claim Input Fields -| Field | Type | Description | -|---|---|---| -| `action_name` | `str` | Logical identifier for the agent action producing the claim. | -| `claimed_value` | `Any` | The specific data value or object subset the agent claims to be true. | -| `verification_mode` | `str` | The verification strategy applied to this specific claim (e.g., `field_extraction`). | -| `range_min` | `float | int | None` | Optional inclusive lower bound boundary used for `range_threshold` mode. | -| `range_max` | `float | int | None` | Optional inclusive upper bound boundary used for `range_threshold` mode. | - - -## Verification Modes -The evaluation engine processes claims using one of four specific strategies: - -- **field_extraction**: Isolates a specific element in the backend record using the `json_path` string and compares it directly to the `claimed_value`. -- **range_threshold**: Verifies that the extracted numeric value matches the `claimed_value` and falls inclusively between defined `range_min` and `range_max` boundaries. - - -## Evaluation Logic -When `evaluate_handoff` is invoked, the evaluator validates data integrity through a multi-layered trust gate: - -1. **Pre-Flight Check**: Any claim missing a valid `query_record_id` fails immediately with a CAUGHT status. - -2. **Retrieve Logs & Proof**: The engine uses the payload credentials to fetch the original HTTP query logs, response headers, and the cryptographic proof recorded during the interception phase. - -3. **Verify Cryptographic Proof**: Before running any logical data checks, the engine validates the proof of the retrieved logs. This guarantees that: - - The network response was actually captured by the runtime agent. - - The logged data has not been modified or tampered with since it was written to the database. - -4. **Run Verification Rules**: The engine applies your chosen `verification_mode` (such as an `field_extraction`) against the cryptographically verified records. - -5. **Final Verdict**: The run receives a final PASS verdict only if the cryptographic proof is valid and every individual claim satisfies its verification rules. - -For details on how database logging works, see [architecture](https://provably.ai/docs/pillars/architecture). To learn how HTTP requests are captured in real-time, see [interceptor](https://provably.ai/docs/pillars/interceptor). diff --git a/src/sourcerykit/evaluator/evaluator.py b/src/sourcerykit/evaluator/evaluator.py index 057d79b..006b7d4 100644 --- a/src/sourcerykit/evaluator/evaluator.py +++ b/src/sourcerykit/evaluator/evaluator.py @@ -71,8 +71,20 @@ async def evaluate_handoff(*, payload: HandoffPayload) -> dict[str, Any]: } per_claim.append(verdict) + # Claims dropped while building the payload never reach the verifier above. Surface those + # reasons — otherwise a payload that arrives with zero claims is an ERROR with no + # explanation of what went wrong. + errors = list(payload.build_errors) + errors + outcome = _resolve_outcome(per_claim, errors) + if outcome == Outcome.ERROR and not per_claim and not errors: + errors = [ + f"no claims were verified: the handoff payload contained {len(payload.claims)} " + "claim(s), and none resolved to a recorded intercept. Check that each claimed " + "value carries a sourcerykit_ref matching a recorded tool call." + ] + return {"outcome": outcome, "per_claim": per_claim, "errors": errors} diff --git a/src/sourcerykit/handoff/payload_builder.py b/src/sourcerykit/handoff/payload_builder.py index 1596f82..f78c9b7 100644 --- a/src/sourcerykit/handoff/payload_builder.py +++ b/src/sourcerykit/handoff/payload_builder.py @@ -61,12 +61,12 @@ async def build_handoff_payload( instr = instructions if instructions is not None else default_instructions() # Run claim resolution and trusted-endpoint fetch concurrently - (claims, query_urls, query_ids), trusted_endpoint_registry = await asyncio.gather( + (claims, query_urls, query_ids, build_errors), trusted_endpoint_registry = await asyncio.gather( _build_claims(trace_id, blob, intercept_agent_id), list_all_trusted_endpoints(), ) - _log.info("build_handoff_payload_completed", claim_count=len(claims)) + _log.info("build_handoff_payload_completed", claim_count=len(claims), dropped=len(build_errors)) return HandoffPayload( provably_mcp_url=settings.provably_mcp, @@ -83,6 +83,7 @@ async def build_handoff_payload( query_urls=query_urls, task=task, reasoning=reasoning, + build_errors=build_errors, ) @@ -123,15 +124,15 @@ async def _build_claims( trace_id: uuid.UUID, fetch_and_claim_json: Any, intercept_agent_id: str, -) -> tuple[list[HandoffClaim], list[str], list[uuid.UUID]]: +) -> tuple[list[HandoffClaim], list[str], list[uuid.UUID], list[str]]: """Aggregates, resolves intercept IDs, and emits tracking metadata.""" if not isinstance(fetch_and_claim_json, dict): - return [], [], [] + return [], [], [], ["fetch_and_claim is not a dict; no claims could be read"] raw_claims = fetch_and_claim_json.get("claims") if not isinstance(raw_claims, list): - return [], [], [] + return [], [], [], ["fetch_and_claim['claims'] is missing or not a list"] # Filter to valid claim dicts up-front valid_raws = [raw for raw in raw_claims if isinstance(raw, dict) and str(raw.get("action_name") or "").strip()] @@ -150,19 +151,22 @@ async def _build_claims( claims = [] urls = [] ids = [] - for raw, r in zip(expanded, results): - if isinstance(r, BaseException): + drop_errors: list[str] = [] + for raw, outcome in zip(expanded, results): + if isinstance(outcome, BaseException): _log.warning( "claim_resolution_failed", action_name=raw.get("action_name"), - error=str(r), + error=str(outcome), ) + drop_errors.append(f"claim '{raw.get('action_name')}' dropped: {outcome}") continue - claims.append(r[0]) - urls.append(r[1]) - ids.append(r[2]) + claim, url, qid = outcome + claims.append(claim) + urls.append(url) + ids.append(qid) - return claims, urls, ids + return claims, urls, ids, drop_errors async def _resolve_claim( diff --git a/src/sourcerykit/schemas/handoff.py b/src/sourcerykit/schemas/handoff.py index fbfc6a7..0a5990c 100644 --- a/src/sourcerykit/schemas/handoff.py +++ b/src/sourcerykit/schemas/handoff.py @@ -107,6 +107,11 @@ class HandoffPayload(BaseModel): ) task: str = Field(default="", description="Short task title.") reasoning: str = Field(default="", description="Agent's natural-language reasoning trace.") + build_errors: list[str] = Field( + default_factory=list, + description="Reasons any claims were dropped while building this payload (e.g. a claim " + "whose sourcerykit_ref matched no recorded intercept). Surfaced by evaluate_handoff.", + ) sdk_precheck: dict[str, Any] | None = Field( default=None, description="Optional SDK-side health/precheck output captured before handoff.", diff --git a/tests/unit/test_evaluator_core.py b/tests/unit/test_evaluator_core.py index a98f9bc..85180b7 100644 --- a/tests/unit/test_evaluator_core.py +++ b/tests/unit/test_evaluator_core.py @@ -228,4 +228,6 @@ async def test_empty_claims_payload_returns_error(self, monkeypatch: pytest.Monk result = await evaluate_handoff(payload=payload) assert result["outcome"] == Outcome.ERROR assert result["per_claim"] == [] - assert result["errors"] == [] + # A zero-claim ERROR now explains itself instead of returning an empty errors list. + assert result["errors"] + assert "no claims were verified" in result["errors"][0] diff --git a/tests/unit/test_payload_builder.py b/tests/unit/test_payload_builder.py index acf353c..c7fcc28 100644 --- a/tests/unit/test_payload_builder.py +++ b/tests/unit/test_payload_builder.py @@ -52,7 +52,7 @@ async def test_returns_empty_claims_for_none_input(self, _env: None) -> None: patch("sourcerykit.handoff.payload_builder.get_settings", return_value=_mock_settings()), patch("sourcerykit.handoff.payload_builder.get_engine", return_value=_mock_engine()), patch("sourcerykit.handoff.payload_builder.list_all_trusted_endpoints", AsyncMock(return_value=[])), - patch("sourcerykit.handoff.payload_builder._build_claims", AsyncMock(return_value=([], [], []))), + patch("sourcerykit.handoff.payload_builder._build_claims", AsyncMock(return_value=([], [], [], []))), ): hp = await build_handoff_payload(None, prompt="test") assert hp.claims == [] @@ -64,7 +64,7 @@ async def test_field_guide_populated(self, _env: None) -> None: patch("sourcerykit.handoff.payload_builder.get_settings", return_value=_mock_settings()), patch("sourcerykit.handoff.payload_builder.get_engine", return_value=_mock_engine()), patch("sourcerykit.handoff.payload_builder.list_all_trusted_endpoints", AsyncMock(return_value=[])), - patch("sourcerykit.handoff.payload_builder._build_claims", AsyncMock(return_value=([], [], []))), + patch("sourcerykit.handoff.payload_builder._build_claims", AsyncMock(return_value=([], [], [], []))), ): hp = await build_handoff_payload(None, prompt="test") assert isinstance(hp.field_guide, dict) @@ -76,7 +76,7 @@ async def test_custom_task_is_propagated(self, _env: None) -> None: patch("sourcerykit.handoff.payload_builder.get_settings", return_value=_mock_settings()), patch("sourcerykit.handoff.payload_builder.get_engine", return_value=_mock_engine()), patch("sourcerykit.handoff.payload_builder.list_all_trusted_endpoints", AsyncMock(return_value=[])), - patch("sourcerykit.handoff.payload_builder._build_claims", AsyncMock(return_value=([], [], []))), + patch("sourcerykit.handoff.payload_builder._build_claims", AsyncMock(return_value=([], [], [], []))), ): hp = await build_handoff_payload(None, prompt="test", task="My custom task") assert hp.task == "My custom task" @@ -97,7 +97,7 @@ async def test_run_id_forwarded(self, _env: None) -> None: patch("sourcerykit.handoff.payload_builder.get_settings", return_value=_mock_settings()), patch("sourcerykit.handoff.payload_builder.get_engine", return_value=_mock_engine()), patch("sourcerykit.handoff.payload_builder.list_all_trusted_endpoints", AsyncMock(return_value=[])), - patch("sourcerykit.handoff.payload_builder._build_claims", AsyncMock(return_value=([], [], []))), + patch("sourcerykit.handoff.payload_builder._build_claims", AsyncMock(return_value=([], [], [], []))), ): hp = await build_handoff_payload(None, prompt="test", run_id=rid) assert hp.run_id == rid @@ -108,7 +108,7 @@ async def test_integration_api_key_from_bootstrap(self, _env: None) -> None: patch("sourcerykit.handoff.payload_builder.get_settings", return_value=_mock_settings()), patch("sourcerykit.handoff.payload_builder.get_engine", return_value=_mock_engine()), patch("sourcerykit.handoff.payload_builder.list_all_trusted_endpoints", AsyncMock(return_value=[])), - patch("sourcerykit.handoff.payload_builder._build_claims", AsyncMock(return_value=([], [], []))), + patch("sourcerykit.handoff.payload_builder._build_claims", AsyncMock(return_value=([], [], [], []))), ): hp = await build_handoff_payload(None, prompt="test") assert hp.integration_api_key == "my-int-key" @@ -119,7 +119,7 @@ async def test_reasoning_extracted_from_blob(self, _env: None) -> None: patch("sourcerykit.handoff.payload_builder.get_settings", return_value=_mock_settings()), patch("sourcerykit.handoff.payload_builder.get_engine", return_value=_mock_engine()), patch("sourcerykit.handoff.payload_builder.list_all_trusted_endpoints", AsyncMock(return_value=[])), - patch("sourcerykit.handoff.payload_builder._build_claims", AsyncMock(return_value=([], [], []))), + patch("sourcerykit.handoff.payload_builder._build_claims", AsyncMock(return_value=([], [], [], []))), ): hp = await build_handoff_payload({"reasoning": "because reasons"}, prompt="test") assert hp.reasoning == "because reasons"