Skip to content

docs: di-bench improvement loop#70

Merged
aural-psynapse merged 15 commits into
mainfrom
docs/di-bench-loop
Jul 17, 2026
Merged

docs: di-bench improvement loop#70
aural-psynapse merged 15 commits into
mainfrom
docs/di-bench-loop

Conversation

@aural-psynapse

@aural-psynapse aural-psynapse commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Changes

Docs

  • AGENTS.md — note it is the canonical agent entry point (mirrored as the provably.ai machine view), and make explicit that claims must come from the agent's own SourceryKitAgentResponse output, not be hand-assembled.
  • docs/handoff.md — add a SourceryKitAgentResponse section describing the typed output and its ClaimedValue fields (path, value, sourcerykit_ref), ordered type → component. Correct the claimed_value payload row from Any to list[ClaimedValue].
  • cookbooks/openai_agents — accept OPENROUTER_API_KEY as well as MODEL_API_KEY, so a verbatim copy authenticates against an OpenAI-compatible gateway (OpenRouter).

Code

  • evaluate_handoff / build_handoff_payload — when a claim can't be resolved it was dropped with only a log line, so a payload that ended up with zero claims surfaced as ERROR with an empty errors list. The drop reason now travels on the payload (build_errors) and evaluate_handoff returns it in errors, so a zero-claim result explains itself.

Why

These changes come out of a discoverability loop that runs a fresh agent from provably.ai to a working SourceryKit integration and turns each observed failure into one fix. Runs kept failing at the same points: agents couldn't find the entry doc, hand-assembled claims instead of binding SourceryKitAgentResponse, couldn't authenticate the cookbook with the key they had, and hit an ERROR with no reason when claims failed to resolve. Each item above addresses one of those. (The separate preprocess bug that blocked verification end-to-end is fixed in #71.)

Comment thread README.md Outdated
Comment thread docs/handoff.md Outdated
Comment thread docs/handoff.md Outdated
…NROUTER_API_KEY, steer off bare HTTP (di-bench binding gap)
…was returning ERROR with an empty errors list when zero claims resolved
…yload to evaluate_handoff instead of only logging them
…nd-built claim PASSes yet proves nothing (di-bench run 20260709-151308-f17d)
Comment thread cookbooks/openai_agents/agent_run.py Outdated
Comment thread src/sourcerykit/handoff/payload_builder.py Outdated
Comment thread src/sourcerykit/handoff/payload_builder.py Outdated
…rop handling, trimmed docstring, updated stale tests
@aural-psynapse aural-psynapse self-assigned this Jul 17, 2026
SimoneBottoni
SimoneBottoni previously approved these changes Jul 17, 2026

@SimoneBottoni SimoneBottoni left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Comment thread docs/pillars/handoff.md Outdated
@SimoneBottoni
SimoneBottoni self-requested a review July 17, 2026 09:40

@SimoneBottoni SimoneBottoni left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@aural-psynapse
aural-psynapse merged commit 98b7ab3 into main Jul 17, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants