docs: di-bench improvement loop#70
Merged
Merged
Conversation
…nch run 20260708-165318-6406)
…nding in handoff.md (di-bench run 20260708-192958-e188)
aural-psynapse
force-pushed
the
docs/di-bench-loop
branch
from
July 8, 2026 20:03
5d7847c to
070031d
Compare
SimoneBottoni
requested changes
Jul 9, 2026
…NROUTER_API_KEY, steer off bare HTTP (di-bench binding gap)
…was returning ERROR with an empty errors list when zero claims resolved
…claimed_values shape
…yload to evaluate_handoff instead of only logging them
…nd-built claim PASSes yet proves nothing (di-bench run 20260709-151308-f17d)
…he instructional block from handoff.md
SimoneBottoni
requested changes
Jul 9, 2026
…rop handling, trimmed docstring, updated stale tests
SimoneBottoni
previously approved these changes
Jul 17, 2026
SimoneBottoni
self-requested a review
July 17, 2026 09:40
…reads MODEL_API_KEY only)
…e the worked example (Simone review)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Changes
Docs
SourceryKitAgentResponseoutput, not be hand-assembled.SourceryKitAgentResponsesection describing the typed output and itsClaimedValuefields (path,value,sourcerykit_ref), ordered type → component. Correct theclaimed_valuepayload row fromAnytolist[ClaimedValue].OPENROUTER_API_KEYas well asMODEL_API_KEY, so a verbatim copy authenticates against an OpenAI-compatible gateway (OpenRouter).Code
ERRORwith an emptyerrorslist. The drop reason now travels on the payload (build_errors) andevaluate_handoffreturns it inerrors, so a zero-claim result explains itself.Why
These changes come out of a discoverability loop that runs a fresh agent from provably.ai to a working SourceryKit integration and turns each observed failure into one fix. Runs kept failing at the same points: agents couldn't find the entry doc, hand-assembled claims instead of binding
SourceryKitAgentResponse, couldn't authenticate the cookbook with the key they had, and hit anERRORwith no reason when claims failed to resolve. Each item above addresses one of those. (The separate preprocess bug that blocked verification end-to-end is fixed in #71.)