Skip to content

[3/5] Make SDK evolution evidence fail closed#53

Draft
ebarti wants to merge 1 commit into
agent/sdk-evolution-locked-baselinesfrom
agent/sdk-evolution-evidence-integrity
Draft

[3/5] Make SDK evolution evidence fail closed#53
ebarti wants to merge 1 commit into
agent/sdk-evolution-locked-baselinesfrom
agent/sdk-evolution-evidence-integrity

Conversation

@ebarti

@ebarti ebarti commented Jul 13, 2026

Copy link
Copy Markdown
Owner

Summary

  • parse resolver updates once as exact package/from/to transitions and reuse them across collection and deterministic gates
  • derive canonical locked-baseline expectations from collected package evidence, including no-update runs
  • contain isolated behavior setup, install, execution, timeout, and malformed-output failures as bounded structured evidence
  • compute fail > incomplete > changed > pass from raw probes, diffs, full probe pairing, and trusted expectations
  • recompute the canonical assessment for reports and implementation gates instead of trusting a cached headline
  • persist behavior_summary.json, include it in the current-state manifest, and surface snapshot/behavior errors and reasons in the Markdown report
  • require exact API-diff transitions and fail closed on missing, malformed, contradictory, incomplete, or breaking behavior evidence

Stack

Review notes

The layer received a two-pass correctness review. Review findings covered drifted no-update baselines, contradictory cached summaries, malformed nested details, failure precedence, partial probe sets, empty self-declared expectation scope, and real breaking-payload coverage; all were repaired and re-reviewed with no remaining actionable findings.

Validation

  • PYTHONPATH=. .venv/bin/python -m pytest -q — 449 passed, 12 skipped
  • focused SDK evolution suite — 90 passed
  • .venv/bin/ruff check .
  • .venv/bin/mypy
  • uv lock --check
  • git diff --check

Security-diff scans were intentionally not part of this phase review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant