Add public beta evidence status manifest#165
Conversation
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
There was a problem hiding this comment.
Claude Code Review
This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.
Tip: disable this comment in your organization's Code Review settings.
|
@coderabbitai review Context: docs/tooling/release-artifact PR only. This adds a no-secret public-beta evidence status manifest, schema, checker/tests, local/CI wiring, release-manifest/checksum coverage, and docs. It intentionally does not change Solidity behavior and does not claim public beta or production readiness. |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
📝 WalkthroughWalkthroughAdds a public-beta evidence JSON Schema and baseline artifact, a strict Python validator with unit tests, release-manifest/checksum integration, CI/Makefile/local check wiring, and documentation and ops updates to track gating evidence for public-beta and production releases. ChangesPublic-beta evidence status system
🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
|
Tip For best results, initiate chat on the files or code changes. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
# List all changed files in PR `#165`
gh pr view 165 --repo 6529-Collections/6529Stream --json files --jq '.files[].path'Length of output: 1044 🏁 Script executed: #!/bin/bash
# Get PR diff for review
gh pr diff 165 --repo 6529-Collections/6529Stream 2>/dev/null | head -3000Length of output: 122205
A few observations worth noting before merge: Missing docstring on
def public_beta_evidence_record(path: Path, repo_root: Path) -> dict[str, Any]:
"""Load, validate, and summarise the public-beta evidence status for the release manifest."""
_ISO_DATE_RE = re.compile(r"^\d{4}-\d{2}-\d{2}$")
def require_iso_date(value: Any, path: str) -> str:
"""Require an ISO-8601 date string (YYYY-MM-DD) at a JSON path."""
text = require_string(value, path)
if not _ISO_DATE_RE.fullmatch(text):
raise PublicBetaEvidenceError(f"{path} must be an ISO-8601 date (YYYY-MM-DD)")
return textThen in require_iso_date(risk.get("accepted_at"), f"{path}.accepted_at")
require_iso_date(risk.get("expires_at"), f"{path}.expires_at")
Everything else looks good:
The docstring gap is the only item that matches the PR |
|
Addressed CodeRabbit review in
Focused validation passed locally for public-beta evidence, release manifest, release artifacts, release readiness, audit package, release checksums, and Python compilation. |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 3
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
docs/tooling.md (1)
159-167:⚠️ Potential issue | 🟡 Minor | ⚡ Quick winAlign the public-beta evidence references in both docs.
docs/tooling.mdstill omitsrelease-artifacts/latest/public-beta-evidence.jsonfrom the checksum-refresh contract, andrelease-artifacts/README.mdstill omitsdocs/public-beta-evidence.mdfrom the governance-doc hash list. Since the new gate treats both as tracked release artifacts, please mention them in the same places so maintainers know to refresh the new manifest and its doc together.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/tooling.md` around lines 159 - 167, Update the release-checksum description and README to include the new public-beta evidence artifact: add release-artifacts/latest/public-beta-evidence.json to the list of inputs covered by the release-checksum/release-checksums.json generation in docs/tooling.md (the "release-checksum step"/"checksum-refresh contract" text) and add docs/public-beta-evidence.md to the governance-doc hash list in release-artifacts/README.md so both docs reference the same tracked artifact set.
🧹 Nitpick comments (3)
release-artifacts/schema/public-beta-evidence.schema.json (1)
60-66: ⚡ Quick winKeep
requirements.minItemssynchronized with checker-required IDs.
Line 65hardcodes17, while enforcement inscripts/check_public_beta_evidence.pyis derived fromPUBLIC_BETA_REQUIREMENTSandPRODUCTION_REQUIREMENTS. Add a contract test that asserts schemaminItemsequals the checker’s required-ID total to prevent silent drift.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@release-artifacts/schema/public-beta-evidence.schema.json` around lines 60 - 66, The schema's requirements.minItems is hardcoded to 17 and can drift from the enforcement logic in scripts/check_public_beta_evidence.py; add a contract test that loads release-artifacts/schema/public-beta-evidence.schema.json, reads the requirements.minItems value, computes the expected total by importing or referencing the same PUBLIC_BETA_REQUIREMENTS and PRODUCTION_REQUIREMENTS used by check_public_beta_evidence.py (or by calling a function that returns that combined list), and assert equality between schema requirements.minItems and the computed required-ID total so any future change to PUBLIC_BETA_REQUIREMENTS/PRODUCTION_REQUIREMENTS will fail the test until the schema is updated.scripts/test_release_manifest.py (1)
385-513: ⚡ Quick winAdd one regression test that uses a non-default
release_artifacts_dir.Current coverage exercises the default
release-artifacts/latestlayout only. A custom artifacts directory test would catch path-coupling regressions inpublic_beta_evidenceresolution.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@scripts/test_release_manifest.py` around lines 385 - 513, The test suite lacks coverage for using a custom release_artifacts_dir which can hide path-coupling bugs in public_beta_evidence resolution; add a new regression test (similar to test_generator_writes_deterministic_manifest) that calls generator.write_output but passes a non-default release_artifacts_dir (e.g., different from "release-artifacts/latest") when seeding the tree and building paths, then assert the manifest fields for release_artifacts.public_beta_evidence and any paths/sha256 checks still resolve correctly; locate helpers seed_release_tree, generator.write_output, and the public_beta_checker usage to adjust the test setup so the generator uses the custom directory and validate the same keys/assertions as in test_generator_writes_deterministic_manifest.docs/public-beta-evidence.md (1)
29-32: ⚡ Quick winAvoid a second source of truth for the requirement count.
requirements.minItemsduplicates the canonical requirement list inscripts/check_public_beta_evidence.py, so the schema can drift from the checker when rows are added or removed. Consider deriving the count from the same source or dropping the hard-coded count altogether.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@docs/public-beta-evidence.md` around lines 29 - 32, The schema's requirements.minItems is a hard-coded duplicate of the canonical requirement list in scripts/check_public_beta_evidence.py; update the code so the schema's minItems is derived from that canonical source (or remove the hard-coded count). Modify the code that builds or validates the schema to import or read the requirement list from scripts/check_public_beta_evidence.py (or a shared module) and set requirements.minItems = len(requirement_list) at generation time, ensuring the symbols scripts/check_public_beta_evidence.py and requirements.minItems are the single source of truth.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/public-beta-evidence.md`:
- Around line 100-102: Update the public beta evidence documentation where
accepted_at and expires_at are described to state they must use the ISO date
format YYYY-MM-DD (match regex ^\d{4}-\d{2}-\d{2}$); mention that the checker
(scripts/check_public_beta_evidence.py) enforces this and tests reject non-ISO
values so authors must provide dates like 2024-06-12 rather than freeform text.
In `@scripts/check_public_beta_evidence.py`:
- Around line 190-195: The require_iso_date function only validates format via
ISO_DATE_RE but accepts invalid calendar dates; after the existing regex check
in require_iso_date, parse the text into a real date (e.g.,
datetime.date.fromisoformat or datetime.datetime.strptime) and catch ValueError,
then raise PublicBetaEvidenceError(f"{path} must be an ISO-8601 date
(YYYY-MM-DD)") on parse failure; ensure datetime is imported and keep the
existing error message and return the original text on success.
In `@scripts/generate_release_manifest.py`:
- Around line 601-604: The manifest builder currently constructs the public beta
evidence path using repo_root / DEFAULT_PUBLIC_BETA_EVIDENCE which ignores the
release_artifacts_dir flag; change the call to public_beta_evidence_record to
resolve the evidence file from release_artifacts_dir (e.g.,
release_artifacts_dir / DEFAULT_PUBLIC_BETA_EVIDENCE or otherwise join
DEFAULT_PUBLIC_BETA_EVIDENCE against release_artifacts_dir) so validation uses
the provided --release-artifacts-dir; update the invocation site where
public_beta_evidence_record(...) is called to pass the path rooted at
release_artifacts_dir instead of repo_root.
---
Outside diff comments:
In `@docs/tooling.md`:
- Around line 159-167: Update the release-checksum description and README to
include the new public-beta evidence artifact: add
release-artifacts/latest/public-beta-evidence.json to the list of inputs covered
by the release-checksum/release-checksums.json generation in docs/tooling.md
(the "release-checksum step"/"checksum-refresh contract" text) and add
docs/public-beta-evidence.md to the governance-doc hash list in
release-artifacts/README.md so both docs reference the same tracked artifact
set.
---
Nitpick comments:
In `@docs/public-beta-evidence.md`:
- Around line 29-32: The schema's requirements.minItems is a hard-coded
duplicate of the canonical requirement list in
scripts/check_public_beta_evidence.py; update the code so the schema's minItems
is derived from that canonical source (or remove the hard-coded count). Modify
the code that builds or validates the schema to import or read the requirement
list from scripts/check_public_beta_evidence.py (or a shared module) and set
requirements.minItems = len(requirement_list) at generation time, ensuring the
symbols scripts/check_public_beta_evidence.py and requirements.minItems are the
single source of truth.
In `@release-artifacts/schema/public-beta-evidence.schema.json`:
- Around line 60-66: The schema's requirements.minItems is hardcoded to 17 and
can drift from the enforcement logic in scripts/check_public_beta_evidence.py;
add a contract test that loads
release-artifacts/schema/public-beta-evidence.schema.json, reads the
requirements.minItems value, computes the expected total by importing or
referencing the same PUBLIC_BETA_REQUIREMENTS and PRODUCTION_REQUIREMENTS used
by check_public_beta_evidence.py (or by calling a function that returns that
combined list), and assert equality between schema requirements.minItems and the
computed required-ID total so any future change to
PUBLIC_BETA_REQUIREMENTS/PRODUCTION_REQUIREMENTS will fail the test until the
schema is updated.
In `@scripts/test_release_manifest.py`:
- Around line 385-513: The test suite lacks coverage for using a custom
release_artifacts_dir which can hide path-coupling bugs in public_beta_evidence
resolution; add a new regression test (similar to
test_generator_writes_deterministic_manifest) that calls generator.write_output
but passes a non-default release_artifacts_dir (e.g., different from
"release-artifacts/latest") when seeding the tree and building paths, then
assert the manifest fields for release_artifacts.public_beta_evidence and any
paths/sha256 checks still resolve correctly; locate helpers seed_release_tree,
generator.write_output, and the public_beta_checker usage to adjust the test
setup so the generator uses the custom directory and validate the same
keys/assertions as in test_generator_writes_deterministic_manifest.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: e8ac2633-a9d8-4a81-97eb-ed1d48c32685
📒 Files selected for processing (30)
.github/workflows/ci.ymlCHANGELOG.mdMakefileREADME.mddocs/audit-package.mddocs/known-blockers.mddocs/public-beta-evidence.mddocs/release-policy.mddocs/release-readiness.mddocs/status.mddocs/tooling.mdops/AUTONOMOUS_RUN.mdops/ROADMAP.mdrelease-artifacts/README.mdrelease-artifacts/latest/SHA256SUMSrelease-artifacts/latest/public-beta-evidence.jsonrelease-artifacts/latest/release-checksums.jsonrelease-artifacts/latest/release-manifest.jsonrelease-artifacts/schema/public-beta-evidence.schema.jsonscripts/check.ps1scripts/check.shscripts/check_audit_package.pyscripts/check_public_beta_evidence.pyscripts/check_release_readiness.pyscripts/generate_release_artifacts.pyscripts/generate_release_manifest.pyscripts/test_public_beta_evidence.pyscripts/test_release_artifacts.pyscripts/test_release_manifest.pyscripts/test_release_readiness.py
|
Addressed the second CodeRabbit follow-up in Changes made:
Focused validation passed locally:
|
1 participant
Summary
Closes #164.
This adds a no-secret public-beta evidence status manifest and validation gate so public-beta and production-release blockers are tracked as machine-checkable release evidence instead of prose-only docs.
Changes
release-artifacts/latest/public-beta-evidence.jsonwith public beta and production release intentionally blocked.release-artifacts/schema/public-beta-evidence.schema.jsonplusdocs/public-beta-evidence.md.scripts/check_public_beta_evidence.pyandscripts/test_public_beta_evidence.pyto validate required requirement rows, path boundaries, retained SHA256 hashes, no-secret policy, risk acceptance metadata, and blocked-vs-ready claims.public-beta-evidence.jsonas a downstream release file, like checksum and manifest outputs.Scope
Documentation/tooling/artifact-only. No Solidity source or protocol behavior changes.
Local validation
python scripts\test_public_beta_evidence.pypython scripts\check_public_beta_evidence.pypython -m py_compile scripts\check_public_beta_evidence.py scripts\test_public_beta_evidence.pypython scripts\test_release_readiness.pypython scripts\test_audit_package.pypython scripts\test_release_manifest.pypython scripts\test_release_artifacts.pypython scripts\generate_release_artifacts.py --checkpython scripts\generate_release_manifest.py --checkpython scripts\generate_release_checksums.py --checkbash -n scripts/check.shscripts\check.ps1git diff --checkpasses with only existing Windows line-ending warning chattermake checkpowershell -ExecutionPolicy Bypass -File scripts\check.ps1Release status
This PR does not claim public beta or production readiness. The committed evidence file keeps both
public_betaandproduction_releaseblocked until external audit, fork/testnet/live deployment evidence, verified addresses, explorer verification, production signatures, signed tag evidence, production address books, retained broadcast outputs, live ceremony/randomizer evidence, and post-audit remediation evidence are retained or explicitly risk-accepted.Summary by CodeRabbit
New Features
Documentation
Tests