Skip to content

Harden release evidence tracker body drift checks#244

Merged
punk6529 merged 2 commits into
mainfrom
codex/release-evidence-body-drift
Jun 13, 2026
Merged

Harden release evidence tracker body drift checks#244
punk6529 merged 2 commits into
mainfrom
codex/release-evidence-body-drift

Conversation

@punk6529

@punk6529 punk6529 commented Jun 13, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add scripts/check_release_evidence_issue_bodies.py with deterministic committed body-sync validation, optional --live-json snapshot auditing, and --write-body-files remediation output for gh issue edit --body-file.
  • Add focused tests for matching snapshots, newline normalization, missing issues, title mismatch, body drift, malformed snapshots, CLI error handling, and deterministic remediation body files.
  • Wire the body-drift checker into Make, CI, scripts/check.sh, scripts/check.ps1, release-readiness command coverage, docs, body-sync validation metadata, release manifest/checksum artifacts, changelog, roadmap, and durable run state.
  • Remediated live tracker issue body drift on issues Retain public beta evidence: external_audit_report #215 through Retain production release evidence: post_audit_remediation #231 from the committed body-sync artifact, then re-ran the live snapshot audit successfully.

Validation

  • python scripts/test_release_evidence_issue_bodies.py
  • python scripts/check_release_evidence_issue_bodies.py
  • python scripts/check_release_evidence_issue_bodies.py --live-json tmp\release-evidence-issue-bodies.json
  • python scripts/test_release_evidence_issue_body_sync.py
  • python scripts/generate_release_evidence_issue_body_sync.py --check
  • python scripts/test_release_readiness.py
  • python scripts/check_release_readiness.py
  • python scripts/test_release_manifest.py
  • python scripts/generate_release_manifest.py --check
  • python scripts/test_release_checksums.py
  • python scripts/generate_release_checksums.py --check
  • python scripts/test_changelog_check.py
  • python scripts/check_changelog.py
  • bash -n scripts/check.sh
  • PowerShell parser check for scripts/check.ps1
  • python -m py_compile scripts/check_release_evidence_issue_bodies.py scripts/test_release_evidence_issue_bodies.py scripts/generate_release_evidence_issue_body_sync.py scripts/check_release_readiness.py
  • rg -n "^#|^##|^###" ops/ROADMAP.md ops/AUTONOMOUS_RUN.md docs/tooling.md docs/public-beta-evidence.md docs/release-readiness.md release-artifacts/README.md
  • git diff --check

Closes #242

Summary by CodeRabbit

  • New Features

    • Added validation capabilities for release evidence synchronization with optional live GitHub auditing and automated remediation file generation.
  • Documentation

    • Expanded documentation with new validation procedures and commands.
  • Tests

    • Added test coverage for release evidence validation.
  • Chores

    • Updated build system and CI configuration.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.

Tip: disable this comment in your organization's Code Review settings.

@coderabbitai

coderabbitai Bot commented Jun 13, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

This PR introduces deterministic GitHub issue body drift detection for release evidence trackers. It adds a new CLI checker script with optional snapshot auditing, wires validation into build gates and CI, documents the audit and remediation flow, refreshes release artifacts and checksums, and advances operational tracking for the automation queue.

Changes

Release Evidence Body Drift Checks

Layer / File(s) Summary
Checker CLI and drift validation core
scripts/check_release_evidence_issue_bodies.py
New CLI loads body-sync JSON and derives expected issue rows with canonicalized bodies and SHA-256 hashes. Accepts optional snapshot JSON and validates by issue number, title, and body equality; detects state issues and reports drift with deterministic remediation command. Supports --write-body-files to generate per-issue Markdown for operator gh issue edit application.
Unit and CLI coverage for body validation
scripts/test_release_evidence_issue_bodies.py
Dedicated test module with deterministic fixtures for body-sync documents and snapshot issues. Validates success, snapshot normalization (newline handling), missing issue rejection, title mismatch, body drift detection (including remediation command text), and CLI entry-point behavior with proper status codes and stderr messaging.
Local and CI gate wiring for new checks
Makefile, scripts/check.sh, scripts/check.ps1, .github/workflows/ci.yml, scripts/check_release_readiness.py, scripts/generate_release_evidence_issue_body_sync.py
Extends check target and its prerequisites with new release-evidence-issue-bodies-check Makefile target; adds test/check invocations to shell and PowerShell check scripts; inserts both commands into CI "Repository hygiene" and "Public beta evidence" phases; updates REQUIRED_COMMANDS in release-readiness validator; and embeds validation commands in generated body-sync documents.
Operator documentation for audit and remediation
docs/public-beta-evidence.md, docs/release-readiness.md, docs/tooling.md, CHANGELOG.md, release-artifacts/README.md
Documents new step 16 in evidence-update checklist for checking committed bodies and optional live drift audit. Adds body-validation to local-evidence and release-command checklists. Expands tooling reference with commands for snapshot export, body validation, and deterministic remediation file generation. Updates changelog and release-artifacts README to describe drift detection and remediation capabilities.
Generated artifact and checksum refresh
release-artifacts/latest/release-evidence-issue-body-sync.json, release-artifacts/latest/release-evidence-issue-body-sync.md, release-artifacts/latest/release-manifest.json, release-artifacts/latest/SHA256SUMS, release-artifacts/latest/release-checksums.json
Adds new body validation commands to generated artifact command lists; updates embedded SHA256 digests and size metadata for body-sync JSON/Markdown, manifest, changelog, and governance docs to reflect content changes.
Roadmap and autonomous run state progression
ops/AUTONOMOUS_RUN.md, ops/ROADMAP.md
Advances queue item 115 to merged state (PR #243), marks queue item 116 as active, switches worklog to PR #244 with branch/commit ancestry and body-drift validation checklist, updates roadmap verification metadata and evidence baseline, and appends decision-log entries documenting the queue progression.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related issues

  • 6529-Collections/6529Stream#242: This PR directly implements the issue body drift validation acceptance criteria including optional snapshot audit mode, deterministic remediation guidance, and no-secret CI checks.
  • 6529-Collections/6529Stream#237: Related through ops/AUTONOMOUS_RUN.md and ops/ROADMAP.md state tracking for queue item 116 progression and verification metadata refresh.

Possibly related PRs

  • 6529-Collections/6529Stream#236: This PR validates the release-evidence-issue-body-sync artifact generated and wired by PR #236; shared release artifact and CI/Makefile/release-readiness dependencies.
  • 6529-Collections/6529Stream#243: Directly preceding PR that merged issue-label drift checks; this PR extends the same pattern to issue bodies and records that merge in its ops state tracking.
  • 6529-Collections/6529Stream#213: Updates scripts/check_release_readiness.py REQUIRED_COMMANDS list in the same way for release-readiness gate wiring; builds on similar CI/dashboard validation infrastructure.

Poem

🐰 A rabbit's release-evidence rhyme:

Bodies in GitHub drift with time,
But now we check 'em—deterministic crime!
With snapshots and fixes in Markdown neat,
Our tracker's evidence stays complete. 📋✨

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title 'Harden release evidence tracker body drift checks' accurately describes the main change—adding body-drift validation and checks for release evidence tracker issues.
Linked Issues check ✅ Passed The PR comprehensively implements issue #242 requirements: deterministic body-sync validation, optional snapshot audit with drift detection, remediation guidance, focused tests, and full integration into CI/docs/manifest/checksums without requiring GitHub network access in CI.
Out of Scope Changes check ✅ Passed All changes are directly scoped to issue #242: new body-drift checker scripts, tests, integration into CI/Make/docs, and manifest/checksum updates. No unrelated or out-of-scope modifications are present.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/release-evidence-body-drift

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jun 13, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
scripts/check_release_evidence_issue_bodies.py (1)

51-76: ⚡ Quick win

Consider extracting type validators into a shared module.

The require_dict, require_list, require_string, and require_positive_int helpers duplicate similar validators from the issue_link_checker module (evidenced by the generator's use of issue_link_checker.require_positive_int in the upstream body-sync generator). Extracting these into a shared validation utility module would reduce duplication and centralize validation logic updates.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/check_release_evidence_issue_bodies.py` around lines 51 - 76, The
four local validators (require_dict, require_list, require_string,
require_positive_int) duplicate logic in issue_link_checker; extract them into a
shared validation utility (e.g., a new module like validators or
validation_utils) and replace the local definitions with imports and usages from
that module: move the implementations into the new module, export functions
named require_dict, require_list, require_string, require_positive_int, update
scripts/check_release_evidence_issue_bodies.py to import these functions instead
of defining them, and update any other modules that currently duplicate the same
validators (such as issue_link_checker) to import from the shared module so all
callers (including code that references require_positive_int) use the single
implementation; remove the now-duplicated local definitions from this file.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@scripts/check_release_evidence_issue_bodies.py`:
- Around line 51-76: The four local validators (require_dict, require_list,
require_string, require_positive_int) duplicate logic in issue_link_checker;
extract them into a shared validation utility (e.g., a new module like
validators or validation_utils) and replace the local definitions with imports
and usages from that module: move the implementations into the new module,
export functions named require_dict, require_list, require_string,
require_positive_int, update scripts/check_release_evidence_issue_bodies.py to
import these functions instead of defining them, and update any other modules
that currently duplicate the same validators (such as issue_link_checker) to
import from the shared module so all callers (including code that references
require_positive_int) use the single implementation; remove the now-duplicated
local definitions from this file.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 768b05c5-13ac-4ee8-8934-9b96cd034dc8

📥 Commits

Reviewing files that changed from the base of the PR and between abe9896 and 378c488.

📒 Files selected for processing (20)
  • .github/workflows/ci.yml
  • CHANGELOG.md
  • Makefile
  • docs/public-beta-evidence.md
  • docs/release-readiness.md
  • docs/tooling.md
  • ops/AUTONOMOUS_RUN.md
  • ops/ROADMAP.md
  • release-artifacts/README.md
  • release-artifacts/latest/SHA256SUMS
  • release-artifacts/latest/release-checksums.json
  • release-artifacts/latest/release-evidence-issue-body-sync.json
  • release-artifacts/latest/release-evidence-issue-body-sync.md
  • release-artifacts/latest/release-manifest.json
  • scripts/check.ps1
  • scripts/check.sh
  • scripts/check_release_evidence_issue_bodies.py
  • scripts/check_release_readiness.py
  • scripts/generate_release_evidence_issue_body_sync.py
  • scripts/test_release_evidence_issue_bodies.py

@punk6529 punk6529 merged commit b618685 into main Jun 13, 2026
2 checks passed
punk6529 added a commit that referenced this pull request Jun 13, 2026
Record PR #244 merge evidence, refresh roadmap metadata, and queue tracker closure/readiness guards.

Closes #245.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Harden release evidence tracker body drift checks

1 participant