Skip to content

Add release evidence live audit orchestrator#266

Merged
punk6529 merged 2 commits into
mainfrom
codex/release-evidence-live-audit-orchestrator
Jun 13, 2026
Merged

Add release evidence live audit orchestrator#266
punk6529 merged 2 commits into
mainfrom
codex/release-evidence-live-audit-orchestrator

Conversation

@punk6529

@punk6529 punk6529 commented Jun 13, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add scripts/audit_release_evidence_issue_snapshots.py, an operator-only no-secret orchestrator that exports and checks live release evidence issue snapshots for labels, bodies, and closure state.
  • Add mocked orchestrator tests and wire them into local/CI gates without adding live GitHub network access to CI.
  • Update release evidence docs, release-readiness command lists, changelog, generated release artifacts, and durable roadmap/run-state metadata after PR Reconcile snapshot exporter merge state #264.

Validation

  • python scripts/test_release_evidence_issue_snapshot_audit.py
  • python scripts/audit_release_evidence_issue_snapshots.py --help
  • python scripts/check_release_readiness.py
  • python scripts/test_release_readiness.py
  • python scripts/generate_release_manifest.py --check
  • python scripts/test_release_manifest.py
  • python scripts/generate_release_checksums.py --check
  • python scripts/test_release_checksums.py
  • python scripts/check_changelog.py
  • python -m py_compile scripts/audit_release_evidence_issue_snapshots.py scripts/test_release_evidence_issue_snapshot_audit.py
  • bash -n scripts/check.sh
  • [scriptblock]::Create((Get-Content scripts/check.ps1 -Raw)) | Out-Null
  • rg -n "^#|^##|^###" ops/ROADMAP.md ops/AUTONOMOUS_RUN.md docs/tooling.md docs/public-beta-evidence.md docs/release-readiness.md release-artifacts/README.md CHANGELOG.md
  • git diff --check passed with only Git's existing Windows line-ending warning for scripts/check.ps1
  • powershell -ExecutionPolicy Bypass -File scripts/bootstrap-windows.ps1
  • powershell -ExecutionPolicy Bypass -File scripts/check.ps1

Closes #265

Summary by CodeRabbit

  • New Features

    • Added a release-evidence snapshot audit orchestrator to run live issue snapshot audits (labels, bodies, closure) in one operator command.
  • Tests

    • Added a focused audit test suite and integrated the audit check into CI and local check sequences.
  • Documentation

    • Updated release readiness, tooling, evidence, ops, and release-artifact docs with audit guidance and workflows.
  • Chores

    • Integrated audit steps into build/check targets, CI workflow, and updated release manifests/checksums.

@claude claude Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Claude Code Review

This repository is configured for manual code reviews. Comment @claude review to trigger a review and subscribe this PR to future pushes, or @claude review once for a one-time review.

Tip: disable this comment in your organization's Code Review settings.

@coderabbitai

coderabbitai Bot commented Jun 13, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

Adds a Python orchestrator to export and validate live release-evidence issue snapshots (profiles: labels, bodies, closure), unit tests that mock subprocesses, CI/build wiring to run the new tests, operator documentation updates, and updated release-artifact checksums and operational run state.

Changes

Live Audit Orchestrator Feature

Layer / File(s) Summary
Orchestrator Core Implementation
scripts/audit_release_evidence_issue_snapshots.py
Configuration (profiles, repo, defaults), custom exception, CLI parser, profile expansion, subprocess orchestration (exporter/checker), and audit_profile workflow producing snapshot JSON and running checkers; returns 0 on success, 1 on audit failure.
Orchestrator Unit Tests
scripts/test_release_evidence_issue_snapshot_audit.py
Unit tests import the orchestrator and mock subprocess.run to verify default profile order, deterministic snapshot paths, all expansion/deduplication, stop-on-export/checker failure, and proper argument forwarding to exporter only.
Shared Argparse helper and exporter update
scripts/argparse_helpers.py, scripts/export_release_evidence_issue_snapshot.py, scripts/test_release_evidence_issue_snapshot.py
Adds positive_int helper, replaces exporter-local validator with the shared helper, and adds a test ensuring argparse error text is preserved for invalid --limit.
Build and CI Integration
.github/workflows/ci.yml, Makefile, scripts/check.sh, scripts/check.ps1, scripts/check_release_readiness.py
Adds the new audit test script to CI compile/test steps and logs, the Makefile release-evidence target, local shell/PowerShell check scripts, and the release-readiness required commands list.
Operator and Release Documentation
CHANGELOG.md, docs/public-beta-evidence.md, docs/release-readiness.md, docs/tooling.md, release-artifacts/README.md
Documents the new one-command orchestrator, explains supported profiles and operator-only live audit usage, and adds the audit test to tooling/release checklists.
Release Artifact Metadata and Checksums
release-artifacts/latest/SHA256SUMS, release-artifacts/latest/release-checksums.json, release-artifacts/latest/release-manifest.json
Regenerates checksums and updates manifest entries for the documentation files changed in this PR.
Operational State and Queue Tracking
ops/AUTONOMOUS_RUN.md, ops/ROADMAP.md
Advances durable queue state and records the prior PR merge details and updated verification metadata.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related issues

Possibly related PRs

Poem

🐰 I hopped to the CLI with a cheerful shout,
One command to gather what labels are about,
Snapshots in hand, checkers at play,
Operators run and keep CI away—
Queue 127 hops onward today.

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and specifically describes the main change: adding a release evidence live audit orchestrator script.
Linked Issues check ✅ Passed The PR successfully implements all coding requirements from issue #265: orchestrator script, mocked tests, operator-only design without CI network access, integration into docs/checks, and durable state recording.
Out of Scope Changes check ✅ Passed All changes are directly scoped to issue #265 requirements: the orchestrator, tests, documentation updates, release artifacts, and operational metadata.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/release-evidence-live-audit-orchestrator

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jun 13, 2026

Copy link
Copy Markdown
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@punk6529 punk6529 force-pushed the codex/release-evidence-live-audit-orchestrator branch from fe618ac to 13a7b80 Compare June 13, 2026 20:04

Copy link
Copy Markdown
Contributor Author

@coderabbitai review latest head

@coderabbitai

coderabbitai Bot commented Jun 13, 2026

Copy link
Copy Markdown

@punk6529 Triggering a full review of the latest head for PR #266.

✅ Action performed

Full review finished.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
scripts/audit_release_evidence_issue_snapshots.py (1)

37-45: ⚡ Quick win

Consider extracting positive_int to a shared utility module.

This function is duplicated from export_release_evidence_issue_snapshot.py (lines 140-150). Extracting it to a shared module would improve maintainability.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/audit_release_evidence_issue_snapshots.py` around lines 37 - 45, The
positive_int function is duplicated; extract it into a shared utility module
(e.g., create a new module like utils/argparse_helpers.py) containing the same
positive_int(value: str) -> int implementation and docstring, preserve the exact
argparse.ArgumentTypeError messages and behavior, then replace the local
definitions in both scripts with a single import (from utils.argparse_helpers
import positive_int) and remove the duplicate function bodies from each script;
update any imports and run linters/tests to ensure no breakage.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@scripts/audit_release_evidence_issue_snapshots.py`:
- Around line 37-45: The positive_int function is duplicated; extract it into a
shared utility module (e.g., create a new module like utils/argparse_helpers.py)
containing the same positive_int(value: str) -> int implementation and
docstring, preserve the exact argparse.ArgumentTypeError messages and behavior,
then replace the local definitions in both scripts with a single import (from
utils.argparse_helpers import positive_int) and remove the duplicate function
bodies from each script; update any imports and run linters/tests to ensure no
breakage.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: edfd66cd-f841-453d-a1af-d1250f88b51d

📥 Commits

Reviewing files that changed from the base of the PR and between 3f32666 and 13a7b80.

📒 Files selected for processing (17)
  • .github/workflows/ci.yml
  • CHANGELOG.md
  • Makefile
  • docs/public-beta-evidence.md
  • docs/release-readiness.md
  • docs/tooling.md
  • ops/AUTONOMOUS_RUN.md
  • ops/ROADMAP.md
  • release-artifacts/README.md
  • release-artifacts/latest/SHA256SUMS
  • release-artifacts/latest/release-checksums.json
  • release-artifacts/latest/release-manifest.json
  • scripts/audit_release_evidence_issue_snapshots.py
  • scripts/check.ps1
  • scripts/check.sh
  • scripts/check_release_readiness.py
  • scripts/test_release_evidence_issue_snapshot_audit.py

Copy link
Copy Markdown
Contributor Author

Addressed the CodeRabbit helper-deduplication nit in 75fa1c6 by extracting the shared positive_int argparse parser into scripts/argparse_helpers.py and importing it from both live snapshot scripts while preserving the exact must be a positive integer error text.

Validation after the follow-up:

  • python scripts/test_release_evidence_issue_snapshot.py
  • python scripts/test_release_evidence_issue_snapshot_audit.py
  • python -m py_compile scripts/argparse_helpers.py scripts/export_release_evidence_issue_snapshot.py scripts/audit_release_evidence_issue_snapshots.py scripts/test_release_evidence_issue_snapshot.py scripts/test_release_evidence_issue_snapshot_audit.py
  • python scripts/check_release_readiness.py
  • python scripts/generate_release_manifest.py --check
  • python scripts/generate_release_checksums.py --check
  • python scripts/check_changelog.py
  • bash -n scripts/check.sh
  • PowerShell syntax check for scripts/check.ps1
  • rg -n "^#|^##|^###" ops/ROADMAP.md ops/AUTONOMOUS_RUN.md docs/tooling.md docs/public-beta-evidence.md docs/release-readiness.md release-artifacts/README.md CHANGELOG.md
  • git diff --check
  • powershell -ExecutionPolicy Bypass -File scripts\check.ps1

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
ops/AUTONOMOUS_RUN.md (1)

202-226: 💤 Low value

Validation list is comprehensive and well-documented.

The follow-up section properly documents the CodeRabbit review nit (helper deduplication), the local fix (extraction to scripts/argparse_helpers.py), and an exhaustive validation run covering unit tests, Python compilation, tooling checks, shell/PowerShell syntax, heading validation, and git diff checks—all aligned with the PR objectives and the earlier validation promise.

However, one minor clarity improvement: the document shows CI run 27477620425 on head 13a7b80bd2dc3cbb695a1d1ef7d6e3e29f20c005, which differs from the opening PR head fe618aca88ba5af68196d61c6cb7af11a2ac0327. A brief note that the CI run was on the follow-up commit after addressing the CodeRabbit nit would clarify the intended timeline (i.e., "...after follow-up extraction work, GitHub Actions CI run 27477620425 passed on updated head 13a7b80bd2dc3cbb695a1d1ef7d6e3e29f20c005").

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@ops/AUTONOMOUS_RUN.md` around lines 202 - 226, Add a brief clarifying note
that the referenced GitHub Actions CI run 27477620425 ran on the follow-up
commit (head 13a7b80bd2dc3cbb695a1d1ef7d6e3e29f20c005) after addressing the
CodeRabbit nit (extraction to argparse_helpers.py), e.g., append a short
sentence to the follow-up/validation paragraph stating that CI run 27477620425
passed on updated head 13a7b80bd2dc3cbb695a1d1ef7d6e3e29f20c005 to make the
timeline explicit.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@ops/AUTONOMOUS_RUN.md`:
- Around line 202-226: Add a brief clarifying note that the referenced GitHub
Actions CI run 27477620425 ran on the follow-up commit (head
13a7b80bd2dc3cbb695a1d1ef7d6e3e29f20c005) after addressing the CodeRabbit nit
(extraction to argparse_helpers.py), e.g., append a short sentence to the
follow-up/validation paragraph stating that CI run 27477620425 passed on updated
head 13a7b80bd2dc3cbb695a1d1ef7d6e3e29f20c005 to make the timeline explicit.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 311c5ebd-60f0-4778-bc52-32222140a82c

📥 Commits

Reviewing files that changed from the base of the PR and between 13a7b80 and 75fa1c6.

📒 Files selected for processing (7)
  • .github/workflows/ci.yml
  • ops/AUTONOMOUS_RUN.md
  • scripts/argparse_helpers.py
  • scripts/audit_release_evidence_issue_snapshots.py
  • scripts/export_release_evidence_issue_snapshot.py
  • scripts/test_release_evidence_issue_snapshot.py
  • scripts/test_release_evidence_issue_snapshot_audit.py
✅ Files skipped from review due to trivial changes (1)
  • scripts/argparse_helpers.py
🚧 Files skipped from review as they are similar to previous changes (3)
  • .github/workflows/ci.yml
  • scripts/test_release_evidence_issue_snapshot_audit.py
  • scripts/audit_release_evidence_issue_snapshots.py

@punk6529 punk6529 merged commit edd5ddd into main Jun 13, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add release evidence live audit orchestrator

1 participant