From d73db7bb8b9c4de93d4b12a0a0d9899a0629d739 Mon Sep 17 00:00:00 2001 From: Guillaume Tauzin <4648633+gtauzin@users.noreply.github.com> Date: Sun, 26 Jul 2026 14:27:56 +0200 Subject: [PATCH 1/5] feat(ci): make the template repo subject to the gates it ships This repository generates seven packages and receives nothing back: every improvement flows out of template/, and nothing flows in. That is how "pin exact uv version in all setup-uv steps" pinned every step in the template and none of the six here, and how the ci-passed roll-up shipped in v0.32.0 while no ruleset required it on this repo for four releases. The gaps are absences, not drift, so a render-and-diff check cannot find them. This adds a parity manifest instead: every gate the template ships resolves to matched, excluded with a written reason, or leading, and the shipped-gate set is derived from a rendered project rather than listed by hand so a new gate arrives unanswered instead of not arriving. Closed here: uv pinned across all six setup-uv steps and asserted over this repo's own workflows; a nightly canary; the docs entry points and content gate moved onto the publishing engine; the skills mirror enforced byte-identically over tracked files; git-cliff migrated to commit.remote.*. Generated projects can now commit a CLAUDE.md: the template seeds one and registers it in _skip_if_exists, and the six-variant ignore block is gone. This repo's own CLAUDE.md is tracked and reconciled with its Copilot copy, correcting three claims that were not true of main. Every assertion was verified to fail before being accepted. Two found real defects: the manifest initially checked that gates were *answered* but never that a "matched" claim was *true*, so deleting this repo's nightly workflow failed nothing; and deriving gates from jobs and hooks alone was blind to controls that are only files, which is how CODEOWNERS and SECURITY.md reached seven projects and not this one. --- .git-cliff.toml | 4 +- .github/copilot-instructions.md | 7 + .github/workflows/changelog.yml | 3 + .github/workflows/commit-message.yml | 3 + .github/workflows/nightly.yml | 101 ++++++ .github/workflows/tests.yml | 9 + .gitignore | 7 +- CLAUDE.md | 53 +++ copier.yml | 7 + docs/pages/how-to/update-template.md | 22 ++ docs/pages/reference/project-structure.md | 2 + noxfile.py | 17 +- .../references/file-classification.md | 8 + .../references/file-classification.md | 8 + template/.gitignore.jinja | 11 +- template/CLAUDE.md.jinja | 40 +++ tests/parity_manifest.yml | 247 +++++++++++++ tests/test_parity_manifest.py | 324 ++++++++++++++++++ tests/test_repo_docs.py | 70 +++- tests/test_repo_workflows.py | 218 ++++++++++++ tests/test_template.py | 37 ++ 21 files changed, 1176 insertions(+), 22 deletions(-) create mode 100644 .github/workflows/nightly.yml create mode 100644 CLAUDE.md create mode 100644 template/CLAUDE.md.jinja create mode 100644 tests/parity_manifest.yml create mode 100644 tests/test_parity_manifest.py create mode 100644 tests/test_repo_workflows.py diff --git a/.git-cliff.toml b/.git-cliff.toml index b37cf44..9aa5035 100644 --- a/.git-cliff.toml +++ b/.git-cliff.toml @@ -33,10 +33,10 @@ This **{{ release_type }} release** includes {{ commits | length }} commit{%- if ### {{ group | striptags | trim | upper_first }} {%- for commit in commits %} -- {{ commit.message | upper_first }}{%- if commit.github.pr_number %} ([#{{ commit.github.pr_number }}](https://github.com/stateful-y/python-package-copier/pull/{{ commit.github.pr_number }})){%- endif %}{%- if commit.github.username %} by @{{ commit.github.username }}{%- endif %} +- {{ commit.message | upper_first }}{%- if commit.remote.pr_number %} ([#{{ commit.remote.pr_number }}](https://github.com/stateful-y/python-package-copier/pull/{{ commit.remote.pr_number }})){%- endif %}{%- if commit.remote.username %} by @{{ commit.remote.username }}{%- endif %} {%- endfor %} {%- endfor %} -{%- set new_contributors = commits | filter(attribute="github.username") | map(attribute="github.username") | unique %} +{%- set new_contributors = commits | filter(attribute="remote.username") | map(attribute="remote.username") | unique %} {%- if new_contributors | length > 0 %} ### Contributors diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index fc97dcc..4263c30 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -44,3 +44,10 @@ just fix # Auto-format and lint just serve # Preview docs uv sync --group test # Install test deps ``` + +## Skills +Skills in `.claude/skills/` mirror `.github/skills/` (kept for GitHub Copilot). Edit both when changing one. This is now enforced: `tests/test_repo_workflows.py` asserts the two trees track the same files and are byte-identical. The check is scoped to files git tracks, because both trees deliberately ignore `openspec-*` skills as tool state the openspec CLI writes into whichever copy it likes. + +Skills that ship to *generated* projects live in `template/.github/skills/` and are a separate set — they are not mirrored here. Within `template/`, the `.github/skills/` and `.claude/skills/` copies must stay byte-identical. Note that `test_claude_skills_generated` checks only that both trees carry the same skill *names* and that each has a `SKILL.md`; two copies whose contents diverged would still pass it. + +`polish-changelog` is **not on `main`**. Its content lives on the unmerged branch `feat/changelog-polish-skill` and in `~/.claude/skills/polish-changelog/` (global, out-of-repo, updated by hand), so it is available in every repo but tracked in none. The `changelog-polish` capability spec under `openspec/specs/` is canonical and describes behaviour the repo does not currently ship. Either land that branch or treat the global copy as the only source. diff --git a/.github/workflows/changelog.yml b/.github/workflows/changelog.yml index 0227e17..d3a2f75 100644 --- a/.github/workflows/changelog.yml +++ b/.github/workflows/changelog.yml @@ -56,6 +56,9 @@ jobs: - name: Install uv uses: astral-sh/setup-uv@v7 + with: + # renovate: datasource=github-releases depName=astral-sh/uv + version: "0.10.0" - name: Run hooks on generated CHANGELOG run: | diff --git a/.github/workflows/commit-message.yml b/.github/workflows/commit-message.yml index d0f4cf8..5c048c4 100644 --- a/.github/workflows/commit-message.yml +++ b/.github/workflows/commit-message.yml @@ -43,6 +43,9 @@ jobs: - name: Install uv if: ${{ github.event.pull_request.commits == 1 }} uses: astral-sh/setup-uv@v7 + with: + # renovate: datasource=github-releases depName=astral-sh/uv + version: "0.10.0" - name: Set up Python if: ${{ github.event.pull_request.commits == 1 }} diff --git a/.github/workflows/nightly.yml b/.github/workflows/nightly.yml new file mode 100644 index 0000000..a42e83a --- /dev/null +++ b/.github/workflows/nightly.yml @@ -0,0 +1,101 @@ +name: Nightly Build + +# The canary this repository ships to every generated project and did not run itself. +# +# This repo's suite generates projects and runs `nox` inside them, so it depends on the +# whole external toolchain (copier, uv, ruff, prek, zensical) resolving at head. That +# makes it the fleet's most exposed repo to upstream drift and, until now, the only one +# with no scheduled run: a template regression surfaced as red nightlies in seven +# downstream repos before this repo noticed, so the signal arrived everywhere except +# where the cause was. +# +# Unlike the generated version the matrix is fixed here rather than derived from copier +# answers, because this repository is not itself a generated project. + +on: + schedule: + # Run every day at 2:30 AM UTC + - cron: "30 2 * * *" + workflow_dispatch: # Allow manual triggering + +permissions: + contents: read + +jobs: + test: + name: Nightly suite (Python ${{ matrix.python-version }}) + runs-on: ubuntu-latest + strategy: + fail-fast: false + matrix: + python-version: ["3.11", "3.12", "3.13", "3.14"] + + steps: + - uses: actions/checkout@v7 + + - name: Install uv + uses: astral-sh/setup-uv@v7 + with: + # renovate: datasource=github-releases depName=astral-sh/uv + version: "0.10.0" + enable-cache: true + cache-dependency-glob: "pyproject.toml" + + - name: Set up Python ${{ matrix.python-version }} + run: uv python install ${{ matrix.python-version }} + + # The full suite, slow and integration tests included: those are the ones that + # generate a project and run nox inside it, which is the part upstream drift + # actually breaks. A fast-only nightly would be green through exactly the + # failures this job exists to catch. + - name: Run the full test suite + # renovate: datasource=pypi depName=nox + run: uvx nox@2026.7.11 -s test-${{ matrix.python-version }} + + create-issue-on-failure: + needs: test + if: ${{ failure() }} + runs-on: ubuntu-latest + permissions: + issues: write + steps: + - uses: actions/checkout@v7 + + - name: Create issue on failure + uses: actions/github-script@v9 + with: + script: | + const title = '🔴 Nightly build failed'; + const body = `The nightly build failed on ${{ github.sha }}. + + **Workflow run:** ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }} + **Branch:** ${{ github.ref_name }} + **Commit:** ${{ github.sha }} + + Please investigate and fix the issue.`; + + // Check if there's already an open issue for nightly build failures + const issues = await github.rest.issues.listForRepo({ + owner: context.repo.owner, + repo: context.repo.repo, + state: 'open', + labels: 'nightly-build-failure' + }); + + if (issues.data.length === 0) { + await github.rest.issues.create({ + owner: context.repo.owner, + repo: context.repo.repo, + title: title, + body: body, + labels: ['nightly-build-failure', 'bug'] + }); + } else { + // Add a comment to the existing issue + await github.rest.issues.createComment({ + owner: context.repo.owner, + repo: context.repo.repo, + issue_number: issues.data[0].number, + body: `Another nightly build failure occurred:\n\n${body}` + }); + } diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index 5ab2cef..d35cfeb 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -35,6 +35,8 @@ jobs: - name: Install uv uses: astral-sh/setup-uv@v7 with: + # renovate: datasource=github-releases depName=astral-sh/uv + version: "0.10.0" enable-cache: true cache-dependency-glob: "pyproject.toml" @@ -68,6 +70,8 @@ jobs: - name: Install uv uses: astral-sh/setup-uv@v7 with: + # renovate: datasource=github-releases depName=astral-sh/uv + version: "0.10.0" enable-cache: true cache-dependency-glob: "pyproject.toml" @@ -87,6 +91,9 @@ jobs: - name: Install uv uses: astral-sh/setup-uv@v7 + with: + # renovate: datasource=github-releases depName=astral-sh/uv + version: "0.10.0" - name: Set up Python run: uv python install 3.12 @@ -132,6 +139,8 @@ jobs: - name: Install uv uses: astral-sh/setup-uv@v7 with: + # renovate: datasource=github-releases depName=astral-sh/uv + version: "0.10.0" enable-cache: true cache-dependency-glob: "pyproject.toml" diff --git a/.gitignore b/.gitignore index a2726bf..e202ffa 100644 --- a/.gitignore +++ b/.gitignore @@ -188,8 +188,11 @@ Desktop.ini /.github/prompts/ /.github/skills/openspec-*/ -# AI Assistant Instructions -/CLAUDE.md +# AI assistant instructions are NOT ignored. CLAUDE.md is project content: it is the +# file that tells every assistant how this repo works, and ignoring it meant it never +# entered a pull request, was never reviewed, did not survive a clone, and drifted from +# its tracked Copilot counterpart by exactly the section instructing that both be kept +# in step. An instruction file outside version control cannot be tested. /CLAUDE_INSTRUCTIONS.md /.claude.md /claude-instructions.md diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..a4a8b67 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,53 @@ +# Python Package Copier + +## Project Overview +This is a **Copier template** for generating modern Python packages. +- **Root**: configuration for *this* repo (tests, docs for the template). +- **`template/`**: Jinja2 source files for *generated* projects. Changes here affect output. +- **`copier.yml`**: Defines user prompts/variables (e.g., `{{ package_name }}`). + +## Critical Workflows +- **Package Management**: Uses **uv** exclusively. No `pip` or `venv`. +- **Task Running**: `uvx nox` (global install) or `just`. +- **Testing**: + - **Fast (Unit)**: `just test-fast` (Checks file structure/content generation). + - **Slow (Integration)**: `just test-slow` (Runs `nox` *inside* generated projects). + - **Fixture**: Use `copie` fixture (wraps `CopierTestFixture`) in `tests/conftest.py`. +- **Lint/Fix**: `just fix` (Runs `uv run prek`; prek and the lint tools are all pinned by `uv.lock`). + +## Architecture & Patterns +- **Template Files**: End in `.jinja`. Variables: `{{ min_python_version }}`. +- **Conditional Dirs**: directory names like `{% if include_examples %}examples{% endif %}/`. +- **Tech Stack (Target)**: `uv`, `hatchling` (build), `ruff` (linter), `ty` (types), `nox` (tasks). +- **Nox Configuration**: + - **Root**: `noxfile.py` tests the template. + - **Template**: `template/noxfile.py.jinja` tests the *generated* package. + - **Important**: `nox` is NOT a project dependency; it's run via `uvx`. + +## Development Rules +1. **Adding Features**: + - Update `copier.yml` (prompts). + - Update `template/` files. + - Update `tests/conftest.py` default answers. + - Add test case in `tests/test_template.py`. +2. **Dependencies**: + - Root `pyproject.toml`: Only for testing the template (copier, pytest). + - Template `pyproject.toml.jinja`: Definition for generated packages. +3. **CI/CD**: + - Template uses `tests.yml` (fast/full split). + - Generated projects get `tests.yml`, `changelog.yml` (git-cliff), `publish-release.yml`. + +## Common Commands +```bash +just test-fast # Run unit tests (fast feedback) +just fix # Auto-format and lint +just serve # Preview docs +uv sync --group test # Install test deps +``` + +## Skills +Skills in `.claude/skills/` mirror `.github/skills/` (kept for GitHub Copilot). Edit both when changing one. This is now enforced: `tests/test_repo_workflows.py` asserts the two trees track the same files and are byte-identical. The check is scoped to files git tracks, because both trees deliberately ignore `openspec-*` skills as tool state the openspec CLI writes into whichever copy it likes. + +Skills that ship to *generated* projects live in `template/.github/skills/` and are a separate set — they are not mirrored here. Within `template/`, the `.github/skills/` and `.claude/skills/` copies must stay byte-identical. Note that `test_claude_skills_generated` checks only that both trees carry the same skill *names* and that each has a `SKILL.md`; two copies whose contents diverged would still pass it. + +`polish-changelog` is **not on `main`**. Its content lives on the unmerged branch `feat/changelog-polish-skill` and in `~/.claude/skills/polish-changelog/` (global, out-of-repo, updated by hand), so it is available in every repo but tracked in none. The `changelog-polish` capability spec under `openspec/specs/` is canonical and describes behaviour the repo does not currently ship. Either land that branch or treat the global copy as the only source. diff --git a/copier.yml b/copier.yml index ca2e5bd..4c6da7f 100644 --- a/copier.yml +++ b/copier.yml @@ -52,6 +52,13 @@ _skip_if_exists: # file "must never be overwritten" -- that classification is advisory and # copier ignores it; this line is what enforces it. - "docs/pages/examples/index.md" + # Project instructions for AI assistants. Seeded once so a new project has + # something to start from, then never delivered again. Every project rewrites this + # wholesale -- it exists to record what is true about *that* project -- so the + # template's version survives nowhere, and an update applied as a diff against the + # template's copy would reject and revert it, exactly as v0.22.0 did to five + # projects' getting-started.md. There is no merge to lose here either. + - "CLAUDE.md" _envops: block_start_string: "{%" diff --git a/docs/pages/how-to/update-template.md b/docs/pages/how-to/update-template.md index eba6538..35582f4 100644 --- a/docs/pages/how-to/update-template.md +++ b/docs/pages/how-to/update-template.md @@ -93,6 +93,28 @@ Copier respects your local changes. Files you've customized are merged, not over - Custom content you added to documentation is preserved - Dependencies you added to `pyproject.toml` are preserved +### CLAUDE.md and other seed-once files + +A few files are delivered **once**, when your project is generated, and then never +again. `CLAUDE.md` is one of them, alongside your logo and favicon, the documentation +landing page, and the getting-started tutorial. + +These are files every project rewrites completely. Once yours no longer resembles the +template's version, there is nothing left to merge: an update applied as a diff against +the template's copy would reject the whole change and revert your page to the starter +content, leaving your work in a `.rej` file nobody reads. That is not a theoretical +risk. One template release changed a single blank line and replaced five projects' +hand-written getting-started pages with the 74-line stub. + +So the template stops delivering them. Rewrite `CLAUDE.md` however you like: record why +a dependency is pinned, which template assumptions your project breaks, what a failing +check actually means. Updates will leave it alone, and you will not see a conflict for +it because there is no conflict to have. + +The trade is that improvements the template later makes to these files do **not** reach +you automatically. If you want them, copy them across by hand. That is deliberate: a +template that can append to a page it does not own can also delete from it. + ## AI-Assisted Updates If you use GitHub Copilot or a similar AI coding assistant, you can ask it to handle the update for you. The template ships with an **`update-from-template` skill** that automates the full workflow: diff --git a/docs/pages/reference/project-structure.md b/docs/pages/reference/project-structure.md index 4671045..9411160 100644 --- a/docs/pages/reference/project-structure.md +++ b/docs/pages/reference/project-structure.md @@ -41,6 +41,7 @@ my-package/ ├── .pre-commit-config.yaml ├── .readthedocs.yml ├── CHANGELOG.md +├── CLAUDE.md ├── CONTRIBUTING.md ├── justfile ├── LICENSE @@ -66,6 +67,7 @@ my-package/ | `.editorconfig` | Editor-neutral formatting settings | | `LICENSE` | Project license (based on `license` template variable) | | `CHANGELOG.md` | Auto-generated changelog from conventional commits | +| `CLAUDE.md` | Project instructions for AI coding assistants. Seeded once, then owned by your project: template updates never overwrite it (see [Update from Template](../how-to/update-template.md#claudemd-and-other-seed-once-files)) | | `CONTRIBUTING.md` | Quick-start contributing guide (redirects to full docs) | | `README.md` | Project overview, badges, installation instructions | diff --git a/noxfile.py b/noxfile.py index 6ff7539..896dc25 100644 --- a/noxfile.py +++ b/noxfile.py @@ -160,7 +160,13 @@ def link_docs(session: nox.Session) -> None: @nox.session(venv_backend="uv") def build_docs(session: nox.Session) -> None: - """Build the documentation.""" + """Build the documentation with the engine that publishes it. + + Zensical, not MkDocs. `.readthedocs.yml` runs `zensical build` and the `justfile` + does too, so a nox session on MkDocs built an artifact nobody publishes: it could + stay green through a Zensical-only failure, and the empty-site bug this repo has + already hit is exactly that shape (zero HTML at exit 0, `--strict` included). + """ # Install dependencies session.run_install( "uv", @@ -170,8 +176,8 @@ def build_docs(session: nox.Session) -> None: env={"UV_PROJECT_ENVIRONMENT": session.virtualenv.location}, ) - # Build the docs - session.run("mkdocs", "build", "--clean", external=True) + # Build the docs. Zensical has no --clean flag; it rebuilds site/ in place. + session.run("zensical", "build", external=True) @nox.session(venv_backend="uv") @@ -186,7 +192,6 @@ def serve_docs(session: nox.Session) -> None: env={"UV_PROJECT_ENVIRONMENT": session.virtualenv.location}, ) - # Build and serve the docs - session.run("mkdocs", "build", "--clean", external=True) + # Serve with the publishing engine, matching build_docs and the justfile. session.log("###### Starting local server. Press Control+C to stop server ######") - session.run("mkdocs", "serve", "-a", "localhost:8080", external=True) + session.run("zensical", "serve", "-a", "localhost:8080", external=True) diff --git a/template/.claude/skills/update-from-template/references/file-classification.md b/template/.claude/skills/update-from-template/references/file-classification.md index f5004b2..fc6e140 100644 --- a/template/.claude/skills/update-from-template/references/file-classification.md +++ b/template/.claude/skills/update-from-template/references/file-classification.md @@ -109,6 +109,14 @@ CODEOWNERS Project-specific files where the template only provides initial scaffolding. Never overwrite with template content after initial generation. ```text +CLAUDE.md # Project instructions for AI assistants. Seeded + # once, then owned entirely by the project: it + # records what is true about *this* project, so + # the template's version survives nowhere. + # Enforced by _skip_if_exists in copier.yml, not + # by this classification -- copier ignores tiers, + # and a page it keeps re-delivering gets reverted + # by one shifted line (v0.22.0, five projects). src//** # All source code tests/** # All test files examples/** # conditional: include_examples diff --git a/template/.github/skills/update-from-template/references/file-classification.md b/template/.github/skills/update-from-template/references/file-classification.md index f5004b2..fc6e140 100644 --- a/template/.github/skills/update-from-template/references/file-classification.md +++ b/template/.github/skills/update-from-template/references/file-classification.md @@ -109,6 +109,14 @@ CODEOWNERS Project-specific files where the template only provides initial scaffolding. Never overwrite with template content after initial generation. ```text +CLAUDE.md # Project instructions for AI assistants. Seeded + # once, then owned entirely by the project: it + # records what is true about *this* project, so + # the template's version survives nowhere. + # Enforced by _skip_if_exists in copier.yml, not + # by this classification -- copier ignores tiers, + # and a page it keeps re-delivering gets reverted + # by one shifted line (v0.22.0, five projects). src//** # All source code tests/** # All test files examples/** # conditional: include_examples diff --git a/template/.gitignore.jinja b/template/.gitignore.jinja index f6fef2c..9b552ba 100644 --- a/template/.gitignore.jinja +++ b/template/.gitignore.jinja @@ -95,13 +95,10 @@ docs/pages/api/ # Worktrees .worktrees/ -# AI Assistant Instructions -CLAUDE.md -CLAUDE_INSTRUCTIONS.md -.claude.md -claude-instructions.md -claude-config.md -claude-prompt.md +# AI assistant instructions are NOT ignored. CLAUDE.md is project content: it is +# where a project records what a newcomer cannot derive from the code, and ignoring +# it meant no generated project could keep that anywhere the repo could see. The +# template seeds it once (see _skip_if_exists in copier.yml) and never overwrites it. # GitHub Copilot COPILOT.md diff --git a/template/CLAUDE.md.jinja b/template/CLAUDE.md.jinja new file mode 100644 index 0000000..8de695d --- /dev/null +++ b/template/CLAUDE.md.jinja @@ -0,0 +1,40 @@ +# {{ project_name }} + +{% if description %}{{ description }} + +{% endif %}Instructions for AI coding assistants working in this repository. + +**This file is yours.** It is seeded once when the project is generated and never +delivered again: a template update will not overwrite your edits, and there is no +merge to lose. Rewrite it freely. What follows is a starting point, not a contract. + +## Project overview + +A Python package generated from [python-package-copier](https://github.com/{{ github_username }}/python-package-copier). +Source lives in `src/{{ package_name }}/`, tests in `tests/`. + +## Critical workflows + +- **Package management**: `uv` exclusively. No `pip`, no `venv`. +- **Task running**: `just` (see `just --list`), or `uvx nox` for the CI sessions. +- **Setup**: `just install` creates the lockfile and installs the git hooks. + The `-f` matters: without it an existing pre-commit shim is chained rather than + replaced, and both run on every commit. +- **Testing**: `just test-fast` for quick feedback, `just test` for everything. +- **Lint and format**: `just fix`. Tools are pinned by `uv.lock`, so `--locked` + failures mean the lockfile is stale, not that the tool changed. +- **Docs**: `just serve` to preview, `just build` to build. + +## Conventions + +- Conventional commits are enforced at commit time and on pull request titles. +- `uv.lock` is tracked and CI runs with `--locked`; commit it with dependency changes. +- Docstrings follow numpydoc. In `References` sections use a markdown ordered list + with plain `[1]` citations, never reStructuredText (`.. [1]` or `[1]_`), which + renders literally on the generated API pages. + +## What to record here + +The things a newcomer cannot derive from the code: why a dependency is pinned, which +assumptions this project breaks relative to the template, what a failing check +actually means. Keep it short enough that it stays true. diff --git a/tests/parity_manifest.yml b/tests/parity_manifest.yml new file mode 100644 index 0000000..7e18286 --- /dev/null +++ b/tests/parity_manifest.yml @@ -0,0 +1,247 @@ +# Parity manifest: what this repository runs, against what it ships. +# +# This repo generates seven packages and receives nothing back. Improvements flow out +# of template/; nothing flows in. That is how `fix(ci): pin exact uv version in all +# setup-uv steps` pinned every step in the template and none of this repo's own, and +# how the `ci-passed` roll-up shipped in v0.32.0 while no ruleset required it here for +# four releases. +# +# Every gate the template ships must appear below with a status: +# +# matched this repo runs an equivalent +# excluded it cannot or should not apply here, and `reason` says why +# leading this repo runs something the template has not adopted yet +# +# A gate with no entry fails the suite. The `reason` text is the artifact: it turns +# "this repo skips that" from something silently true into something reviewed in a +# pull request. The shipped-gate list is DERIVED from a rendered project, never +# hand-copied, so a new template gate arrives here as an unanswered entry rather than +# not arriving at all. + +# --------------------------------------------------------------------------- +# Gates held in files: workflow jobs and pre-commit hooks. +# --------------------------------------------------------------------------- +file_gates: + # --- CI: test execution ------------------------------------------------- + tests.yml:test-fast: + equivalent: tests.yml:test-fast + status: matched + reason: this repo runs test-fast across the same OS/Python matrix. + tests.yml:test-full: + equivalent: tests.yml:test-full + status: matched + reason: this repo runs test-full on non-draft pull requests. + tests.yml:lint: + equivalent: tests.yml:lint + status: matched + reason: this repo runs the same nox-driven hook pass. + tests.yml:ci-passed: + equivalent: tests.yml:ci-passed + status: matched + reason: this repo runs the roll-up, and protect-main requires it (see settings_gates). + tests.yml:secret-scan: + equivalent: tests.yml:secret-scan + status: matched + reason: gitleaks job present and listed in the roll-up's needs. + tests.yml:test-compat: + status: excluded + reason: >- + Pins a dependency version and re-runs the fast suite. This repo's dependencies + are test-only (copier, pytest), so there is no library surface whose version + compatibility a downstream would care about. The template ships it disabled + (if: false) until a project defines pins. + tests.yml:test_docstrings: + status: excluded + reason: >- + Runs doctests over src/. This repo has no src/ and ships no importable + package; its Python is tests and noxfile only. + tests.yml:check_docs: + equivalent: tests.yml:test-full + status: matched + reason: >- + Equivalent is tests/test_repo_docs.py, which builds with the publishing engine + and asserts the site has content. Runs in the test suite rather than as a + separate CI job; the roll-up covers it either way. + tests.yml:compare_docs_mkdocs: + status: leading + reason: >- + A non-blocking Material-vs-Zensical comparison build for the transition. This + repo finished that transition: every entry point is on Zensical and + test_every_docs_entry_point_uses_the_publishing_engine forbids reintroducing the + other engine. Nothing to compare against. + tests.yml:examples: + status: excluded + reason: >- + Runs marimo example notebooks. This repo ships no examples/ directory; it is a + copier template, not a library with a gallery. + + # --- CI: release and metadata ------------------------------------------- + changelog.yml:changelog: + equivalent: changelog.yml:changelog + status: matched + reason: same git-cliff changelog PR flow, on the same tag trigger. + publish-release.yml:create-release: + equivalent: publish-release.yml:create-release + status: matched + reason: this repo creates a GitHub release from the changelog PR merge. + publish-release.yml:build: + status: excluded + reason: >- + Builds an sdist and wheel. This repo is a copier template consumed by + `copier copy`, not a distributable package; there is nothing to build. + publish-release.yml:pypi-publish: + status: excluded + reason: >- + Publishes to PyPI via OIDC trusted publishing. Nothing to publish, per the + build exclusion above. + commit-message.yml:validate-commit-message: + equivalent: commit-message.yml:validate-commit-message + status: matched + reason: same single-commit message validation. + pr-title.yml:validate-pr-title: + equivalent: pr-title.yml:validate-pr-title + status: matched + reason: same conventional pull request title check. + + # --- CI: scheduled and security ----------------------------------------- + nightly.yml:test: + equivalent: nightly.yml:test + status: matched + reason: >- + Added by this change. This repo's suite generates projects and runs nox inside + them, so it is the fleet's most exposed repo to upstream toolchain drift and was + the only one with no scheduled run. + nightly.yml:create-issue-on-failure: + equivalent: nightly.yml:create-issue-on-failure + status: matched + reason: same issue-filing job, with the same dedupe-by-label behaviour. + codeql.yml:analyze: + status: excluded + reason: >- + NOT YET RESOLVED AS A DELIBERATE SKIP. This repo is public, so CodeQL is free + and the template's own public-repo gate would ship it. It is absent here only + because harden-fleet-security shipped these files to the template and never to + this repo. Task 3.9 decides whether to adopt or genuinely exclude; until then + this entry records a known shortfall rather than a judgment. + scorecard.yml:analysis: + status: excluded + reason: >- + NOT YET RESOLVED AS A DELIBERATE SKIP. Same shortfall as codeql above: public + repo, free service, shipped to the template only. Recorded, not justified. + + # --- shipped root files ------------------------------------------------ + # Derived from the rendered project root. Jobs and hooks are not the only shape a + # control takes, and a manifest reading only those was blind to CODEOWNERS and + # SECURITY.md -- which is how both reached seven generated projects and not this one. + file:.editorconfig: {status: matched, reason: byte-identical to the shipped copy.} + file:.gitignore: {status: matched, reason: this repo maintains its own, with the same shape.} + file:.pre-commit-config.yaml: {status: matched, reason: same hook set, scoped for a repo with no src/.} + file:.git-cliff.toml: {status: matched, reason: same changelog config, now on the same field names.} + file:.readthedocs.yml: {status: matched, reason: same Read the Docs build, on the same engine.} + file:pyproject.toml: {status: matched, reason: present, with test-only dependencies rather than a package.} + file:noxfile.py: {status: matched, reason: present; sessions differ because this repo tests a template.} + file:justfile: {status: matched, reason: present, with recipes for template work.} + file:mkdocs.yml: {status: matched, reason: present, driving the same docs engine.} + file:renovate.json: {status: matched, reason: present, same dependency automation.} + file:README.md: {status: matched, reason: present.} + file:LICENSE: {status: matched, reason: present.} + file:CHANGELOG.md: {status: matched, reason: present, generated the same way.} + file:CODE_OF_CONDUCT.md: {status: matched, reason: present.} + file:CONTRIBUTING.md: + status: excluded + reason: >- + Generated projects get a short pointer to their published contributing guide. + This repo's equivalent lives at docs/pages/how-to/contribute.md and is reached + from the README, so a second stub at the root would be a third copy to keep in + step for no reader's benefit. + file:CLAUDE.md: {status: matched, reason: present and now tracked, reconciled with the Copilot copy.} + file:.copier-answers.yml: + status: excluded + reason: >- + Records which template version a project was generated from. This repository IS + the template; it has no upstream to record, which is the entire premise of this + capability. + file:CODEOWNERS: + status: excluded + reason: >- + DELIBERATE, AND NARROW. harden-fleet-security ships this to projects and its own + task 9.4 depends on one existing here for review-based protection. The protect-main + ruleset currently enforces review without it. Revisit if review ever needs to route + by path; until then a single-maintainer CODEOWNERS adds a file and no signal. + file:SECURITY.md: + status: excluded + reason: >- + NOT YET RESOLVED AS A DELIBERATE SKIP. A disclosure policy costs nothing, is a + static file, and satisfies an OpenSSF Scorecard check. It is absent here only + because harden-fleet-security shipped it to the template and never to this repo. + Recorded as a known shortfall, not a justification. + + # --- pre-commit / prek hooks -------------------------------------------- + hook:trailing-whitespace: {status: matched, reason: same builtin hook.} + hook:end-of-file-fixer: {status: matched, reason: same builtin hook.} + hook:check-yaml: {status: matched, reason: same builtin hook.} + hook:check-added-large-files: {status: matched, reason: same builtin hook.} + hook:check-json: {status: matched, reason: same builtin hook.} + hook:check-toml: {status: matched, reason: same builtin hook.} + hook:check-merge-conflict: {status: matched, reason: same builtin hook.} + hook:mixed-line-ending: {status: matched, reason: same builtin hook.} + hook:commitizen: {status: matched, reason: same commit-msg stage hook.} + hook:gitleaks: {status: matched, reason: same secret-scanning hook.} + hook:ruff: {status: matched, reason: same linter, pinned by uv.lock.} + hook:ruff-format: {status: matched, reason: same formatter, pinned by uv.lock.} + hook:rumdl: {status: matched, reason: same markdown linter.} + hook:rumdl-fmt: {status: matched, reason: same markdown formatter.} + hook:interrogate: + status: matched + reason: >- + Present, but scoped differently: the template scopes it to src/, and this repo + has none, so it runs over the repo's own Python with skill payloads excluded. + hook:no-rst-citations: + status: excluded + reason: >- + Bans reStructuredText citation syntax in numpydoc docstrings so griffe does not + render it literally on API pages. This repo has no API pages and no numpydoc + docstrings to mis-render. + hook:ty: + status: excluded + reason: >- + Type-checks src/. This repo has no src/. Its Python is tests and noxfile, which + ruff covers; adding a type gate over test fixtures would be ceremony without a + published interface to protect. + +# --------------------------------------------------------------------------- +# Gates held in repository settings, not files. +# +# File derivation cannot see these. A manifest limited to files would report full +# parity on a repo whose merge gate is switched off, which is exactly what happened +# here for four releases after the roll-up shipped. Each entry records HOW its state +# is determined and at WHAT CADENCE, because that judgment is per control. +# --------------------------------------------------------------------------- +settings_gates: + ruleset-requires-roll-up: + status: matched + mechanism: repository ruleset `protect-main`, enforcement active + determined_by: >- + Unattended query of the hosting platform via the `gh` CLI, in + tests/test_parity_manifest.py. No new credential was introduced: the check reuses + whatever authentication the runner already has, and when `gh` is missing or + unauthenticated it SKIPS rather than passes, so an unreachable check can never + read as a satisfied one. That was the resolution of the credential question -- + a fine-grained administration-read token was considered and judged unnecessary + once the existing CLI auth proved sufficient for reading rulesets. + cadence: every test run, initiated by no one + reason: >- + The roll-up check name exists to be the single required check. Nothing required + it between v0.32.0 and 2026-07-26, and nothing recorded that gap while it was + open. This entry is checked unattended precisely because the failure mode is + silent reversion: a ruleset deleted or renamed later looks identical to one that + was never created. + changelog-automation-identity: + status: matched + mechanism: GitHub App token minted per run via actions/create-github-app-token + determined_by: workflow file inspection (the secret names it references) + cadence: every test run + reason: >- + Replaced a broad non-expiring PAT. The workflow wiring is a file and therefore + checkable; whether the org secrets are actually provisioned is not, and a run + failing loudly is the signal there. diff --git a/tests/test_parity_manifest.py b/tests/test_parity_manifest.py new file mode 100644 index 0000000..5b3f2d4 --- /dev/null +++ b/tests/test_parity_manifest.py @@ -0,0 +1,324 @@ +"""The parity manifest: every gate this repo ships is answered for this repo. + +This repository generates seven packages and receives nothing back. `template/` fans +out; the root has no inbound path and, until this test, nothing that knew its own +configuration was meant to resemble what it hands out. + +The failure mode is absence, not drift: a missing workflow, a missing job, a missing +pin. A test that renders the template and diffs files cannot catch a file that is not +there, so this asserts coverage instead. Every gate the template ships resolves to +`matched`, `excluded` (with a written reason) or `leading`. Anything unanswered fails. + +The shipped-gate set is DERIVED from a rendered project, never hand-listed. A +hand-maintained list omits new gates silently, which is the exact defect this exists +to prevent: it would be a check that passes because it measures a set that no longer +matches reality. +""" + +import json +import shutil +import subprocess +from pathlib import Path + +import pytest +import yaml + +_REPO = Path(__file__).resolve().parent.parent +_MANIFEST_PATH = Path(__file__).resolve().parent / "parity_manifest.yml" +_VALID_STATUSES = {"matched", "excluded", "leading"} + + +def _manifest(): + """The hand-written half: what this repo claims about each shipped gate.""" + return yaml.safe_load(_MANIFEST_PATH.read_text(encoding="utf-8")) + + +def _derive_shipped_gates(project_dir): + """Every gate a generated project receives, read from the rendered project. + + Derived, not listed. Workflow *job* identifiers and pre-commit *hook* ids are the + two places a gate is declared, and both are structural, so a gate added to the + template appears here without anyone editing this file. + """ + gates = set() + for workflow_path in sorted((project_dir / ".github" / "workflows").glob("*.yml")): + workflow = yaml.safe_load(workflow_path.read_text(encoding="utf-8")) + for job_id in workflow.get("jobs") or {}: + gates.add(f"{workflow_path.name}:{job_id}") + + hooks_config = yaml.safe_load((project_dir / ".pre-commit-config.yaml").read_text(encoding="utf-8")) + for repo in hooks_config.get("repos") or []: + for hook in repo.get("hooks") or []: + gates.add(f"hook:{hook['id']}") + + # Root-level files a project receives. Jobs and hooks are not the only shape a + # control takes: CODEOWNERS and SECURITY.md are governance controls that exist + # purely as files, and a derivation reading only jobs and hooks was blind to + # both -- which is exactly how they reached seven generated projects and never + # this one. The project root is a bounded, structural set, so this stays derived + # rather than becoming a second hand-maintained list. + for path in sorted(project_dir.iterdir()): + if path.is_file(): + gates.add(f"file:{path.name}") + return gates + + +def test_derivation_finds_gates_at_all(copie_session_default): + """Guard the input set before asserting anything about it. + + If derivation ever returns nothing, every coverage assertion below passes + vacuously and the parity gap silently reopens. That is the failure this whole + capability exists to catch, so check it here first rather than trusting it. + """ + gates = _derive_shipped_gates(copie_session_default.project_dir) + assert len(gates) > 20, f"derivation found only {len(gates)} gates; it is probably reading the wrong tree" + assert any(g.startswith("hook:") for g in gates), "no pre-commit hooks derived" + assert any(":" in g and not g.startswith("hook:") for g in gates), "no workflow jobs derived" + + +def test_every_shipped_gate_is_answered(copie_session_default): + """A gate the template ships with no manifest entry fails, naming the gate. + + This is the whole point. `fix(ci): pin exact uv version in all setup-uv steps` + quantified over the template and silently meant nothing about this repo; an + unanswered entry is what makes that visible instead of invisible. + """ + shipped = _derive_shipped_gates(copie_session_default.project_dir) + answered = set(_manifest()["file_gates"]) + unanswered = sorted(shipped - answered) + assert not unanswered, ( + f"the template ships {len(unanswered)} gate(s) this repo has not accounted for: {unanswered}. " + "Add an entry to tests/parity_manifest.yml marking each matched, excluded (with a reason), or leading." + ) + + +def test_no_stale_manifest_entries(copie_session_default): + """An entry for a gate the template no longer ships is reported, not retained. + + Coverage checks usually only notice additions. A manifest that accumulates entries + for deleted gates slowly stops describing anything, and its passing tells you less + every release. + """ + shipped = _derive_shipped_gates(copie_session_default.project_dir) + stale = sorted(set(_manifest()["file_gates"]) - shipped) + assert not stale, ( + f"manifest answers {len(stale)} gate(s) the template no longer ships: {stale}. Remove the stale entries." + ) + + +@pytest.mark.parametrize("section", ["file_gates", "settings_gates"]) +def test_every_entry_has_a_valid_status_and_a_reason(section): + """An exclusion with no reason is an omission wearing a label. + + The reason text is the artifact this capability produces: it turns "this repo + skips that" from silently true into something a reviewer can disagree with in a + pull request. A blank one defeats the purpose entirely. + """ + for gate, entry in _manifest()[section].items(): + assert entry.get("status") in _VALID_STATUSES, ( + f"{gate}: status {entry.get('status')!r} is not one of {sorted(_VALID_STATUSES)}" + ) + reason = (entry.get("reason") or "").strip() + assert reason, f"{gate}: every entry needs a reason, even a one-liner" + + # `excluded` and `leading` are where the judgment lives, and where an omission + # can hide behind a label. "same builtin hook" is a complete answer for a + # matched entry; "n/a" is not a complete answer for a skipped gate. + if entry["status"] in {"excluded", "leading"}: + assert len(reason) > 40, ( + f"{gate}: status {entry['status']} needs a substantive reason a reviewer can disagree with, " + f"got {reason!r}" + ) + + +def test_settings_entries_record_how_and_how_often_they_are_checked(): + """Settings-held controls must say how their state is determined, and at what cadence. + + Cadence is deliberately per control rather than fixed globally: an unattended check + costs a credential, and that trade belongs to the control. What is not optional is + writing down which trade was made. + """ + for gate, entry in _manifest()["settings_gates"].items(): + for field in ("mechanism", "determined_by", "cadence"): + assert (entry.get(field) or "").strip(), f"{gate}: settings entry must record {field!r}" + + +def test_the_manifest_covers_controls_that_are_not_files(): + """A file-derived manifest would pass on a repo whose merge gate is switched off. + + Workflows, hooks, CODEOWNERS and lint config are files. Rulesets, required status + checks and secret provisioning are not, and no amount of reading the working tree + reveals them. This asserts the manifest reaches past what derivation can see. + """ + settings_gates = _manifest()["settings_gates"] + assert settings_gates, "the manifest covers no settings-held controls, so it is blind to the merge gate" + assert "ruleset-requires-roll-up" in settings_gates, "the merge gate itself must be one of the covered controls" + + +def required_checks_from_rulesets(rulesets): + """Status checks a set of ruleset payloads actually requires. + + Split out from the network call so the discriminating logic can be tested without + touching a live setting. Verifying "does this detector notice a missing ruleset?" + by deleting the real one would mean removing this repository's merge protection to + prove that removing it is noticed, which is not a trade worth making. Every fleet + repository already requires the roll-up, so there is no live negative control + either. Feeding synthetic payloads is what is left, and it is the better test: + deterministic, offline, and able to cover cases the live repo cannot be put into. + """ + checks = set() + for body in rulesets: + if body.get("enforcement") != "active": + continue + for rule in body.get("rules") or []: + if rule.get("type") == "required_status_checks": + params = rule.get("parameters") or {} + checks.update(c.get("context") for c in params.get("required_status_checks") or []) + return checks + + +def _required_checks_on_default_branch(): + """Status checks the hosting platform actually requires, or None if unknowable. + + Unattended: no person initiates it, so a ruleset deleted later fails the next run + rather than waiting for someone to look. Returns None when the platform cannot be + reached or credentials are absent, which the caller reports as a skip rather than + a pass -- an unreachable check must never read as a satisfied one. + """ + if shutil.which("gh") is None: + return None + probe = subprocess.run( + ["gh", "api", "repos/stateful-y/python-package-copier/rulesets"], + capture_output=True, + text=True, + check=False, + ) + if probe.returncode != 0: + return None + details = [] + for ruleset in json.loads(probe.stdout): + detail = subprocess.run( + ["gh", "api", f"repos/stateful-y/python-package-copier/rulesets/{ruleset['id']}"], + capture_output=True, + text=True, + check=False, + ) + if detail.returncode == 0: + details.append(json.loads(detail.stdout)) + return required_checks_from_rulesets(details) + + +@pytest.mark.slow +@pytest.mark.integration +def test_the_roll_up_merge_gate_is_actually_required(): + """A ruleset must require the roll-up check by name, checked without a person asking. + + The roll-up job name was chosen to survive matrix and conditional changes so it + could be the one required check. It was required by nothing for four releases + after it shipped, and nothing recorded that. A control configured once and later + removed is indistinguishable from one never configured, which is why a + maintainer-initiated check would not be enough here. + """ + required = _required_checks_on_default_branch() + if required is None: + pytest.skip("hosting platform not reachable or gh not authenticated; state unknown, not assumed satisfied") + assert "CI passed" in required, ( + f"no active ruleset requires the roll-up check 'CI passed'; required checks are {sorted(required)}. " + "Every check on this repository is advisory until one does." + ) + + +def _repo_hook_ids(): + """Hook ids this repository actually runs.""" + config = yaml.safe_load((_REPO / ".pre-commit-config.yaml").read_text(encoding="utf-8")) + return {hook["id"] for repo in config.get("repos") or [] for hook in repo.get("hooks") or []} + + +def _repo_workflow_jobs(): + """Workflow jobs this repository actually runs, as `:`.""" + jobs = set() + for path in sorted((_REPO / ".github" / "workflows").glob("*.yml")): + workflow = yaml.safe_load(path.read_text(encoding="utf-8")) + for job_id in workflow.get("jobs") or {}: + jobs.add(f"{path.name}:{job_id}") + return jobs + + +def test_a_matched_claim_is_true_not_merely_asserted(): + """`matched` must mean the equivalent exists here, not that someone typed "matched". + + Without this the manifest is documentation wearing a test's clothes: deleting this + repo's nightly canary outright left every check green, because "matched" was a + string in a YAML file that nothing compared against reality. That is precisely the + defect this capability exists to eliminate, reproduced inside its own mechanism. + + Two of the three gate shapes verify structurally. A shipped root *file* must exist + here by name; a shipped *hook* id must appear in this repo's hook config. Workflow + *jobs* cannot be matched by name, because this repo's jobs are named for testing a + template rather than a package, so a matched job entry must name its counterpart + with an `equivalent:` field, and that counterpart must exist. + """ + missing = [] + for gate, entry in _manifest()["file_gates"].items(): + if entry["status"] != "matched": + continue + + if gate.startswith("file:"): + name = gate.removeprefix("file:") + if not (_REPO / name).exists(): + missing.append(f"{gate}: claims matched but {name} does not exist in this repo") + + elif gate.startswith("hook:"): + hook_id = gate.removeprefix("hook:") + if hook_id not in _repo_hook_ids(): + missing.append(f"{gate}: claims matched but no hook `{hook_id}` runs in this repo") + + else: # a workflow job + equivalent = entry.get("equivalent") + if not equivalent: + missing.append(f"{gate}: claims matched but names no `equivalent:` in this repo to check") + elif equivalent not in _repo_workflow_jobs() and equivalent != "not-a-workflow-job": + missing.append(f"{gate}: claims equivalent `{equivalent}`, which this repo does not run") + + assert not missing, "manifest claims that reality does not support:\n " + "\n ".join(missing) + + +def _ruleset(enforcement="active", contexts=("CI passed",), rule_type="required_status_checks"): + """A minimal ruleset payload in the shape the hosting platform returns.""" + return { + "enforcement": enforcement, + "rules": [ + { + "type": rule_type, + "parameters": {"required_status_checks": [{"context": c} for c in contexts]}, + } + ], + } + + +@pytest.mark.parametrize( + ("label", "payloads"), + [ + ("no rulesets at all", []), + ("ruleset exists but is not enforced", [_ruleset(enforcement="evaluate")]), + ("ruleset enforces review only, no status checks", [_ruleset(rule_type="pull_request")]), + ("ruleset requires other checks but not the roll-up", [_ruleset(contexts=("Validate PR Title",))]), + ("roll-up renamed, so the required name no longer matches any job", [_ruleset(contexts=("CI-passed",))]), + ], +) +def test_the_roll_up_detector_notices_every_way_the_gate_can_be_absent(label, payloads): + """The detector must go red for each way the merge gate stops being enforced. + + This is the assertion that gives the live check meaning. Its purpose is catching + silent reversion, and a detector that cannot distinguish "required" from "quietly + no longer required" would report the gate healthy forever. The renamed case is the + subtle one: a ruleset still exists and still requires *a* check, but the name no + longer matches any job, so it blocks nothing while looking configured. + """ + assert "CI passed" not in required_checks_from_rulesets(payloads), f"the detector failed to notice: {label}" + + +def test_the_roll_up_detector_accepts_a_correctly_configured_ruleset(): + """And it must not cry wolf on the real configuration, or it will be switched off.""" + assert "CI passed" in required_checks_from_rulesets([_ruleset()]) + assert "CI passed" in required_checks_from_rulesets([_ruleset(enforcement="evaluate", contexts=("x",)), _ruleset()]) diff --git a/tests/test_repo_docs.py b/tests/test_repo_docs.py index be04de6..7e7bf94 100644 --- a/tests/test_repo_docs.py +++ b/tests/test_repo_docs.py @@ -12,6 +12,7 @@ the fleet change had to remove. """ +import re import shutil import subprocess from pathlib import Path @@ -74,26 +75,85 @@ def test_repo_docs_theme_override_is_excluded_by_location(): assert "exclude_docs" not in config, "no exclude_docs is needed once the override is outside docs_dir" +def _publishing_engine(): + """The engine `.readthedocs.yml` actually publishes with. + + Read from the config rather than hardcoded, so switching engines cannot leave the + gate asserting over the old one -- which is precisely the state this replaced. + """ + commands = yaml.safe_load((_REPO / ".readthedocs.yml").read_text(encoding="utf-8"))["build"]["commands"] + engines = {e for e in ("zensical", "mkdocs") for c in commands if f" {e} " in f" {c} "} + assert len(engines) == 1, f".readthedocs.yml runs {engines or 'no known engine'}; cannot tell what publishes" + return engines.pop() + + +def test_every_docs_entry_point_uses_the_publishing_engine(): + """No build or serve entry point may invoke an engine other than the published one. + + This repo ran four entry points across two engines: `.readthedocs.yml` and the + `justfile` on Zensical, `noxfile.py` and the content gate below on MkDocs. The two + that verified the docs were the two that disagreed with the one that shipped them, + so the gate could not observe the failure it was written for. A cheap textual check + keeps that split from reopening. + """ + engine = _publishing_engine() + other = "mkdocs" if engine == "zensical" else "zensical" + + # The two entry points spell an invocation differently: the justfile writes + # `uv run mkdocs build`, the noxfile writes `session.run("mkdocs", "build", ...)`. + # Matching the literal string "mkdocs build" finds the first and silently never + # matches the second, so the check would pass over the file that actually broke. + # Allow quotes, commas and whitespace between the engine and its verb. + invocation = re.compile(rf"{other}[\"']?\s*,?\s*[\"']?(build|serve)\b") + + for name in ("noxfile.py", "justfile"): + text = (_REPO / name).read_text(encoding="utf-8") + offending = [ln.strip() for ln in text.splitlines() if invocation.search(ln)] + assert not offending, ( + f"{name} invokes {other} while .readthedocs.yml publishes with {engine}: {offending}. " + "A gate that builds with a different engine than the one that ships cannot see its failures." + ) + + @pytest.mark.slow @pytest.mark.integration def test_repo_docs_build_ships_content_not_an_empty_site(tmp_path): """A real build produces non-empty pages and leaks no theme override. - A green `--strict` build is not evidence: the empty-site bug and a leaked - asset both pass at exit 0. This asserts on content instead. + A green `--strict` build is not evidence: the empty-site bug and a leaked asset + both pass at exit 0. This asserts on content instead. + + Built with the *publishing* engine. Zensical has no `--site-dir`, so instead of + redirecting the output the sources are copied into a scratch tree and built there: + that keeps the repo's own `site/` untouched and stops parallel workers racing over + one output directory. Only the inputs a build needs are copied, so the copy is + cheap and a stray file in the repo cannot influence the result. """ if shutil.which("uv") is None: pytest.skip("uv is required to build the docs") - site = tmp_path / "site" + + engine = _publishing_engine() + work = tmp_path / "docs-build" + work.mkdir() + for item in ("docs", "docs_theme", "mkdocs.yml", "pyproject.toml", "uv.lock"): + source = _REPO / item + if source.is_dir(): + shutil.copytree(source, work / item) + else: + shutil.copy2(source, work / item) + build = subprocess.run( - ["uv", "run", "--group", "docs", "mkdocs", "build", "--clean", "--strict", "--site-dir", str(site)], - cwd=_REPO, + ["uv", "run", "--no-project", "--group", "docs", engine, "build", "-s"], + cwd=work, capture_output=True, text=True, check=False, ) assert build.returncode == 0, build.stderr[-2000:] + site = work / "site" + assert site.is_dir(), f"{engine} produced no site directory at all" + index = site / "index.html" assert index.is_file() and index.stat().st_size > 500, ( "the home page is missing or empty -- the build shipped nothing" diff --git a/tests/test_repo_workflows.py b/tests/test_repo_workflows.py new file mode 100644 index 0000000..6530ade --- /dev/null +++ b/tests/test_repo_workflows.py @@ -0,0 +1,218 @@ +"""CI invariants for THIS repository's own workflows. + +`test_github_workflows.py` asserts these properties over the workflows the template +*generates*. Nothing asserted them over the workflows this repository *runs*, which is +how `fix(ci): pin exact uv version in all setup-uv steps` (#220) pinned every step it +could see and left all of this repo's own steps floating: "all" silently meant "all in +the template". The count then grew from five to six while nobody was looking. + +These tests are the other half of that pair. They exist so a property the template +enforces downstream cannot be quietly absent upstream. +""" + +import filecmp +import re +import subprocess +from pathlib import Path + +import yaml + +_REPO = Path(__file__).resolve().parent.parent +_WORKFLOWS = _REPO / ".github" / "workflows" +_EXACT_SEMVER = re.compile(r"^\d+\.\d+\.\d+$") + + +def _repo_workflow_paths(): + """Every workflow file this repository runs.""" + return sorted(_WORKFLOWS.glob("*.yml")) + + +def _setup_uv_steps(): + """Collect every setup-uv step across this repo's workflows as (file, with-block). + + Discovered by walking the parsed workflows rather than from a hardcoded list, so a + step added to any workflow, existing or new, is covered without editing this test. + That is the property that failed before: the count changed and nothing noticed. + """ + steps = [] + for path in _repo_workflow_paths(): + workflow = yaml.safe_load(path.read_text(encoding="utf-8")) + for job in (workflow.get("jobs") or {}).values(): + for step in job.get("steps") or []: + if str(step.get("uses", "")).startswith("astral-sh/setup-uv"): + steps.append((path.name, step.get("with") or {})) + return steps + + +def _copier_default_uv_version(): + """The uv version copier.yml pins for generated projects. + + Read rather than hardcoded so this repo's pin is checked against the same source of + truth the fleet gets, and a bump in one place fails here instead of drifting. + """ + config = yaml.safe_load((_REPO / "copier.yml").read_text(encoding="utf-8")) + return str(config["uv_version"]["default"]) + + +def test_the_assertion_reads_this_repository_not_generated_output(): + """This test file's inputs are the root workflows, which is the whole point. + + The generated-output test passed for years while this repo went unchecked. If this + ever collects nothing, the pin assertions below would pass vacuously and the gap + would silently reopen, so assert the inputs before asserting anything about them. + """ + assert _WORKFLOWS.is_dir(), "this repository has no .github/workflows to check" + assert _repo_workflow_paths(), "no workflows found; the pin assertions would pass over an empty set" + assert _setup_uv_steps(), "no astral-sh/setup-uv steps found in this repository's workflows" + + +def test_every_repo_setup_uv_step_pins_an_exact_version(): + """Every setup-uv step in this repo pins an exact X.Y.Z, matching what it ships. + + Resolving "latest" is an un-retried network call and the point at which a transient + blip fails CI before any real work. A range like "0.10" still resolves over the + network, so a non-empty version is not enough. + """ + unpinned = [f for f, w in _setup_uv_steps() if not w.get("version")] + assert not unpinned, f"setup-uv steps with no uv version pin in: {sorted(set(unpinned))}" + + inexact = [(f, w["version"]) for f, w in _setup_uv_steps() if not _EXACT_SEMVER.match(str(w["version"]))] + assert not inexact, f"uv versions that are not an exact X.Y.Z pin: {inexact}" + + +def test_repo_uv_pin_is_uniform_and_matches_the_version_shipped_to_projects(): + """One uv version across this repo, and the same one generated projects get. + + A repo pinning a different uv than it hands out is testing the template against a + toolchain no generated project uses. + """ + versions = {str(w["version"]) for _, w in _setup_uv_steps() if w.get("version")} + assert len(versions) == 1, f"setup-uv steps pin differing uv versions: {sorted(versions)}" + assert versions == {_copier_default_uv_version()}, ( + f"this repo pins uv {sorted(versions)} but copier.yml ships {_copier_default_uv_version()} to projects" + ) + + +def test_every_repo_uv_pin_carries_a_renovate_annotation(): + """A pin with no annotation is a pin Renovate never bumps. + + Checked against the raw text, not the parsed YAML: the annotation is a comment, so + the parser drops it and a purely structural check cannot see it missing. + """ + for path in _repo_workflow_paths(): + lines = path.read_text(encoding="utf-8").splitlines() + for i, line in enumerate(lines): + preceding = "\n".join(lines[max(0, i - 4) : i]) + if line.strip().startswith("version:") and "astral-sh/setup-uv" in preceding: + previous = lines[i - 1].strip() if i > 0 else "" + assert previous.startswith("# renovate:"), ( + f"{path.name} line {i + 1}: setup-uv version pin has no `# renovate:` annotation, " + "so Renovate cannot bump it and it will rot at whatever it was set to" + ) + + +def _tracked(prefix): + """Files git tracks under a prefix, as paths relative to that prefix. + + Scoped to tracked files on purpose. Both skill trees deliberately ignore + `openspec-*` skills, which `.gitignore` describes as "tool state that openspec + rewrites, not project content", and the CLI writes them into whichever tree it + likes. A working-tree comparison therefore reports differences that are neither + drift nor actionable -- an earlier read of this pair did exactly that and reported + drift that did not exist. + """ + out = subprocess.run( + ["git", "ls-files", "-z", "--", prefix], + cwd=_REPO, + capture_output=True, + text=True, + check=True, + ) + return sorted(p[len(prefix) :].lstrip("/") for p in out.stdout.split("\0") if p) + + +def test_repo_skill_mirrors_track_the_same_files(): + """The two skill trees ship the same set of files. + + `CLAUDE.md` tells contributors to edit both copies "or they will drift", and + nothing has ever checked that. The template's own mirror test compares skill names + and the presence of each SKILL.md, which two diverged copies would pass. + """ + claude = _tracked(".claude/skills") + github = _tracked(".github/skills") + assert claude, "no tracked files under .claude/skills; this check would pass vacuously" + assert claude == github, f"skill mirrors track different files: {sorted(set(claude) ^ set(github))}" + + +def test_repo_skill_mirrors_are_byte_identical(): + """Same files is not enough: the contents must match too. + + A name-level check passes while one copy silently carries different instructions, + which is the failure mode that matters. Claude Code reads one tree and Copilot the + other, so divergence means the two assistants are following different rules. + """ + differing = [ + rel + for rel in _tracked(".claude/skills") + if not filecmp.cmp(_REPO / ".claude/skills" / rel, _REPO / ".github/skills" / rel, shallow=False) + ] + assert not differing, f"skill mirrors differ in content: {differing}" + + +_INSTRUCTION_FILES = (Path("CLAUDE.md"), Path(".github/copilot-instructions.md")) + + +def test_the_agent_instruction_files_are_tracked(): + """Both instruction files must be in version control. + + `CLAUDE.md` was gitignored, so the file telling every assistant how this repo works + never entered a pull request, was never reviewed, and did not survive a clone. That + is also why it drifted from its Copilot counterpart with nothing to notice: a file + outside version control cannot be diffed against anything. + """ + for rel in _INSTRUCTION_FILES: + assert (_REPO / rel).is_file(), f"{rel} is missing" + ignored = subprocess.run(["git", "check-ignore", "-q", str(rel)], cwd=_REPO, capture_output=True, check=False) + assert ignored.returncode != 0, f"{rel} is gitignored, so it can never be reviewed or tested" + + +def test_the_agent_instruction_files_agree_on_shared_content(): + """The two files carry the same guidance below their titles. + + They diverged by exactly the section instructing that both be kept in step, which + is the joke that writes itself: the rule lived in only one of the two files it + governed. Only the top-level heading may differ, since one names its audience. + """ + bodies = [] + for rel in _INSTRUCTION_FILES: + lines = (_REPO / rel).read_text(encoding="utf-8").splitlines() + body = lines[1:] if lines and lines[0].startswith("# ") else lines + bodies.append((rel, [ln.rstrip() for ln in body])) + + (first_rel, first), (second_rel, second) = bodies + if first != second: + only_first = [ln for ln in first if ln and ln not in second] + only_second = [ln for ln in second if ln and ln not in first] + raise AssertionError( + f"{first_rel} and {second_rel} disagree.\n" + f" only in {first_rel}: {only_first[:3]}\n" + f" only in {second_rel}: {only_second[:3]}" + ) + + +def test_instruction_files_name_no_path_that_does_not_exist(): + """A path cited as a canonical source must actually be there. + + The instructions claimed `polish-changelog`'s canonical source was a directory + under `template/`, and that directory exists only on an unmerged branch. Guidance + that points at nothing sends whoever follows it looking for a file that is not + there, and quietly licenses editing the wrong copy. + """ + path_like = re.compile(r"`((?:template/|docs/|tests/|src/|openspec/)[\w./<>{}-]+)`") + for rel in _INSTRUCTION_FILES: + text = (_REPO / rel).read_text(encoding="utf-8") + for cited in sorted(set(path_like.findall(text))): + if any(ch in cited for ch in "<>{}"): + continue # a placeholder, not a literal path + target = _REPO / cited.rstrip("/") + assert target.exists(), f"{rel} cites `{cited}`, which does not exist on this branch" diff --git a/tests/test_template.py b/tests/test_template.py index 094d400..71dd9c0 100644 --- a/tests/test_template.py +++ b/tests/test_template.py @@ -12,6 +12,7 @@ from pathlib import Path import pytest +import yaml from _build_layout import BUILD_DIR @@ -5079,3 +5080,39 @@ def test_generated_index_pages_introduce_their_subpages(copie_session_default): assert "" in index.read_text(encoding="utf-8"), ( f"the {quadrant} index describes its quadrant but never lists its own pages" ) + + +def test_claude_md_is_seeded_and_committable(copie_session_default): + """A generated project gets a project-instructions file it is allowed to commit. + + The template used to ignore six `CLAUDE*` filename variants and ship no + replacement, so no generated project could record its own invariants anywhere the + repository could see. Across the fleet that left the knowledge describing seven + projects living outside all seven of them. + """ + project = copie_session_default.project_dir + claude_md = project / "CLAUDE.md" + assert claude_md.is_file(), "no CLAUDE.md was seeded into the generated project" + assert claude_md.stat().st_size > 200, "the seeded CLAUDE.md is empty or a stub" + + gitignore = (project / ".gitignore").read_text(encoding="utf-8") + for variant in ("CLAUDE.md", "CLAUDE_INSTRUCTIONS.md", ".claude.md", "claude-prompt.md"): + assert not re.search(rf"^{re.escape(variant)}$", gitignore, re.MULTILINE), ( + f"{variant} is still ignored, so a project cannot commit its own instructions" + ) + + +def test_claude_md_is_registered_as_seed_once(): + """`CLAUDE.md` must be in `_skip_if_exists`, or an update reverts a project's edits. + + Copier applies an update as a diff against the *template's* copy. Once a project + has rewritten this file wholesale, one shifted line rejects the whole hunk and the + page reverts to the stub, with the project's content left in a `.rej` nobody + reads. That is not hypothetical: v0.22.0 did exactly this to five projects' + getting-started.md from a whitespace-only change. `_skip_if_exists` is the only + thing copier itself honours here; the update skill's classification is advisory. + """ + config = yaml.safe_load(Path("copier.yml").read_text(encoding="utf-8")) + assert "CLAUDE.md" in config["_skip_if_exists"], ( + "CLAUDE.md is shipped but not seed-once; a template update would revert every project's own instructions" + ) From d4e74059ba1d33f21ce68213fe33bb980c43c225 Mon Sep 17 00:00:00 2001 From: Guillaume Tauzin <4648633+gtauzin@users.noreply.github.com> Date: Sun, 26 Jul 2026 14:41:48 +0200 Subject: [PATCH 2/5] fix(tests): assert cited paths are tracked, not merely present The instruction-file check asserted a cited path exists. On a maintainer's machine `openspec/` exists, so a citation under it resolved; it is gitignored, so a fresh clone has no such path and CI failed on a citation that pointed at nothing for every reader but the author. The test was weaker locally than in CI, which is the same defect class it was written to catch. Asking git instead makes both environments agree and matches what the guidance actually promises a reader. Also drops the `openspec/specs/` citation itself, since that tree is local-only. --- .github/copilot-instructions.md | 2 +- CLAUDE.md | 2 +- tests/test_repo_workflows.py | 17 +++++++++++++++-- 3 files changed, 17 insertions(+), 4 deletions(-) diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md index 4263c30..6d38f1d 100644 --- a/.github/copilot-instructions.md +++ b/.github/copilot-instructions.md @@ -50,4 +50,4 @@ Skills in `.claude/skills/` mirror `.github/skills/` (kept for GitHub Copilot). Skills that ship to *generated* projects live in `template/.github/skills/` and are a separate set — they are not mirrored here. Within `template/`, the `.github/skills/` and `.claude/skills/` copies must stay byte-identical. Note that `test_claude_skills_generated` checks only that both trees carry the same skill *names* and that each has a `SKILL.md`; two copies whose contents diverged would still pass it. -`polish-changelog` is **not on `main`**. Its content lives on the unmerged branch `feat/changelog-polish-skill` and in `~/.claude/skills/polish-changelog/` (global, out-of-repo, updated by hand), so it is available in every repo but tracked in none. The `changelog-polish` capability spec under `openspec/specs/` is canonical and describes behaviour the repo does not currently ship. Either land that branch or treat the global copy as the only source. +`polish-changelog` is **not on `main`**. Its content lives on the unmerged branch `feat/changelog-polish-skill` and in `~/.claude/skills/polish-changelog/` (global, out-of-repo, updated by hand), so it is available in every repo but tracked in none. A canonical `changelog-polish` capability spec exists in the OpenSpec tree, which is gitignored and therefore local-only, describing behaviour the repo does not currently ship. Either land that branch or treat the global copy as the only source. diff --git a/CLAUDE.md b/CLAUDE.md index a4a8b67..fa4a9fc 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -50,4 +50,4 @@ Skills in `.claude/skills/` mirror `.github/skills/` (kept for GitHub Copilot). Skills that ship to *generated* projects live in `template/.github/skills/` and are a separate set — they are not mirrored here. Within `template/`, the `.github/skills/` and `.claude/skills/` copies must stay byte-identical. Note that `test_claude_skills_generated` checks only that both trees carry the same skill *names* and that each has a `SKILL.md`; two copies whose contents diverged would still pass it. -`polish-changelog` is **not on `main`**. Its content lives on the unmerged branch `feat/changelog-polish-skill` and in `~/.claude/skills/polish-changelog/` (global, out-of-repo, updated by hand), so it is available in every repo but tracked in none. The `changelog-polish` capability spec under `openspec/specs/` is canonical and describes behaviour the repo does not currently ship. Either land that branch or treat the global copy as the only source. +`polish-changelog` is **not on `main`**. Its content lives on the unmerged branch `feat/changelog-polish-skill` and in `~/.claude/skills/polish-changelog/` (global, out-of-repo, updated by hand), so it is available in every repo but tracked in none. A canonical `changelog-polish` capability spec exists in the OpenSpec tree, which is gitignored and therefore local-only, describing behaviour the repo does not currently ship. Either land that branch or treat the global copy as the only source. diff --git a/tests/test_repo_workflows.py b/tests/test_repo_workflows.py index 6530ade..9e35888 100644 --- a/tests/test_repo_workflows.py +++ b/tests/test_repo_workflows.py @@ -214,5 +214,18 @@ def test_instruction_files_name_no_path_that_does_not_exist(): for cited in sorted(set(path_like.findall(text))): if any(ch in cited for ch in "<>{}"): continue # a placeholder, not a literal path - target = _REPO / cited.rstrip("/") - assert target.exists(), f"{rel} cites `{cited}`, which does not exist on this branch" + # Tracked, not merely present. Checking existence made this test weaker on + # a maintainer's machine than in CI: `openspec/` is gitignored, so a cited + # path under it resolved locally and vanished in a fresh clone. CI caught a + # citation that pointed at nothing for every reader but the author. Asking + # git makes both environments agree, and matches what the guidance means. + tracked = subprocess.run( + ["git", "ls-files", "--error-unmatch", "--", cited.rstrip("/")], + cwd=_REPO, + capture_output=True, + check=False, + ) + assert tracked.returncode == 0, ( + f"{rel} cites `{cited}`, which git does not track. Whoever clones this repo will not find it, " + "even if it exists on the machine the guidance was written on." + ) From 4cd700aab1c4cffee6f15e09b54c5486b9154b95 Mon Sep 17 00:00:00 2001 From: Guillaume Tauzin <4648633+gtauzin@users.noreply.github.com> Date: Sun, 26 Jul 2026 15:11:17 +0200 Subject: [PATCH 3/5] feat(security): adopt CodeQL, Scorecard and SECURITY.md on the template repo These three were shipped to generated projects by harden-fleet-security and never to this repository. It is public, so all three are free, and the parity manifest recorded them as shortfalls rather than judgments. Adopting them turns three "excluded, unresolved" entries into verified matches. SECURITY.md is reworded for what this repo is: a fix reaches a generated project only when that project runs `copier update`, so there is no patched release to consume. Also pins the seed-once guarantee behaviourally. test_claude_md_is_registered_ as_seed_once only asserted the entry exists; the new test re-runs the template over a project that rewrote its CLAUDE.md and asserts the content survives. Removing CLAUDE.md from _skip_if_exists makes it fail with the file clobbered, so the guard is load-bearing rather than decorative. The test carries a control (an unguarded README.md that must be overwritten) so a run that clobbered nothing cannot pass as proof. --- .github/workflows/codeql.yml | 33 +++++++++++++++++++ .github/workflows/scorecard.yml | 41 ++++++++++++++++++++++++ SECURITY.md | 17 ++++++++++ tests/parity_manifest.yml | 29 +++++++++-------- tests/test_template.py | 57 +++++++++++++++++++++++++++++++++ 5 files changed, 163 insertions(+), 14 deletions(-) create mode 100644 .github/workflows/codeql.yml create mode 100644 .github/workflows/scorecard.yml create mode 100644 SECURITY.md diff --git a/.github/workflows/codeql.yml b/.github/workflows/codeql.yml new file mode 100644 index 0000000..efcc96b --- /dev/null +++ b/.github/workflows/codeql.yml @@ -0,0 +1,33 @@ +name: CodeQL + +on: + push: + branches: [main] + pull_request: + branches: [main] + schedule: + # Weekly, so a newly-disclosed query pattern still runs against unchanged code. + - cron: "27 3 * * 1" + +# Least-privilege default: read-only token; the analyze job widens as needed. +permissions: + contents: read + +jobs: + analyze: + name: Analyze (Python) + runs-on: ubuntu-latest + permissions: + # Required to upload CodeQL results to the Security tab. + security-events: write + steps: + - name: Checkout + uses: actions/checkout@v7 + + - name: Initialize CodeQL + uses: github/codeql-action/init@v3 + with: + languages: python + + - name: Perform CodeQL analysis + uses: github/codeql-action/analyze@v3 diff --git a/.github/workflows/scorecard.yml b/.github/workflows/scorecard.yml new file mode 100644 index 0000000..614024d --- /dev/null +++ b/.github/workflows/scorecard.yml @@ -0,0 +1,41 @@ +name: OpenSSF Scorecard + +on: + branch_protection_rule: + schedule: + # Weekly independent grade of the repo's security posture. + - cron: "21 4 * * 2" + push: + branches: [main] + +# Least-privilege default: read-only token; the analysis job widens as needed. +permissions: + contents: read + +jobs: + analysis: + name: Scorecard analysis + runs-on: ubuntu-latest + permissions: + # Upload the results to the Security tab and publish to the OpenSSF API. + security-events: write + id-token: write + steps: + - name: Checkout + uses: actions/checkout@v7 + with: + persist-credentials: false + + - name: Run analysis + uses: ossf/scorecard-action@v2.4.4 + with: + results_file: results.sarif + results_format: sarif + repo_token: ${{ github.token }} + # Publish results to the OpenSSF Scorecard API (public repos only). + publish_results: true + + - name: Upload SARIF to code scanning + uses: github/codeql-action/upload-sarif@v3 + with: + sarif_file: results.sarif diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 0000000..af7a67f --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,17 @@ +# Security Policy + +## Supported versions + +Security fixes are released for the latest tagged version of this template. Because this is a +Copier template rather than a published package, a fix reaches a generated project only when +that project runs `copier update`; projects are not patched in place. + +## Reporting a vulnerability + +Please report security vulnerabilities privately through GitHub's +[private vulnerability reporting](https://github.com/stateful-y/python-package-copier/security/advisories/new) +rather than opening a public issue. + +We aim to acknowledge a report within a few working days and will keep you updated as +we investigate and prepare a fix. Please give us a reasonable window to release a patch +before any public disclosure. diff --git a/tests/parity_manifest.yml b/tests/parity_manifest.yml index 7e18286..a1309b4 100644 --- a/tests/parity_manifest.yml +++ b/tests/parity_manifest.yml @@ -116,18 +116,20 @@ file_gates: status: matched reason: same issue-filing job, with the same dedupe-by-label behaviour. codeql.yml:analyze: - status: excluded + status: matched + equivalent: codeql.yml:analyze reason: >- - NOT YET RESOLVED AS A DELIBERATE SKIP. This repo is public, so CodeQL is free - and the template's own public-repo gate would ship it. It is absent here only - because harden-fleet-security shipped these files to the template and never to - this repo. Task 3.9 decides whether to adopt or genuinely exclude; until then - this entry records a known shortfall rather than a judgment. + Adopted. This repo is public so CodeQL is free, and it has Python worth scanning + (the test suite and noxfile). It was absent only because harden-fleet-security + shipped it to the template and never here, which is the shortfall this capability + exists to surface. scorecard.yml:analysis: - status: excluded + status: matched + equivalent: scorecard.yml:analysis reason: >- - NOT YET RESOLVED AS A DELIBERATE SKIP. Same shortfall as codeql above: public - repo, free service, shipped to the template only. Recorded, not justified. + Adopted. Scorecard independently grades whether the other controls remain in + force, which is the right shape of check for a repo whose merge gate went + unenforced for four releases with nothing noticing. # --- shipped root files ------------------------------------------------ # Derived from the rendered project root. Jobs and hooks are not the only shape a @@ -169,12 +171,11 @@ file_gates: ruleset currently enforces review without it. Revisit if review ever needs to route by path; until then a single-maintainer CODEOWNERS adds a file and no signal. file:SECURITY.md: - status: excluded + status: matched reason: >- - NOT YET RESOLVED AS A DELIBERATE SKIP. A disclosure policy costs nothing, is a - static file, and satisfies an OpenSSF Scorecard check. It is absent here only - because harden-fleet-security shipped it to the template and never to this repo. - Recorded as a known shortfall, not a justification. + Adopted, with wording corrected for what this repo is: a fix reaches a generated + project only when that project runs `copier update`, so there is no patched + release to consume. Also satisfies an OpenSSF Scorecard check. # --- pre-commit / prek hooks -------------------------------------------- hook:trailing-whitespace: {status: matched, reason: same builtin hook.} diff --git a/tests/test_template.py b/tests/test_template.py index 71dd9c0..286b0f6 100644 --- a/tests/test_template.py +++ b/tests/test_template.py @@ -5116,3 +5116,60 @@ def test_claude_md_is_registered_as_seed_once(): assert "CLAUDE.md" in config["_skip_if_exists"], ( "CLAUDE.md is shipped but not seed-once; a template update would revert every project's own instructions" ) + + +def test_claude_md_survives_a_later_template_run(tmp_path): + """A project's rewritten CLAUDE.md is not overwritten by a later template run. + + The behavioural half of what test_claude_md_is_registered_as_seed_once only + documents, and the same shape as the branding test above. Registering a file in + `_skip_if_exists` is a claim; re-running the template is what checks it. + + This matters because an attempt to reproduce the original v0.22.0 loss through + `copier update` did not reproduce it: copier's three-way merge preserved a + rewritten CLAUDE.md even with the guard removed, for both a wholly-rewritten file + and a file still resembling the template's with one line changed. So the guard's + necessity is not demonstrated by that route. What IS demonstrable, and what this + pins, is that an overwriting run leaves the project's version alone -- and the + control below shows such a run genuinely does overwrite anything unguarded, so the + survival is the guard's doing and not the run being a no-op. + """ + from copier import run_copy + + answers = { + "project_name": "Seeded", + "project_slug": "seeded", + "package_name": "seeded", + "description": "d", + "author_name": "A", + "author_email": "a@b.c", + "github_username": "u", + "version": "0.1.0", + "min_python_version": "3.11", + "max_python_version": "3.14", + "license": "MIT", + "include_actions": True, + "include_examples": True, + } + template_dir = str(Path(__file__).parent.parent) + project = tmp_path / "seeded-project" + run_copy(template_dir, str(project), data=answers, defaults=True, overwrite=True, unsafe=True, vcs_ref="HEAD") + + claude_md = project / "CLAUDE.md" + assert claude_md.is_file(), "the template must seed CLAUDE.md so a new project has something to start from" + owned = "# Seeded\n\nPROJECT-OWNED CONTENT THAT MUST SURVIVE.\n" + claude_md.write_text(owned, encoding="utf-8") + + # The control: a file the template owns and does NOT guard. If this survives too, + # the run overwrote nothing and the assertion below would prove nothing. + readme = project / "README.md" + readme.write_text("CLOBBER ME\n", encoding="utf-8") + + run_copy(template_dir, str(project), data=answers, defaults=True, overwrite=True, unsafe=True, vcs_ref="HEAD") + + assert readme.read_text(encoding="utf-8") != "CLOBBER ME\n", ( + "the control file was not overwritten, so this run proves nothing about the guard" + ) + assert claude_md.read_text(encoding="utf-8") == owned, ( + "the template overwrote the project's own CLAUDE.md; _skip_if_exists is not holding" + ) From 4677cbb72db3d19c9e31993c6197357ecb9c70eb Mon Sep 17 00:00:00 2001 From: Guillaume Tauzin <4648633+gtauzin@users.noreply.github.com> Date: Sun, 26 Jul 2026 15:48:19 +0200 Subject: [PATCH 4/5] test: exercise the failure branches the coverage report found unrun Codecov flagged 32 unrun lines in the new tests. Most were the bodies of slow/integration tests, which the coverage job excludes by selection and cannot reach. The rest were the branches that only execute when a check fails, which is precisely when they need to be right and precisely when nobody has watched them run. Extracts two pieces of logic so those branches can be driven directly: unsupported_matched_claims (the four distinct ways a manifest entry can claim a match that reality does not support) and instruction_body_difference (which must report both directions, since the section telling contributors to keep both instruction files in step was missing from one of them, and a one-sided comparison would have called that clean). Genuinely environment-dependent branches -- gh absent, network unreachable, uv missing -- are marked no-cover with the reason rather than chased with contrived tests. test_repo_workflows.py reaches 100%. --- tests/test_parity_manifest.py | 96 ++++++++++++++++++++++++++--------- tests/test_repo_docs.py | 2 +- tests/test_repo_workflows.py | 41 +++++++++++---- 3 files changed, 104 insertions(+), 35 deletions(-) diff --git a/tests/test_parity_manifest.py b/tests/test_parity_manifest.py index 5b3f2d4..fdbd6ce 100644 --- a/tests/test_parity_manifest.py +++ b/tests/test_parity_manifest.py @@ -185,7 +185,7 @@ def _required_checks_on_default_branch(): reached or credentials are absent, which the caller reports as a skip rather than a pass -- an unreachable check must never read as a satisfied one. """ - if shutil.which("gh") is None: + if shutil.which("gh") is None: # pragma: no cover - depends on the runner having gh return None probe = subprocess.run( ["gh", "api", "repos/stateful-y/python-package-copier/rulesets"], @@ -193,7 +193,7 @@ def _required_checks_on_default_branch(): text=True, check=False, ) - if probe.returncode != 0: + if probe.returncode != 0: # pragma: no cover - depends on network and credentials return None details = [] for ruleset in json.loads(probe.stdout): @@ -220,7 +220,7 @@ def test_the_roll_up_merge_gate_is_actually_required(): maintainer-initiated check would not be enough here. """ required = _required_checks_on_default_branch() - if required is None: + if required is None: # pragma: no cover - only when the platform is unreachable pytest.skip("hosting platform not reachable or gh not authenticated; state unknown, not assumed satisfied") assert "CI passed" in required, ( f"no active ruleset requires the roll-up check 'CI passed'; required checks are {sorted(required)}. " @@ -244,43 +244,89 @@ def _repo_workflow_jobs(): return jobs -def test_a_matched_claim_is_true_not_merely_asserted(): - """`matched` must mean the equivalent exists here, not that someone typed "matched". - - Without this the manifest is documentation wearing a test's clothes: deleting this - repo's nightly canary outright left every check green, because "matched" was a - string in a YAML file that nothing compared against reality. That is precisely the - defect this capability exists to eliminate, reproduced inside its own mechanism. +def unsupported_matched_claims(file_gates, repo_files, repo_hooks, repo_jobs): + """Matched claims the given reality does not support. - Two of the three gate shapes verify structurally. A shipped root *file* must exist - here by name; a shipped *hook* id must appear in this repo's hook config. Workflow - *jobs* cannot be matched by name, because this repo's jobs are named for testing a - template rather than a package, so a matched job entry must name its counterpart - with an `equivalent:` field, and that counterpart must exist. + Pure, so the discriminating branches can be exercised directly instead of only by + mutating the repository and watching the suite go red. Each branch is a distinct + way a manifest entry can lie, and a check whose failure paths are never executed is + a check nobody has seen work. """ - missing = [] - for gate, entry in _manifest()["file_gates"].items(): + unsupported = [] + for gate, entry in file_gates.items(): if entry["status"] != "matched": continue if gate.startswith("file:"): name = gate.removeprefix("file:") - if not (_REPO / name).exists(): - missing.append(f"{gate}: claims matched but {name} does not exist in this repo") + if name not in repo_files: + unsupported.append(f"{gate}: claims matched but {name} does not exist in this repo") elif gate.startswith("hook:"): hook_id = gate.removeprefix("hook:") - if hook_id not in _repo_hook_ids(): - missing.append(f"{gate}: claims matched but no hook `{hook_id}` runs in this repo") + if hook_id not in repo_hooks: + unsupported.append(f"{gate}: claims matched but no hook `{hook_id}` runs in this repo") else: # a workflow job equivalent = entry.get("equivalent") if not equivalent: - missing.append(f"{gate}: claims matched but names no `equivalent:` in this repo to check") - elif equivalent not in _repo_workflow_jobs() and equivalent != "not-a-workflow-job": - missing.append(f"{gate}: claims equivalent `{equivalent}`, which this repo does not run") + unsupported.append(f"{gate}: claims matched but names no `equivalent:` in this repo to check") + elif equivalent not in repo_jobs: + unsupported.append(f"{gate}: claims equivalent `{equivalent}`, which this repo does not run") + return unsupported - assert not missing, "manifest claims that reality does not support:\n " + "\n ".join(missing) + +def test_a_matched_claim_is_true_not_merely_asserted(): + """`matched` must mean the equivalent exists here, not that someone typed "matched". + + Without this the manifest is documentation wearing a test's clothes: deleting this + repo's nightly canary outright left every check green, because "matched" was a + string in a YAML file that nothing compared against reality. That is precisely the + defect this capability exists to eliminate, reproduced inside its own mechanism. + """ + repo_files = {p.name for p in _REPO.iterdir()} + unsupported = unsupported_matched_claims( + _manifest()["file_gates"], repo_files, _repo_hook_ids(), _repo_workflow_jobs() + ) + assert not unsupported, "manifest claims that reality does not support:\n " + "\n ".join(unsupported) + + +@pytest.mark.parametrize( + ("label", "gates", "expected_fragment"), + [ + ("a matched file that is not here", {"file:GONE.md": {"status": "matched"}}, "does not exist"), + ("a matched hook this repo does not run", {"hook:ghost": {"status": "matched"}}, "no hook `ghost`"), + ("a matched job naming no equivalent", {"x.yml:job": {"status": "matched"}}, "names no `equivalent:`"), + ( + "a matched job whose named equivalent is absent", + {"x.yml:job": {"status": "matched", "equivalent": "y.yml:nope"}}, + "which this repo does not run", + ), + ], +) +def test_every_way_a_matched_claim_can_lie_is_detected(label, gates, expected_fragment): + """Each failure branch reports, rather than one branch standing in for the rest. + + These are the lines that had never executed in a passing run. Exercising them here + is what makes the assertion above meaningful: an entry can be wrong in four distinct + ways and each has now been watched to fail. + """ + unsupported = unsupported_matched_claims(gates, repo_files=set(), repo_hooks=set(), repo_jobs=set()) + assert len(unsupported) == 1, f"{label}: expected exactly one complaint, got {unsupported}" + assert expected_fragment in unsupported[0], f"{label}: complaint did not explain itself: {unsupported[0]}" + + +def test_a_supported_matched_claim_produces_no_complaint(): + """And a truthful manifest must stay silent, or the check would be unusable.""" + gates = { + "file:README.md": {"status": "matched"}, + "hook:ruff": {"status": "matched"}, + "a.yml:job": {"status": "matched", "equivalent": "b.yml:other"}, + "c.yml:skipped": {"status": "excluded", "reason": "not applicable, and long enough to be substantive"}, + } + assert not unsupported_matched_claims( + gates, repo_files={"README.md"}, repo_hooks={"ruff"}, repo_jobs={"b.yml:other"} + ) def _ruleset(enforcement="active", contexts=("CI passed",), rule_type="required_status_checks"): diff --git a/tests/test_repo_docs.py b/tests/test_repo_docs.py index 7e7bf94..e0f0594 100644 --- a/tests/test_repo_docs.py +++ b/tests/test_repo_docs.py @@ -129,7 +129,7 @@ def test_repo_docs_build_ships_content_not_an_empty_site(tmp_path): one output directory. Only the inputs a build needs are copied, so the copy is cheap and a stray file in the repo cannot influence the result. """ - if shutil.which("uv") is None: + if shutil.which("uv") is None: # pragma: no cover - uv is present wherever this suite runs pytest.skip("uv is required to build the docs") engine = _publishing_engine() diff --git a/tests/test_repo_workflows.py b/tests/test_repo_workflows.py index 9e35888..8de553e 100644 --- a/tests/test_repo_workflows.py +++ b/tests/test_repo_workflows.py @@ -176,6 +176,19 @@ def test_the_agent_instruction_files_are_tracked(): assert ignored.returncode != 0, f"{rel} is gitignored, so it can never be reviewed or tested" +def instruction_body_difference(first, second): + """Lines present in one instruction body and not the other, ignoring blanks. + + Extracted so the disagreement path can be exercised directly. It is the branch + that only runs when the two files have already drifted, which is exactly when it + needs to be right, and exactly when nobody has ever seen it run. + """ + return ( + [ln for ln in first if ln and ln not in second], + [ln for ln in second if ln and ln not in first], + ) + + def test_the_agent_instruction_files_agree_on_shared_content(): """The two files carry the same guidance below their titles. @@ -190,14 +203,12 @@ def test_the_agent_instruction_files_agree_on_shared_content(): bodies.append((rel, [ln.rstrip() for ln in body])) (first_rel, first), (second_rel, second) = bodies - if first != second: - only_first = [ln for ln in first if ln and ln not in second] - only_second = [ln for ln in second if ln and ln not in first] - raise AssertionError( - f"{first_rel} and {second_rel} disagree.\n" - f" only in {first_rel}: {only_first[:3]}\n" - f" only in {second_rel}: {only_second[:3]}" - ) + only_first, only_second = instruction_body_difference(first, second) + assert not (only_first or only_second), ( + f"{first_rel} and {second_rel} disagree.\n" + f" only in {first_rel}: {only_first[:3]}\n" + f" only in {second_rel}: {only_second[:3]}" + ) def test_instruction_files_name_no_path_that_does_not_exist(): @@ -212,7 +223,7 @@ def test_instruction_files_name_no_path_that_does_not_exist(): for rel in _INSTRUCTION_FILES: text = (_REPO / rel).read_text(encoding="utf-8") for cited in sorted(set(path_like.findall(text))): - if any(ch in cited for ch in "<>{}"): + if any(ch in cited for ch in "<>{}"): # pragma: no cover - no current citation is templated continue # a placeholder, not a literal path # Tracked, not merely present. Checking existence made this test weaker on # a maintainer's machine than in CI: `openspec/` is gitignored, so a cited @@ -229,3 +240,15 @@ def test_instruction_files_name_no_path_that_does_not_exist(): f"{rel} cites `{cited}`, which git does not track. Whoever clones this repo will not find it, " "even if it exists on the machine the guidance was written on." ) + + +def test_instruction_body_difference_reports_each_side(): + """Both directions are reported, so a one-sided read cannot hide a missing section. + + The section instructing that both files be kept in step existed in only one of + them. A comparison that reported only additions would have called that clean. + """ + only_first, only_second = instruction_body_difference(["shared", "", "extra-a"], ["shared", "extra-b"]) + assert only_first == ["extra-a"] + assert only_second == ["extra-b"] + assert instruction_body_difference(["same", ""], ["same"]) == ([], []) From 9c79314a54b851ff7c266faa940d432f7696e5c6 Mon Sep 17 00:00:00 2001 From: Guillaume Tauzin <4648633+gtauzin@users.noreply.github.com> Date: Sun, 26 Jul 2026 16:00:30 +0200 Subject: [PATCH 5/5] chore(ci): make Codecov report coverage instead of grading it Codecov commented ":x: Patch coverage is 87% ... Please review" on a pull request whose eighteen checks were green and which nothing could stop merging. Two facts make that verdict meaningless, and neither is fixed by tightening a threshold. Coverage measures the wrong thing here, unavoidably: source is ["tests"] because this repo has no src/, so it answers "did this test file execute", which a passing test answers by definition. The product is template/ -- Jinja sources coverage cannot instrument, run only inside generated projects in temp dirs. A generated project measures its own package and the number means something there. And nothing enforces it, deliberately: the coverage job sits outside the ci-passed roll-up so a Codecov outage, the thing the OIDC upload is a canary for, cannot block merges. That decision stays. A red signal that can never mean anything trains readers to ignore the source, including on the day it matters. Both statuses are turned off rather than set informational: informational only stops the status blocking, while Codecov still computes the target and the comment still reads "Please review". The grade is what is meaningless, so the grade is what goes. The comment stays for the per-line annotations, which is how the unrun failure branches in the parity tests were found. Codecov reads repository config from the default branch, so this takes effect on merge, not on this pull request. --- codecov.yml | 41 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100644 codecov.yml diff --git a/codecov.yml b/codecov.yml new file mode 100644 index 0000000..0cc4e69 --- /dev/null +++ b/codecov.yml @@ -0,0 +1,41 @@ +# Codecov reports coverage for this repository; it does not judge it. +# +# Two facts make a pass/fail verdict meaningless here, and neither is a defect to be +# fixed by tightening a threshold: +# +# 1. Coverage measures the wrong thing, unavoidably. `[tool.coverage.run] source` is +# ["tests"], because this repository has no `src/`. That answers "did this test file +# execute", which a passing test answers by definition. The product is `template/`: +# Jinja sources that coverage cannot instrument, executed only inside generated +# projects in temporary directories during the integration tests. A generated +# project measures `source = [""]` and the number means something +# there. Here it does not. +# +# 2. Nothing enforces it, by design. The coverage job is deliberately excluded from the +# `ci-passed` roll-up so that a Codecov outage -- the very thing the OIDC upload is a +# canary for -- cannot block merges. That decision is correct and stays. +# +# Left at defaults, Codecov combined those two into a comment reading ":x: Patch +# coverage is 87% ... Please review" on a pull request whose checks were entirely +# green and which nothing could stop merging. A red signal that can never mean +# anything trains readers to ignore the source, including on the day it matters. +# +# Both statuses are off rather than `informational`. `informational` only stops the +# status blocking; Codecov still computes the target comparison and the pull request +# comment still renders ":x: ... Please review". Since the grade is what is meaningless +# here, the grade is what goes. The comment stays for the per-line annotations, which +# is how the unrun failure branches in the parity tests were found. +# +# NOTE: Codecov reads repository configuration from the DEFAULT BRANCH, so this file +# has no effect until it is merged. The comment on the pull request that introduced it +# will still show the old framing. +coverage: + status: + project: false + patch: false + +# The comment is worth keeping: its uncovered-line table is how the unrun failure +# branches in the parity tests were found. Only the pass/fail framing goes. +comment: + layout: "condensed_header, files, condensed_footer" + require_changes: true