feat(ci): make the template repo subject to the gates it ships - #252
Merged
Conversation
This repository generates seven packages and receives nothing back: every improvement flows out of template/, and nothing flows in. That is how "pin exact uv version in all setup-uv steps" pinned every step in the template and none of the six here, and how the ci-passed roll-up shipped in v0.32.0 while no ruleset required it on this repo for four releases. The gaps are absences, not drift, so a render-and-diff check cannot find them. This adds a parity manifest instead: every gate the template ships resolves to matched, excluded with a written reason, or leading, and the shipped-gate set is derived from a rendered project rather than listed by hand so a new gate arrives unanswered instead of not arriving. Closed here: uv pinned across all six setup-uv steps and asserted over this repo's own workflows; a nightly canary; the docs entry points and content gate moved onto the publishing engine; the skills mirror enforced byte-identically over tracked files; git-cliff migrated to commit.remote.*. Generated projects can now commit a CLAUDE.md: the template seeds one and registers it in _skip_if_exists, and the six-variant ignore block is gone. This repo's own CLAUDE.md is tracked and reconciled with its Copilot copy, correcting three claims that were not true of main. Every assertion was verified to fail before being accepted. Two found real defects: the manifest initially checked that gates were *answered* but never that a "matched" claim was *true*, so deleting this repo's nightly workflow failed nothing; and deriving gates from jobs and hooks alone was blind to controls that are only files, which is how CODEOWNERS and SECURITY.md reached seven projects and not this one.
The instruction-file check asserted a cited path exists. On a maintainer's machine `openspec/` exists, so a citation under it resolved; it is gitignored, so a fresh clone has no such path and CI failed on a citation that pointed at nothing for every reader but the author. The test was weaker locally than in CI, which is the same defect class it was written to catch. Asking git instead makes both environments agree and matches what the guidance actually promises a reader. Also drops the `openspec/specs/` citation itself, since that tree is local-only.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
…te repo These three were shipped to generated projects by harden-fleet-security and never to this repository. It is public, so all three are free, and the parity manifest recorded them as shortfalls rather than judgments. Adopting them turns three "excluded, unresolved" entries into verified matches. SECURITY.md is reworded for what this repo is: a fix reaches a generated project only when that project runs `copier update`, so there is no patched release to consume. Also pins the seed-once guarantee behaviourally. test_claude_md_is_registered_ as_seed_once only asserted the entry exists; the new test re-runs the template over a project that rewrote its CLAUDE.md and asserts the content survives. Removing CLAUDE.md from _skip_if_exists makes it fail with the file clobbered, so the guard is load-bearing rather than decorative. The test carries a control (an unguarded README.md that must be overwritten) so a run that clobbered nothing cannot pass as proof.
|
You are seeing this message because GitHub Code Scanning has recently been set up for this repository, or this pull request contains the workflow file for the Code Scanning tool. What Enabling Code Scanning Means:
For more information about GitHub Code Scanning, check out the documentation. |
Codecov flagged 32 unrun lines in the new tests. Most were the bodies of slow/integration tests, which the coverage job excludes by selection and cannot reach. The rest were the branches that only execute when a check fails, which is precisely when they need to be right and precisely when nobody has watched them run. Extracts two pieces of logic so those branches can be driven directly: unsupported_matched_claims (the four distinct ways a manifest entry can claim a match that reality does not support) and instruction_body_difference (which must report both directions, since the section telling contributors to keep both instruction files in step was missing from one of them, and a one-sided comparison would have called that clean). Genuinely environment-dependent branches -- gh absent, network unreachable, uv missing -- are marked no-cover with the reason rather than chased with contrived tests. test_repo_workflows.py reaches 100%.
Codecov commented ":x: Patch coverage is 87% ... Please review" on a pull request whose eighteen checks were green and which nothing could stop merging. Two facts make that verdict meaningless, and neither is fixed by tightening a threshold. Coverage measures the wrong thing here, unavoidably: source is ["tests"] because this repo has no src/, so it answers "did this test file execute", which a passing test answers by definition. The product is template/ -- Jinja sources coverage cannot instrument, run only inside generated projects in temp dirs. A generated project measures its own package and the number means something there. And nothing enforces it, deliberately: the coverage job sits outside the ci-passed roll-up so a Codecov outage, the thing the OIDC upload is a canary for, cannot block merges. That decision stays. A red signal that can never mean anything trains readers to ignore the source, including on the day it matters. Both statuses are turned off rather than set informational: informational only stops the status blocking, while Codecov still computes the target and the comment still reads "Please review". The grade is what is meaningless, so the grade is what goes. The comment stays for the per-line annotations, which is how the unrun failure branches in the parity tests were found. Codecov reads repository config from the default branch, so this takes effect on merge, not on this pull request.
gtauzin
force-pushed
the
feat/template-repo-parity
branch
from
July 26, 2026 14:13
4acd263 to
9c79314
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
This repository generates seven packages and receives nothing back. Every improvement flows out of
template/; nothing flows in. Two measured consequences:fix(ci): pin exact uv version in **all** setup-uv steps(fix(ci): pin exact uv version in all setup-uv steps #220) touched eleven files, none of them.github/workflows/. "All" silently meant "all in the template". Six steps here stayed unpinned, and the count grew from five while nobody was looking.ci-passedroll-up shipped in v0.32.0 as the check name meant to be the required one. No ruleset required it here for four releases. Every generated project already required it; only the factory did not.The gaps are absences, not drift, so a render-and-diff check cannot find them: you cannot diff a file that is not there.
What this adds
A parity manifest (
tests/parity_manifest.yml): every gate the template ships resolves tomatched,excludedwith a written reason, orleading. The shipped-gate set is derived from a rendered project, so a new template gate arrives as an unanswered entry rather than not arriving at all.Closed alongside it: uv pinned across all six
setup-uvsteps and asserted over this repo's own workflows; a nightly canary with issue filing; the docs entry points and content gate moved onto the publishing engine; the skills mirror enforced byte-identically over tracked files;.git-cliff.tomlmigrated tocommit.remote.*with byte-identical output.Generated projects can now commit a
CLAUDE.md: the template seeds one, registers it in_skip_if_exists, and the six-variant ignore block is gone. This repo's ownCLAUDE.mdis tracked and reconciled with its Copilot copy.Every assertion was verified to fail first
Two of those checks found real defects in this PR's own work:
matchedclaim was true. Deleting this repo's nightly workflow outright failed nothing. Now a matched file must exist, a matched hook must run here, and a matched job must name anequivalent:this repo actually runs.CODEOWNERSandSECURITY.mdreached seven projects and not this one. Derivation now covers the project root too: 37 gates became 56.Recorded as shortfalls, not justified
SECURITY.md,codeql.ymlandscorecard.ymlare absent here only becauseharden-fleet-securityshipped them to the template and never to this repo. This repo is public and all three are free. The manifest says so out loud rather than leaving it silently true.Known caveat
_skip_if_existsguardingCLAUDE.mdis verified to preserve a rewritten file through a realcopier update. The negative control did not reproduce a loss: removing the guard preserved the content anyway, for both a wholly-rewritten file and the v0.22.0 partial-divergence shape. The guard's necessity rests on that documented incident, not on anything reproduced here.Separately:
template/.copier-answers.yml.jinjahardcodes_src_path: gh:stateful-y/python-package-copierand an update rewrites the answers file, so a local override is wiped by the first update. That makes local template testing impossible after one update.