From c525b3cadaf1a3148e09aa0bdfadcad129d1eedf Mon Sep 17 00:00:00 2001 From: Alex Reznichenko Date: Fri, 16 Oct 2026 21:53:35 +0200 Subject: [PATCH 1/6] feat: make agent-rules adoption explicit Add required/exempt repository metadata, fail-closed adoption validation, explicit missing-rule findings, dashboard status labels, and regression tests. Signed-off-by: Alex Reznichenko --- WATCHDOG.md | 10 ++- rollout/repositories.yaml | 29 ++++--- tests/test_check_agent_adoption.py | 73 ++++++++++++++++ tests/test_org_scan.py | 7 +- tests/test_watchdog.py | 16 +++- tests/test_watchdog_issue_sync.py | 7 ++ tools/check_agent_adoption.py | 134 +++++++++++++++++++++++++++++ tools/watchdog.py | 29 ++++++- tools/watchdog_issue_sync.py | 20 ++++- tools/watchdog_scan_support.py | 4 +- 10 files changed, 301 insertions(+), 28 deletions(-) create mode 100644 tests/test_check_agent_adoption.py create mode 100644 tools/check_agent_adoption.py mode change 100755 => 100644 tools/watchdog.py mode change 100755 => 100644 tools/watchdog_issue_sync.py diff --git a/WATCHDOG.md b/WATCHDOG.md index 36bc635..19726ea 100644 --- a/WATCHDOG.md +++ b/WATCHDOG.md @@ -34,7 +34,7 @@ only read files and never change them. |---|---|---|---|---|---| | Decisions of record (`tools/check_decisions.py`) | Text files covered by each entry in [decisions.yaml](decisions.yaml) | Builders and reviewers must see the approved value, not an old one | `Mismatch with approved decision: COMPUTE (14 finding(s))`, then `README.md:44: found 'Raspberry Pi 5'` | Correct the line, or label it as history with an accepted word (table below), or propose a decision change | Each entry's `owner` role, usually the platform lead | | Public extract (`tools/check_public_extract.py`) | `docs/`, `assets/`, every `README.md` and any path containing `public` | Public pages must not leak internal links, personal contact data, prices or secrets | `Should not be public: google-drive-link (1 finding(s))`, then `docs/a.md:3: found 'https://drive.google.com/...'` | Remove the value or move it to an internal place; approved public contacts are listed below, and new ones go on the allowlist ([public-extract-allowlist.yaml](public-extract-allowlist.yaml)) after review | Docs owner | -| Shared agent rules (`tools/check_agent_rules.py`) | `AGENTS.md` and `CLAUDE.md` | Every contributor and agent follows the same rules; a changed copy quietly changes them | `Shared agent rules out of date: shared-rules (1 finding(s))`, then `AGENTS.md:1: found 'shared block differs from canonical'` | Copy the block between the BEGIN and END markers from [agent-rules/SHARED_RULES.md](agent-rules/SHARED_RULES.md) unchanged | Software lead | +| Shared agent rules (`tools/check_agent_rules.py` + `tools/check_agent_adoption.py`) | `rollout/repositories.yaml`, `AGENTS.md` and `CLAUDE.md` | Every listed repository is either required to carry the shared rules or explicitly exempted with an owner-approved reason | `Agent rules missing: agent-rules (1 finding(s))`, then `AGENTS.md:1: found 'file is missing'` | Add the canonical AGENTS.md, or record a reviewed `agent_rules: exempt` entry with a reason | Software lead | | Workflow policy (`tools/check_workflow_policy.py`) | `.github/workflows/*.yml` | A tag or branch can be moved to new code; a full commit SHA cannot | `Workflow not pinned: unpinned-action (1 finding(s))`, then `.github/workflows/ci.yml:12: found 'actions/checkout@v4'` | Pin the action to its 40-character commit SHA and keep the version as a comment | CI owner | | PR evidence and AI disclosure (`tools/check_pr_evidence.py`) | The pull request description | Reviewers need the base and head commits, exact commands, test counts, a Not verified section and an honest AI disclosure | `Missing section: Not verified` | Fill every section of the pull request template | CI owner | | verify.sh (`rollout/verify.sh`) | The repository's build and tests | A test run that runs zero tests proves nothing; a skipped test must say why | `FAIL: zero tests executed` or `FAIL: skip/xfail/importorskip without a tracking issue` | Add real tests; put the tracking issue on the same line as each skip | CI owner | @@ -190,7 +190,7 @@ python3 -m pip install --user PyYAML # once, if PyYAML is missing python3 tools/watchdog.py --root ../openamrobot-docs ``` -It runs decisions of record, public extract, shared agent rules (when the repository has an +It runs decisions of record, public extract, shared agent rules according to `rollout/repositories.yaml`, workflow policy and decision freshness. A missing AGENTS.md is a finding unless the entry is explicitly exempt. `AGENTS.md`), workflow policy and decision freshness, prints each finding with its fix, then a summary table. Exit status: 0 clean, 1 findings, 2 a configuration or usage error. Useful options: @@ -247,7 +247,7 @@ The weekly loop is deliberately split at the ground-truth boundary: guesses a new value and never edits the register itself. 6. The next run verifies the fixing PR's result. Human owners close issues only after review. -The dashboard is the durable weekly record. Its `Shared rules` column distinguishes `pass`, +The dashboard is the durable weekly record. Its `Shared rules` column distinguishes `required/pass`, `required/drift`, `required/missing`, `exempt` and legacy `not enrolled` status. An exemption is an explicit governance decision, not a clean adoption result. `drift` and `not enrolled`; `not enrolled` is rollout status, not a contradiction. The uploaded JSON artifacts preserve the exact repository SHAs and evidence used by the run. @@ -291,6 +291,10 @@ back, delete the variable or set it to `dashboard`; issues that already exist st first repositories have finished rollout and their owners are ready to work from one issue per group. Until then the dashboard alone keeps the organization view in one place. +## Agent-rules adoption policy + +`rollout/repositories.yaml` is the authoritative enrollment manifest. Every entry declares `agent_rules: required` or `agent_rules: exempt`. An exempt entry must include `agent_rules_reason`; the Watchdog reports the exemption explicitly. It never treats a missing AGENTS.md as clean merely because the file is absent. + ## Adopting it in a repository The full steps are in [rollout/README.md](rollout/README.md). In short, one pull request per diff --git a/rollout/repositories.yaml b/rollout/repositories.yaml index 50b6cdd..cd392a1 100644 --- a/rollout/repositories.yaml +++ b/rollout/repositories.yaml @@ -1,19 +1,20 @@ # Active public openAMRobot repositories, the single list the Watchdog organization scan # (.github/workflows/watchdog-org-scan.yml) clones and checks on their default branch. # Add or remove a repository here only; tests/test_org_scan.py validates this file. +# agent_rules is required or exempt; exempt entries must include an owner-approved reason. organization: openAMRobot repositories: - - {name: .github, default_branch: main} - - {name: openamr-platform-sw, default_branch: main} - - {name: openamr-platform-fw, default_branch: main} - - {name: openamr-platform-hw, default_branch: main} - - {name: openamr-upperbody-sw, default_branch: main} - - {name: openamr-upperbody-fw, default_branch: main} - - {name: openamr-upperbody-hw, default_branch: main} - - {name: openamrobot-interfaces, default_branch: main} - - {name: openamrobot-manipulation, default_branch: main} - - {name: openamrobot-ui, default_branch: main} - - {name: openamrobot-comm, default_branch: main} - - {name: openamrobot-docs, default_branch: main} - - {name: openamrobot-manifest, default_branch: main} - - {name: openamrobot-release, default_branch: main} + - {name: .github, default_branch: main, agent_rules: required} + - {name: openamr-platform-sw, default_branch: main, agent_rules: required} + - {name: openamr-platform-fw, default_branch: main, agent_rules: required} + - {name: openamr-platform-hw, default_branch: main, agent_rules: required} + - {name: openamr-upperbody-sw, default_branch: main, agent_rules: required} + - {name: openamr-upperbody-fw, default_branch: main, agent_rules: required} + - {name: openamr-upperbody-hw, default_branch: main, agent_rules: required} + - {name: openamrobot-interfaces, default_branch: main, agent_rules: required} + - {name: openamrobot-manipulation, default_branch: main, agent_rules: required} + - {name: openamrobot-ui, default_branch: main, agent_rules: required} + - {name: openamrobot-comm, default_branch: main, agent_rules: required} + - {name: openamrobot-docs, default_branch: main, agent_rules: required} + - {name: openamrobot-manifest, default_branch: main, agent_rules: required} + - {name: openamrobot-release, default_branch: main, agent_rules: required} diff --git a/tests/test_check_agent_adoption.py b/tests/test_check_agent_adoption.py new file mode 100644 index 0000000..2ccca3b --- /dev/null +++ b/tests/test_check_agent_adoption.py @@ -0,0 +1,73 @@ +"""Tests for rollout/repositories.yaml agent-rules adoption metadata.""" + +import sys +import tempfile +import unittest +from pathlib import Path + +import yaml + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT / "tools")) +import check_agent_adoption as aa # noqa: E402 + + +class Adoption(unittest.TestCase): + def write(self, data): + path = Path(self.tmp.name, "repositories.yaml") + path.write_text(yaml.safe_dump(data), encoding="utf-8") + return path + + def setUp(self): + self.tmp = tempfile.TemporaryDirectory() + self.addCleanup(self.tmp.cleanup) + + def test_required_missing_agents_is_a_finding(self): + data = {"organization": "openAMRobot", "repositories": [ + {"name": "demo", "default_branch": "main", "agent_rules": "required"}, + ]} + policy = aa.entry_for(self.write(data), "demo") + root = Path(self.tmp.name, "repo") + root.mkdir() + findings = aa.missing_findings(root, policy) + self.assertEqual(len(findings), 1) + self.assertEqual(findings[0]["group"], "agent-rules") + self.assertEqual(findings[0]["found"], "file is missing") + + def test_exempt_missing_agents_is_not_a_finding(self): + data = {"organization": "openAMRobot", "repositories": [ + {"name": "release", "default_branch": "main", "agent_rules": "exempt", + "agent_rules_reason": "release process owns this boundary"}, + ]} + policy = aa.entry_for(self.write(data), "release") + root = Path(self.tmp.name, "repo") + root.mkdir() + self.assertEqual(aa.missing_findings(root, policy), []) + self.assertEqual(aa.policy_status(root, policy), { + "status": "exempt", "reason": "release process owns this boundary", + }) + + def test_exempt_requires_reason(self): + data = {"organization": "openAMRobot", "repositories": [ + {"name": "release", "default_branch": "main", "agent_rules": "exempt"}, + ]} + with self.assertRaisesRegex(ValueError, "agent_rules_reason"): + aa.validate_document(data) + + def test_unknown_status_is_rejected(self): + data = {"organization": "openAMRobot", "repositories": [ + {"name": "demo", "default_branch": "main", "agent_rules": "optional"}, + ]} + with self.assertRaisesRegex(ValueError, "required or exempt"): + aa.validate_document(data) + + def test_missing_status_is_rejected(self): + data = {"organization": "openAMRobot", "repositories": [ + {"name": "demo", "default_branch": "main"}, + ]} + with self.assertRaisesRegex(ValueError, "required or exempt"): + aa.validate_document(data) + + +if __name__ == "__main__": + unittest.main() diff --git a/tests/test_org_scan.py b/tests/test_org_scan.py index 77ff3d2..a21a17a 100644 --- a/tests/test_org_scan.py +++ b/tests/test_org_scan.py @@ -69,9 +69,14 @@ def test_lists_all_active_repositories_once(self): self.assertEqual(set(names), ACTIVE) for r in data["repositories"]: with self.subTest(repository=r["name"]): - self.assertEqual(set(r), {"name", "default_branch"}) + self.assertEqual(set(r), {"name", "default_branch", "agent_rules"}) self.assertRegex(r["name"], r"^[A-Za-z0-9._-]+$") self.assertRegex(r["default_branch"], r"^[A-Za-z0-9._/-]+$") + self.assertIn(r["agent_rules"], {"required", "exempt"}) + if r["agent_rules"] == "exempt": + self.assertTrue(r.get("agent_rules_reason", "").strip()) + else: + self.assertNotIn("agent_rules_reason", r) class Workflow(unittest.TestCase): diff --git a/tests/test_watchdog.py b/tests/test_watchdog.py index c320b6c..f6bedd3 100644 --- a/tests/test_watchdog.py +++ b/tests/test_watchdog.py @@ -80,6 +80,16 @@ def test_clean_repository_and_missing_agents(self): self.assertIn("Total: 0 finding(s)", out) self.assertIn("Next step: nothing to fix.", out) + def test_required_repository_without_agents_is_a_finding(self): + with tempfile.TemporaryDirectory() as tmp: + repo = Path(tmp, "openamr-platform-sw") + repo.mkdir() + (repo / "README.md").write_text("Reference compute is the Jetson.\n", encoding="utf-8") + report = watchdog.run(repo, "openamr-platform-sw") + self.assertEqual(report["agent_rules"], {"status": "required", "present": False}) + self.assertEqual(report["results"]["shared-rules"][0]["group"], "agent-rules") + self.assertEqual(report["results"]["shared-rules"][0]["found"], "file is missing") + def test_this_repository_passes_its_own_agents_check(self): report = watchdog.run(ROOT, ".github") self.assertEqual(report["results"]["shared-rules"], []) @@ -104,9 +114,9 @@ def test_json_and_markdown_outputs(self): run("--root", repo, "--report-only", "--markdown", md, "--json", js, "--repository", "x") run("--root", repo, "--report-only", "--markdown", md, "--repository", "y") text = md.read_text(encoding="utf-8") - self.assertIn("### x: 3 finding(s)", text) - self.assertIn("### y: 3 finding(s)", text) - self.assertIn("| Shared agent rules | not run (no AGENTS.md) |", text) + self.assertIn("### x: 4 finding(s)", text) + self.assertIn("### y: 4 finding(s)", text) + self.assertIn("| Shared agent rules | 1 |", text) self.assertIn("| COMPUTE | 1 |", text) data = json.loads(js.read_text(encoding="utf-8")) self.assertEqual(data["repository"], "x") diff --git a/tests/test_watchdog_issue_sync.py b/tests/test_watchdog_issue_sync.py index 865e48d..f307c24 100644 --- a/tests/test_watchdog_issue_sync.py +++ b/tests/test_watchdog_issue_sync.py @@ -93,6 +93,13 @@ def test_dashboard_observation_is_stable_marker(self): self.assertIsNotNone(wis.OBSERVATION_RE.search(body)) self.assertIn("Shared rules", body) + def test_dashboard_labels_exempt_agent_rules_explicitly(self): + data = report() + data["agent_rules"] = {"status": "exempt", "reason": "release process owns this boundary"} + plan = wis.plan([data], [], {"COMPUTE": {"owner": "platform-lead"}}, MAINTAINERS, [], "run", "2026-10-08") + body = wis.dashboard_body([data], [], plan, "run", "2026-10-08", []) + self.assertIn("| openamrobot-docs |", body) + self.assertIn("| exempt |", body) class Api(unittest.TestCase): def test_apply_reopens_and_comments_without_closing(self): diff --git a/tools/check_agent_adoption.py b/tools/check_agent_adoption.py new file mode 100644 index 0000000..c52dcd2 --- /dev/null +++ b/tools/check_agent_adoption.py @@ -0,0 +1,134 @@ +#!/usr/bin/env python3 +"""Validate repository-level agent-rules adoption metadata and requirements.""" + +import argparse +import re +import sys +from pathlib import Path + +import yaml + +TOOLS = Path(__file__).resolve().parent +sys.path.insert(0, str(TOOLS)) +import watchdog_report as wr # noqa: E402 + +NAME = re.compile(r"^[A-Za-z0-9._-]+$") +BRANCH = re.compile(r"^[A-Za-z0-9._/-]+$") +VALID_STATUSES = {"required", "exempt"} + +LABEL = "Agent rules missing" +GROUP = "agent-rules" +WHY = ("The organization scan must distinguish a repository that has not adopted the shared " + "rules from one that was explicitly exempted.") +NEXT_STEP = ("Add AGENTS.md with the canonical shared block and a repository-specific section, " + "or record an owner-approved agent_rules: exempt entry with a non-empty reason.") + + +class RepositoryNotListed(LookupError): + """The requested repository is not present in the active repository manifest.""" + + +def validate_document(data): + """Validate and return repository entries from rollout/repositories.yaml.""" + if not isinstance(data, dict) or not data.get("organization"): + raise ValueError("repositories.yaml needs an organization") + entries = data.get("repositories") + if not isinstance(entries, list) or not entries: + raise ValueError("repositories.yaml needs a non-empty repositories list") + + seen = set() + allowed_keys = {"name", "default_branch", "agent_rules", "agent_rules_reason"} + for number, entry in enumerate(entries, 1): + if not isinstance(entry, dict): + raise ValueError(f"repository entry {number} is not a mapping") + unknown = set(entry) - allowed_keys + if unknown: + raise ValueError(f"repository entry {number} has unknown field(s): {', '.join(sorted(unknown))}") + name = entry.get("name") + branch = entry.get("default_branch") + if not isinstance(name, str) or not NAME.fullmatch(name): + raise ValueError(f"repository entry {number} has an invalid name") + if name in seen: + raise ValueError(f"duplicate repository: {name}") + seen.add(name) + if not isinstance(branch, str) or not BRANCH.fullmatch(branch): + raise ValueError(f"{name}: invalid default_branch") + + status = entry.get("agent_rules") + if status not in VALID_STATUSES: + raise ValueError(f"{name}: agent_rules must be required or exempt") + reason = entry.get("agent_rules_reason") + if status == "exempt": + if not isinstance(reason, str) or not reason.strip(): + raise ValueError(f"{name}: agent_rules_reason is required for exempt repositories") + elif reason is not None: + raise ValueError(f"{name}: agent_rules_reason is only valid with agent_rules: exempt") + return entries + + +def load_document(path): + try: + data = yaml.safe_load(Path(path).read_text(encoding="utf-8")) + except (OSError, yaml.YAMLError) as exc: + raise ValueError(f"cannot read repository policy: {exc}") from exc + return validate_document(data) + + +def entry_for(path, repository): + for entry in load_document(path): + if entry["name"] == repository: + return dict(entry) + raise RepositoryNotListed(f"{repository}: not listed in repositories.yaml") + + +def policy_status(root, policy): + """Return JSON-safe policy state for the Watchdog report.""" + status = policy["agent_rules"] + result = {"status": status} + if status == "exempt": + result["reason"] = policy["agent_rules_reason"].strip() + else: + result["present"] = Path(root, "AGENTS.md").is_file() + return result + + +def missing_findings(root, policy): + """Return a finding only when required AGENTS.md is absent.""" + if policy["agent_rules"] == "exempt" or Path(root, "AGENTS.md").is_file(): + return [] + return [wr.finding( + LABEL, + GROUP, + "AGENTS.md", + 1, + "file is missing", + "AGENTS.md is required for this repository.", + WHY, + "Add AGENTS.md using the canonical shared block, then keep repository-specific rules below it.", + [("WATCHDOG.md", f"{wr.DOCS}#shared-agent-rules")], + )] + + +def main(argv=None): + parser = argparse.ArgumentParser(description=__doc__.splitlines()[0]) + parser.add_argument("--repositories", required=True, type=Path) + parser.add_argument("--repository", required=True) + parser.add_argument("--root", required=True, type=Path) + args = parser.parse_args(argv) + try: + policy = entry_for(args.repositories, args.repository) + findings = missing_findings(args.root, policy) + except (OSError, ValueError, RepositoryNotListed) as exc: + print(f"INVALID configuration: {exc}", file=sys.stderr) + return 2 + if findings: + for line in wr.render(findings, "Agent rules adoption", NEXT_STEP, decision_word="Rule"): + print(line) + wr.emit_github(findings, "Agent rules adoption", NEXT_STEP) + return 1 + print(f"PASS {args.repository}: agent_rules={policy['agent_rules']}") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/tools/watchdog.py b/tools/watchdog.py old mode 100755 new mode 100644 index a609dd8..ef587b1 --- a/tools/watchdog.py +++ b/tools/watchdog.py @@ -26,6 +26,7 @@ TOOLS = Path(__file__).resolve().parent sys.path.insert(0, str(TOOLS)) +import check_agent_adoption as aa # noqa: E402 import check_agent_rules as car # noqa: E402 import check_decisions as cd # noqa: E402 import check_public_extract as pe # noqa: E402 @@ -40,11 +41,17 @@ "shared-rules": car.NEXT_STEP, "workflow-policy": wp.NEXT_STEP} -def shared_rules(root, canonical): - """Findings for AGENTS.md drift; None when the repository has no AGENTS.md.""" +def shared_rules(root, canonical, policy=None): + """Findings for adoption and AGENTS.md drift. + + Unknown repositories keep the historical None result for direct local checks. Listed + repositories must declare whether rules are required or explicitly exempt. + """ path = Path(root, "AGENTS.md") if not path.is_file(): - return None + if policy and policy["agent_rules"] == "required": + return aa.missing_findings(root, policy) + return None if policy is None else [] version, expected = car.block(canonical) try: text = path.read_text(encoding="utf-8") @@ -79,15 +86,27 @@ def run(root, repository, harness=HARNESS, today=None): allow = pe.load_allowlist(Path(harness, "public-extract-allowlist.yaml")) except (OSError, ValueError, yaml.YAMLError) as exc: raise ValueError(f"invalid public extract allowlist: {exc}") from exc + policy = None + policy_path = Path(harness, "rollout", "repositories.yaml") + if policy_path.is_file(): + try: + policy = aa.entry_for(policy_path, repository) + except aa.RepositoryNotListed: + # Direct local checks may target a temporary or external checkout. The organization + # scan supplies only repositories listed in rollout/repositories.yaml. + policy = None + except (OSError, ValueError, yaml.YAMLError) as exc: + raise ValueError(f"invalid agent-rules adoption policy: {exc}") from exc stats = {} found, allowed = cd.scan(root, decisions, repository, None, stats) results = { "decisions": cd.to_watchdog(found, decisions), "public-extract": pe.to_watchdog(pe.scan(root, allow, repository)), - "shared-rules": shared_rules(root, Path(harness, "agent-rules", "SHARED_RULES.md")), + "shared-rules": shared_rules(root, Path(harness, "agent-rules", "SHARED_RULES.md"), policy), "workflow-policy": wp.to_watchdog(wp.records(root)), } return {"repository": repository, "results": results, "allowed": allowed, + "agent_rules": aa.policy_status(root, policy) if policy else None, "freshness": freshness(register, today), "unscanned": stats.get("unscanned", 0)} @@ -178,6 +197,8 @@ def render(report): if items is None: lines += ["Not run: the repository has no AGENTS.md.", ""] continue + if check == "shared-rules" and (report.get("agent_rules") or {}).get("status") == "exempt": + lines += [f"Policy: agent rules exempt ({report['agent_rules']['reason']}).", ""] if check == "decisions": for a in report["allowed"]: lines.append(f"Allowed by decision-allow marker: {a['file']}:{a['line']}: {a['id']} " diff --git a/tools/watchdog_issue_sync.py b/tools/watchdog_issue_sync.py old mode 100755 new mode 100644 index ae91cdf..ecf85f0 --- a/tools/watchdog_issue_sync.py +++ b/tools/watchdog_issue_sync.py @@ -430,6 +430,23 @@ def group_rows(active): return rows + +def agent_rules_label(report): + """Render adoption state without confusing an exemption with a clean required repo.""" + policy = report.get("agent_rules") + results = report.get("results") or {} + shared = results.get("shared-rules") + if not isinstance(policy, dict) or not policy.get("status"): + return "not enrolled" if shared is None else ("drift" if shared else "pass") + if policy["status"] == "exempt": + return "exempt" + if shared is None: + return "required/not checked" + if any(item.get("group") == "agent-rules" for item in shared): + return "required/missing" + return "required/drift" if shared else "required/pass" + + def dashboard_body(reports, blocked, plan_data, run_url, run_date, links): mode = plan_data.get("mode", DEFAULT_MODE) status = plan_data.get("status") @@ -443,8 +460,7 @@ def dashboard_body(reports, blocked, plan_data, run_url, run_date, links): for report in reports: findings = sum(len(value or []) for value in (report.get("results") or {}).values()) warnings = len(report.get("freshness") or []) - shared = report.get("results", {}).get("shared-rules") - adoption = "not enrolled" if shared is None else ("drift" if shared else "pass") + adoption = agent_rules_label(report) status = "findings" if findings or warnings else "clean" lines.append(f"| {scrub(report.get('repository'))} | `{scrub(report.get('commit', 'unknown'))}` | {findings} | {warnings} | {adoption} | {status} |") for item in blocked: diff --git a/tools/watchdog_scan_support.py b/tools/watchdog_scan_support.py index b94b719..f7b0acd 100644 --- a/tools/watchdog_scan_support.py +++ b/tools/watchdog_scan_support.py @@ -19,6 +19,8 @@ import yaml +import check_agent_adoption as aa + NAME = re.compile(r"^[A-Za-z0-9._-]+$") BRANCH = re.compile(r"^[A-Za-z0-9._/-]+$") @@ -27,7 +29,7 @@ def cmd_list(args): data = yaml.safe_load(Path(args.repositories).read_text(encoding="utf-8")) if not isinstance(data, dict) or not isinstance(data.get("repositories"), list) or not data.get("organization"): raise ValueError("repositories.yaml needs an organization and a repositories list") - entries = data["repositories"] + entries = aa.validate_document(data) expected = [] lines = [] for number, entry in enumerate(entries, 1): From 5e24e28704e202bf77ae445f6c5fa57e2e305684 Mon Sep 17 00:00:00 2001 From: Alex Reznichenko Date: Fri, 16 Oct 2026 21:55:32 +0200 Subject: [PATCH 2/6] test: exercise adoption policy for listed repositories --- tests/test_watchdog.py | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/tests/test_watchdog.py b/tests/test_watchdog.py index f6bedd3..514672c 100644 --- a/tests/test_watchdog.py +++ b/tests/test_watchdog.py @@ -111,15 +111,15 @@ def test_json_and_markdown_outputs(self): with tempfile.TemporaryDirectory() as tmp: repo = self.make(tmp, agents=False) md, js = Path(tmp, "summary.md"), Path(tmp, "r.json") - run("--root", repo, "--report-only", "--markdown", md, "--json", js, "--repository", "x") - run("--root", repo, "--report-only", "--markdown", md, "--repository", "y") + run("--root", repo, "--report-only", "--markdown", md, "--json", js, "--repository", "openamr-platform-sw") + run("--root", repo, "--report-only", "--markdown", md, "--repository", "openamr-platform-fw") text = md.read_text(encoding="utf-8") - self.assertIn("### x: 4 finding(s)", text) - self.assertIn("### y: 4 finding(s)", text) + self.assertIn("### openamr-platform-sw: 4 finding(s)", text) + self.assertIn("### openamr-platform-fw: 4 finding(s)", text) self.assertIn("| Shared agent rules | 1 |", text) self.assertIn("| COMPUTE | 1 |", text) data = json.loads(js.read_text(encoding="utf-8")) - self.assertEqual(data["repository"], "x") + self.assertEqual(data["repository"], "openamr-platform-sw") self.assertEqual(len(data["results"]["decisions"]), 1) finding = data["results"]["decisions"][0] # JSON keeps full detail per finding, unlike the compact text report. From aca7baa0d26243bf80df1dba9657fcdd591ed1e2 Mon Sep 17 00:00:00 2001 From: Alex Reznichenko Date: Fri, 16 Oct 2026 21:55:34 +0200 Subject: [PATCH 3/6] test: count missing agent rules in organization scan --- tests/test_org_scan.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/tests/test_org_scan.py b/tests/test_org_scan.py index a21a17a..977c58a 100644 --- a/tests/test_org_scan.py +++ b/tests/test_org_scan.py @@ -125,8 +125,8 @@ def test_scan_reports_every_repository_and_blocks_on_clone_failure(self): self.assertIn("BLOCKED repositories were not scanned and are not clean.", result.summary) rows = re.findall(r"^\| ([.\w-]+) \| abc1234 \| (\d+) \|$", result.summary, re.M) self.assertEqual(len(rows), 13) - self.assertIn(("openamr-platform-sw", "1"), rows) - self.assertIn("### openamr-platform-sw: 1 finding(s)", result.summary) + self.assertIn(("openamr-platform-sw", "2"), rows) + self.assertIn("### openamr-platform-sw: 2 finding(s)", result.summary) self.assertIn("| COMPUTE | 1 |", result.summary) self.assertEqual(result.blocked, [{"repository": "openamrobot-comm", "reason": "clone failed"}]) # Default dashboard mode: the summary must not promise per-group issues. From b9e2a803d374e5621fe7cbe9da788c3ed782997b Mon Sep 17 00:00:00 2001 From: Alex Reznichenko Date: Fri, 16 Oct 2026 21:56:55 +0200 Subject: [PATCH 4/6] fix: preserve precise incomplete-scan counts --- tools/watchdog_scan_support.py | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/tools/watchdog_scan_support.py b/tools/watchdog_scan_support.py index f7b0acd..916b294 100644 --- a/tools/watchdog_scan_support.py +++ b/tools/watchdog_scan_support.py @@ -29,7 +29,7 @@ def cmd_list(args): data = yaml.safe_load(Path(args.repositories).read_text(encoding="utf-8")) if not isinstance(data, dict) or not isinstance(data.get("repositories"), list) or not data.get("organization"): raise ValueError("repositories.yaml needs an organization and a repositories list") - entries = aa.validate_document(data) + entries = data["repositories"] expected = [] lines = [] for number, entry in enumerate(entries, 1): @@ -40,8 +40,12 @@ def cmd_list(args): lines.append(f"{data['organization']} {name} {branch}") # expected.json is written first and always reflects every entry, valid or not, so the # issue sync can report entries that never reached list.txt as BLOCKED. + # Write the structural inventory before validating policy metadata. This lets the + # workflow report precise usable/expected counts for an invalid entry while still + # stopping before any clone or Watchdog run. Path(args.expected).write_text(json.dumps(expected), encoding="utf-8") Path(args.list).write_text("".join(f"{line}\n" for line in lines), encoding="utf-8") + aa.validate_document(data) return 0 From 8fb9b89b73e63b5db38213e62a19ef8dd2b3d0bf Mon Sep 17 00:00:00 2001 From: Alex Reznichenko Date: Fri, 16 Oct 2026 21:56:56 +0200 Subject: [PATCH 5/6] fix: retain manifest failure counts in scan summary --- .github/workflows/watchdog-org-scan.yml | 10 ++++++++-- 1 file changed, 8 insertions(+), 2 deletions(-) diff --git a/.github/workflows/watchdog-org-scan.yml b/.github/workflows/watchdog-org-scan.yml index 522e032..cedcd24 100644 --- a/.github/workflows/watchdog-org-scan.yml +++ b/.github/workflows/watchdog-org-scan.yml @@ -79,12 +79,18 @@ jobs: # Repository list: a failure stops the step; expected.json is written before list.txt. if ! python3 "$support" list --repositories "$HARNESS/rollout/repositories.yaml" \ --list "$WORK/list.txt" --expected "$WORK/expected.json"; then + detail="" + if [ -f "$WORK/expected.json" ] && [ -f "$WORK/list.txt" ]; then + expected_count=$(python3 -c 'import json,sys; print(len(json.load(open(sys.argv[1]))))' "$WORK/expected.json") + listed_count=$(wc -l < "$WORK/list.txt") + detail=" ($listed_count of $expected_count entries usable)" + fi { echo "## OpenAMRobot Watchdog organization scan: INCOMPLETE" echo - echo "Scan INCOMPLETE: repository list could not be generated." + echo "Scan INCOMPLETE: repository list could not be generated$detail." } >> "$summary" - echo "::error::Scan INCOMPLETE: repository list could not be generated" + echo "::error::Scan INCOMPLETE: repository list could not be generated$detail" finished=1 exit 1 fi From 135ae6add0b19d103487cea2cefa39ff84cde2c7 Mon Sep 17 00:00:00 2001 From: Alex Reznichenko Date: Sat, 17 Oct 2026 00:57:32 +0200 Subject: [PATCH 6/6] docs: repair two merged WATCHDOG.md paragraphs Clarified the description of the Watchdog's functionality and its reporting structure, including exit statuses and dashboard distinctions. Signed-off-by: Alex Reznichenko --- WATCHDOG.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/WATCHDOG.md b/WATCHDOG.md index 19726ea..cbc905d 100644 --- a/WATCHDOG.md +++ b/WATCHDOG.md @@ -190,8 +190,9 @@ python3 -m pip install --user PyYAML # once, if PyYAML is missing python3 tools/watchdog.py --root ../openamrobot-docs ``` -It runs decisions of record, public extract, shared agent rules according to `rollout/repositories.yaml`, workflow policy and decision freshness. A missing AGENTS.md is a finding unless the entry is explicitly exempt. -`AGENTS.md`), workflow policy and decision freshness, prints each finding with its fix, then a +It runs decisions of record, public extract, shared agent rules (according to +`rollout/repositories.yaml`; a missing AGENTS.md is a finding unless the entry is exempt), +workflow policy and decision freshness, prints each finding with its fix, then a summary table. Exit status: 0 clean, 1 findings, 2 a configuration or usage error. Useful options: @@ -247,8 +248,10 @@ The weekly loop is deliberately split at the ground-truth boundary: guesses a new value and never edits the register itself. 6. The next run verifies the fixing PR's result. Human owners close issues only after review. -The dashboard is the durable weekly record. Its `Shared rules` column distinguishes `required/pass`, `required/drift`, `required/missing`, `exempt` and legacy `not enrolled` status. An exemption is an explicit governance decision, not a clean adoption result. -`drift` and `not enrolled`; `not enrolled` is rollout status, not a contradiction. The uploaded +The dashboard is the durable weekly record. Its `Shared rules` column distinguishes +`required/pass`, `required/drift`, `required/missing`, `exempt` and legacy `not enrolled`; +`not enrolled` is rollout status, not a contradiction, and an exemption is a governance +decision, not a clean adoption result. The uploaded JSON artifacts preserve the exact repository SHAs and evidence used by the run. ### Incomplete scans