Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions .github/workflows/watchdog-org-scan.yml
Original file line number Diff line number Diff line change
Expand Up @@ -79,12 +79,18 @@ jobs:
# Repository list: a failure stops the step; expected.json is written before list.txt.
if ! python3 "$support" list --repositories "$HARNESS/rollout/repositories.yaml" \
--list "$WORK/list.txt" --expected "$WORK/expected.json"; then
detail=""
if [ -f "$WORK/expected.json" ] && [ -f "$WORK/list.txt" ]; then
expected_count=$(python3 -c 'import json,sys; print(len(json.load(open(sys.argv[1]))))' "$WORK/expected.json")
listed_count=$(wc -l < "$WORK/list.txt")
detail=" ($listed_count of $expected_count entries usable)"
fi
{
echo "## OpenAMRobot Watchdog organization scan: INCOMPLETE"
echo
echo "Scan INCOMPLETE: repository list could not be generated."
echo "Scan INCOMPLETE: repository list could not be generated$detail."
} >> "$summary"
echo "::error::Scan INCOMPLETE: repository list could not be generated"
echo "::error::Scan INCOMPLETE: repository list could not be generated$detail"
finished=1
exit 1
fi
Expand Down
17 changes: 12 additions & 5 deletions WATCHDOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ only read files and never change them.
|---|---|---|---|---|---|
| <a id="decisions-of-record"></a>Decisions of record (`tools/check_decisions.py`) | Text files covered by each entry in [decisions.yaml](decisions.yaml) | Builders and reviewers must see the approved value, not an old one | `Mismatch with approved decision: COMPUTE (14 finding(s))`, then `README.md:44: found 'Raspberry Pi 5'` | Correct the line, or label it as history with an accepted word (table below), or propose a decision change | Each entry's `owner` role, usually the platform lead |
| <a id="public-extract"></a>Public extract (`tools/check_public_extract.py`) | `docs/`, `assets/`, every `README.md` and any path containing `public` | Public pages must not leak internal links, personal contact data, prices or secrets | `Should not be public: google-drive-link (1 finding(s))`, then `docs/a.md:3: found 'https://drive.google.com/...'` | Remove the value or move it to an internal place; approved public contacts are listed below, and new ones go on the allowlist ([public-extract-allowlist.yaml](public-extract-allowlist.yaml)) after review | Docs owner |
| <a id="shared-agent-rules"></a>Shared agent rules (`tools/check_agent_rules.py`) | `AGENTS.md` and `CLAUDE.md` | Every contributor and agent follows the same rules; a changed copy quietly changes them | `Shared agent rules out of date: shared-rules (1 finding(s))`, then `AGENTS.md:1: found 'shared block differs from canonical'` | Copy the block between the BEGIN and END markers from [agent-rules/SHARED_RULES.md](agent-rules/SHARED_RULES.md) unchanged | Software lead |
| <a id="shared-agent-rules"></a>Shared agent rules (`tools/check_agent_rules.py` + `tools/check_agent_adoption.py`) | `rollout/repositories.yaml`, `AGENTS.md` and `CLAUDE.md` | Every listed repository is either required to carry the shared rules or explicitly exempted with an owner-approved reason | `Agent rules missing: agent-rules (1 finding(s))`, then `AGENTS.md:1: found 'file is missing'` | Add the canonical AGENTS.md, or record a reviewed `agent_rules: exempt` entry with a reason | Software lead |
| <a id="workflow-policy"></a>Workflow policy (`tools/check_workflow_policy.py`) | `.github/workflows/*.yml` | A tag or branch can be moved to new code; a full commit SHA cannot | `Workflow not pinned: unpinned-action (1 finding(s))`, then `.github/workflows/ci.yml:12: found 'actions/checkout@v4'` | Pin the action to its 40-character commit SHA and keep the version as a comment | CI owner |
| <a id="pr-evidence"></a>PR evidence and AI disclosure (`tools/check_pr_evidence.py`) | The pull request description | Reviewers need the base and head commits, exact commands, test counts, a Not verified section and an honest AI disclosure | `Missing section: Not verified` | Fill every section of the pull request template | CI owner |
| <a id="verify"></a>verify.sh (`rollout/verify.sh`) | The repository's build and tests | A test run that runs zero tests proves nothing; a skipped test must say why | `FAIL: zero tests executed` or `FAIL: skip/xfail/importorskip without a tracking issue` | Add real tests; put the tracking issue on the same line as each skip | CI owner |
Expand Down Expand Up @@ -190,8 +190,9 @@ python3 -m pip install --user PyYAML # once, if PyYAML is missing
python3 tools/watchdog.py --root ../openamrobot-docs
```

It runs decisions of record, public extract, shared agent rules (when the repository has an
`AGENTS.md`), workflow policy and decision freshness, prints each finding with its fix, then a
It runs decisions of record, public extract, shared agent rules (according to
`rollout/repositories.yaml`; a missing AGENTS.md is a finding unless the entry is exempt),
workflow policy and decision freshness, prints each finding with its fix, then a
summary table. Exit status: 0 clean, 1 findings, 2 a configuration or usage error. Useful
options:

Expand Down Expand Up @@ -247,8 +248,10 @@ The weekly loop is deliberately split at the ground-truth boundary:
guesses a new value and never edits the register itself.
6. The next run verifies the fixing PR's result. Human owners close issues only after review.

The dashboard is the durable weekly record. Its `Shared rules` column distinguishes `pass`,
`drift` and `not enrolled`; `not enrolled` is rollout status, not a contradiction. The uploaded
The dashboard is the durable weekly record. Its `Shared rules` column distinguishes
`required/pass`, `required/drift`, `required/missing`, `exempt` and legacy `not enrolled`;
`not enrolled` is rollout status, not a contradiction, and an exemption is a governance
decision, not a clean adoption result. The uploaded
JSON artifacts preserve the exact repository SHAs and evidence used by the run.

### Incomplete scans
Expand Down Expand Up @@ -291,6 +294,10 @@ back, delete the variable or set it to `dashboard`; issues that already exist st
first repositories have finished rollout and their owners are ready to work from one issue per
group. Until then the dashboard alone keeps the organization view in one place.

## Agent-rules adoption policy

`rollout/repositories.yaml` is the authoritative enrollment manifest. Every entry declares `agent_rules: required` or `agent_rules: exempt`. An exempt entry must include `agent_rules_reason`; the Watchdog reports the exemption explicitly. It never treats a missing AGENTS.md as clean merely because the file is absent.

## Adopting it in a repository

The full steps are in [rollout/README.md](rollout/README.md). In short, one pull request per
Expand Down
29 changes: 15 additions & 14 deletions rollout/repositories.yaml
Original file line number Diff line number Diff line change
@@ -1,19 +1,20 @@
# Active public openAMRobot repositories, the single list the Watchdog organization scan
# (.github/workflows/watchdog-org-scan.yml) clones and checks on their default branch.
# Add or remove a repository here only; tests/test_org_scan.py validates this file.
# agent_rules is required or exempt; exempt entries must include an owner-approved reason.
organization: openAMRobot
repositories:
- {name: .github, default_branch: main}
- {name: openamr-platform-sw, default_branch: main}
- {name: openamr-platform-fw, default_branch: main}
- {name: openamr-platform-hw, default_branch: main}
- {name: openamr-upperbody-sw, default_branch: main}
- {name: openamr-upperbody-fw, default_branch: main}
- {name: openamr-upperbody-hw, default_branch: main}
- {name: openamrobot-interfaces, default_branch: main}
- {name: openamrobot-manipulation, default_branch: main}
- {name: openamrobot-ui, default_branch: main}
- {name: openamrobot-comm, default_branch: main}
- {name: openamrobot-docs, default_branch: main}
- {name: openamrobot-manifest, default_branch: main}
- {name: openamrobot-release, default_branch: main}
- {name: .github, default_branch: main, agent_rules: required}
- {name: openamr-platform-sw, default_branch: main, agent_rules: required}
- {name: openamr-platform-fw, default_branch: main, agent_rules: required}
- {name: openamr-platform-hw, default_branch: main, agent_rules: required}
- {name: openamr-upperbody-sw, default_branch: main, agent_rules: required}
- {name: openamr-upperbody-fw, default_branch: main, agent_rules: required}
- {name: openamr-upperbody-hw, default_branch: main, agent_rules: required}
- {name: openamrobot-interfaces, default_branch: main, agent_rules: required}
- {name: openamrobot-manipulation, default_branch: main, agent_rules: required}
- {name: openamrobot-ui, default_branch: main, agent_rules: required}
- {name: openamrobot-comm, default_branch: main, agent_rules: required}
- {name: openamrobot-docs, default_branch: main, agent_rules: required}
- {name: openamrobot-manifest, default_branch: main, agent_rules: required}
- {name: openamrobot-release, default_branch: main, agent_rules: required}
73 changes: 73 additions & 0 deletions tests/test_check_agent_adoption.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
"""Tests for rollout/repositories.yaml agent-rules adoption metadata."""

import sys
import tempfile
import unittest
from pathlib import Path

import yaml

ROOT = Path(__file__).resolve().parents[1]
sys.path.insert(0, str(ROOT / "tools"))
import check_agent_adoption as aa # noqa: E402


class Adoption(unittest.TestCase):
def write(self, data):
path = Path(self.tmp.name, "repositories.yaml")
path.write_text(yaml.safe_dump(data), encoding="utf-8")
return path

def setUp(self):
self.tmp = tempfile.TemporaryDirectory()
self.addCleanup(self.tmp.cleanup)

def test_required_missing_agents_is_a_finding(self):
data = {"organization": "openAMRobot", "repositories": [
{"name": "demo", "default_branch": "main", "agent_rules": "required"},
]}
policy = aa.entry_for(self.write(data), "demo")
root = Path(self.tmp.name, "repo")
root.mkdir()
findings = aa.missing_findings(root, policy)
self.assertEqual(len(findings), 1)
self.assertEqual(findings[0]["group"], "agent-rules")
self.assertEqual(findings[0]["found"], "file is missing")

def test_exempt_missing_agents_is_not_a_finding(self):
data = {"organization": "openAMRobot", "repositories": [
{"name": "release", "default_branch": "main", "agent_rules": "exempt",
"agent_rules_reason": "release process owns this boundary"},
]}
policy = aa.entry_for(self.write(data), "release")
root = Path(self.tmp.name, "repo")
root.mkdir()
self.assertEqual(aa.missing_findings(root, policy), [])
self.assertEqual(aa.policy_status(root, policy), {
"status": "exempt", "reason": "release process owns this boundary",
})

def test_exempt_requires_reason(self):
data = {"organization": "openAMRobot", "repositories": [
{"name": "release", "default_branch": "main", "agent_rules": "exempt"},
]}
with self.assertRaisesRegex(ValueError, "agent_rules_reason"):
aa.validate_document(data)

def test_unknown_status_is_rejected(self):
data = {"organization": "openAMRobot", "repositories": [
{"name": "demo", "default_branch": "main", "agent_rules": "optional"},
]}
with self.assertRaisesRegex(ValueError, "required or exempt"):
aa.validate_document(data)

def test_missing_status_is_rejected(self):
data = {"organization": "openAMRobot", "repositories": [
{"name": "demo", "default_branch": "main"},
]}
with self.assertRaisesRegex(ValueError, "required or exempt"):
aa.validate_document(data)


if __name__ == "__main__":
unittest.main()
11 changes: 8 additions & 3 deletions tests/test_org_scan.py
Original file line number Diff line number Diff line change
Expand Up @@ -69,9 +69,14 @@ def test_lists_all_active_repositories_once(self):
self.assertEqual(set(names), ACTIVE)
for r in data["repositories"]:
with self.subTest(repository=r["name"]):
self.assertEqual(set(r), {"name", "default_branch"})
self.assertEqual(set(r), {"name", "default_branch", "agent_rules"})
self.assertRegex(r["name"], r"^[A-Za-z0-9._-]+$")
self.assertRegex(r["default_branch"], r"^[A-Za-z0-9._/-]+$")
self.assertIn(r["agent_rules"], {"required", "exempt"})
if r["agent_rules"] == "exempt":
self.assertTrue(r.get("agent_rules_reason", "").strip())
else:
self.assertNotIn("agent_rules_reason", r)


class Workflow(unittest.TestCase):
Expand Down Expand Up @@ -120,8 +125,8 @@ def test_scan_reports_every_repository_and_blocks_on_clone_failure(self):
self.assertIn("BLOCKED repositories were not scanned and are not clean.", result.summary)
rows = re.findall(r"^\| ([.\w-]+) \| abc1234 \| (\d+) \|$", result.summary, re.M)
self.assertEqual(len(rows), 13)
self.assertIn(("openamr-platform-sw", "1"), rows)
self.assertIn("### openamr-platform-sw: 1 finding(s)", result.summary)
self.assertIn(("openamr-platform-sw", "2"), rows)
self.assertIn("### openamr-platform-sw: 2 finding(s)", result.summary)
self.assertIn("| COMPUTE | 1 |", result.summary)
self.assertEqual(result.blocked, [{"repository": "openamrobot-comm", "reason": "clone failed"}])
# Default dashboard mode: the summary must not promise per-group issues.
Expand Down
22 changes: 16 additions & 6 deletions tests/test_watchdog.py
Original file line number Diff line number Diff line change
Expand Up @@ -80,6 +80,16 @@ def test_clean_repository_and_missing_agents(self):
self.assertIn("Total: 0 finding(s)", out)
self.assertIn("Next step: nothing to fix.", out)

def test_required_repository_without_agents_is_a_finding(self):
with tempfile.TemporaryDirectory() as tmp:
repo = Path(tmp, "openamr-platform-sw")
repo.mkdir()
(repo / "README.md").write_text("Reference compute is the Jetson.\n", encoding="utf-8")
report = watchdog.run(repo, "openamr-platform-sw")
self.assertEqual(report["agent_rules"], {"status": "required", "present": False})
self.assertEqual(report["results"]["shared-rules"][0]["group"], "agent-rules")
self.assertEqual(report["results"]["shared-rules"][0]["found"], "file is missing")

def test_this_repository_passes_its_own_agents_check(self):
report = watchdog.run(ROOT, ".github")
self.assertEqual(report["results"]["shared-rules"], [])
Expand All @@ -101,15 +111,15 @@ def test_json_and_markdown_outputs(self):
with tempfile.TemporaryDirectory() as tmp:
repo = self.make(tmp, agents=False)
md, js = Path(tmp, "summary.md"), Path(tmp, "r.json")
run("--root", repo, "--report-only", "--markdown", md, "--json", js, "--repository", "x")
run("--root", repo, "--report-only", "--markdown", md, "--repository", "y")
run("--root", repo, "--report-only", "--markdown", md, "--json", js, "--repository", "openamr-platform-sw")
run("--root", repo, "--report-only", "--markdown", md, "--repository", "openamr-platform-fw")
text = md.read_text(encoding="utf-8")
self.assertIn("### x: 3 finding(s)", text)
self.assertIn("### y: 3 finding(s)", text)
self.assertIn("| Shared agent rules | not run (no AGENTS.md) |", text)
self.assertIn("### openamr-platform-sw: 4 finding(s)", text)
self.assertIn("### openamr-platform-fw: 4 finding(s)", text)
self.assertIn("| Shared agent rules | 1 |", text)
self.assertIn("| COMPUTE | 1 |", text)
data = json.loads(js.read_text(encoding="utf-8"))
self.assertEqual(data["repository"], "x")
self.assertEqual(data["repository"], "openamr-platform-sw")
self.assertEqual(len(data["results"]["decisions"]), 1)
finding = data["results"]["decisions"][0]
# JSON keeps full detail per finding, unlike the compact text report.
Expand Down
7 changes: 7 additions & 0 deletions tests/test_watchdog_issue_sync.py
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,13 @@ def test_dashboard_observation_is_stable_marker(self):
self.assertIsNotNone(wis.OBSERVATION_RE.search(body))
self.assertIn("Shared rules", body)

def test_dashboard_labels_exempt_agent_rules_explicitly(self):
data = report()
data["agent_rules"] = {"status": "exempt", "reason": "release process owns this boundary"}
plan = wis.plan([data], [], {"COMPUTE": {"owner": "platform-lead"}}, MAINTAINERS, [], "run", "2026-10-08")
body = wis.dashboard_body([data], [], plan, "run", "2026-10-08", [])
self.assertIn("| openamrobot-docs |", body)
self.assertIn("| exempt |", body)

class Api(unittest.TestCase):
def test_apply_reopens_and_comments_without_closing(self):
Expand Down
Loading