Improve evaluate-pr-tests workflow: slash_command + workflow_dispatch, security hardening

[PureWeen]

Description

Overhauls the copilot-evaluate-tests gh-aw workflow — switches to on-demand triggers only (/evaluate-tests slash command + manual workflow_dispatch), adds security hardening, and improves error handling. No auto-runs on PR create/update.

What Changed

Triggers (on-demand only — no auto-runs)

• Add slash_command: evaluate-tests — comment /evaluate-tests on a PR to trigger
• Keep workflow_dispatch — manual trigger from Actions tab with PR number input
• Disable pull_request_target — no auto-evaluation on PR create/update
• Add bots: ["copilot-swe-agent[bot]"] — Copilot-authored PRs can be evaluated
• Add labels: ["pr-review", "testing"] — workflow runs are labeled

Gate step (fast-fail for invalid requests)

• Check PR is OPEN before evaluating (rejects closed/merged PRs with clear message)
• Check for test source files in diff before spinning up agent
• Fall back to REST API for PRs with 300+ files (where gh pr diff returns HTTP 406)
• All API errors surfaced with clear messages — no silent masking
• exit 1 stops the workflow immediately — no wasted agent compute

Access gating (Checkout-GhAwPr.ps1)

• Reject fork PRs (isCrossRepository check)
• Verify PR author has write access (admin/write/maintain roles)
• Fix ConvertFrom-Json ordering — check exit code before JSON parsing
• Make infrastructure restore fatal on failure (was soft warning)
• Remove pre-delete pattern — git checkout overwrites in-place

Workflow improvements

• hide-older-comments: true — previous evaluations auto-collapse
• report-as-issue: false for noop — no issue created when nothing to evaluate
• Timeout bumped 15 → 20 minutes
• Dry-run mode via suppress_output input (workflow_dispatch only)
• Gather-TestContext.ps1 now receives -PrNumber parameter

Security documentation (gh-aw-workflows.instructions.md)

• Add "Before You Build" anti-patterns table — prefer built-in gh-aw features
• Add Security Boundaries section with defense layers table
• Add Rules for gh-aw Workflow Authors (DO/DON'T list)
• Document COPILOT_TOKEN exposure and mitigations
• Add slash_command to fork behavior table
• Update Checkout-GhAwPr.ps1 description to match current behavior

Trigger Behavior

Trigger	When it fires	Who can trigger
/evaluate-tests comment	Comment on a PR	Write-access collaborators + copilot-swe-agent[bot]
workflow_dispatch	Actions tab → "Run workflow" → enter PR number	Write-access collaborators
pull_request_target	Auto on PR create/update	Disabled

Security Model

Based on GitHub Security Lab guidance:

• PR contents treated as passive data (read/analyze, never built or executed)
• Agent runs in sandboxed container with GITHUB_TOKEN and gh CLI scrubbed
• Write operations in separate safe_outputs job (not the agent)
• Agent output limited to max: 1 comment via safe-outputs
• Checkout-GhAwPr.ps1 rejects fork PRs and verifies write access before checkout
• Infrastructure restore is fatal on failure — prevents running with untrusted infra

Validation

Test	PR	Result
Open PR with tests	#34983	✅ Full success (gate → checkout → agent → comment)
No-test PR	#34876	✅ Gate fast-fail ("no test source files")
Merged PR	#34932	✅ Gate fast-fail ("MERGED — skipping")

Known Limitations

• Fork PRs via /evaluate-tests can supply modified .github/skills/ — accepted residual risk (agent sandboxed, output bounded). Tracked as gh-aw#18481
• exit 1 in gate step shows ❌ in GitHub checks for no-test/closed PRs — intentional (no built-in "skip" mechanism in gh-aw steps)
• pull_request_target commented out — can be re-enabled later for auto-evaluation

[github-actions]

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.sh | bash -s -- 34678

Or

• Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/dotnet/maui/main/eng/scripts/get-maui-pr.ps1) } 34678"

[Copilot]

Pull request overview

Updates the gh-aw Evaluate PR Tests workflow triggers so fork PRs can be evaluated (by switching from pull_request to pull_request_target) while keeping the workflow’s gating/conditions aligned with the new event.

Changes:

• Switched workflow trigger from pull_request to pull_request_target and updated related if: conditions.
• Updated the gate step condition to run under pull_request_target.
• Updated the compiled .lock.yml to reflect the new trigger configuration.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 3 comments.

File	Description
.github/workflows/copilot-evaluate-tests.md	Moves evaluation to pull_request_target and updates workflow conditions/gate accordingly.
.github/workflows/copilot-evaluate-tests.lock.yml	Regenerated compiled workflow reflecting the trigger/condition changes.

[Copilot]

The compiled lock file shows roles: all commented out under the on: block, which indicates the roles setting was not applied. This also coincides with the removal of the pre_activation/membership check job, so the workflow no longer gates who can run it. Fix by moving roles: to the correct top-level location in the .md frontmatter and re-compiling so the lock file includes the intended role check behavior.

[Copilot]

jobs.activation no longer depends on a role/membership gate (needs: pre_activation and the needs.pre_activation.outputs.activated check are gone). With pull_request_target, this means the workflow can run with secrets for any matching PR/comment, which is a significant security/cost exposure. After fixing the roles placement in the .md, ensure the compiled lock restores the gating (or add an explicit guard) before allowing activation to proceed.

Suggested change

	(github.event_name == 'pull_request_target' && github.event.pull_request.draft == false) || github.event_name == 'workflow_dispatch' || (github.event_name == 'issue_comment' &&
	(github.event_name == 'pull_request_target' && github.event.pull_request.draft == false && github.event.pull_request.head.repo.fork == false) || github.event_name == 'workflow_dispatch' || (github.event_name == 'issue_comment' &&

[jeffhandley]

You might want to configure to not generate a no-op run report issue (within the frontmatter).

https://github.github.com/gh-aw/patterns/monitoring/#no-op-run-reports

[jeffhandley]

💯

[jeffhandley]

I do think this will still lead to the 'Approve and run workflows' button showing up for PRs from untrusted forks. We need to solidify the guidance we give for when to hit that button. I really wish that button navigated into a list of workflows needing approval for the PR with boxes to select which to approve.

[jeffhandley]

Future-proofing: I suggest renaming to suppress_output in case the output changes (to a PR review for example).

Suggested change

	suppress_comment:
	description: 'Dry-run — evaluate but do not post a comment on the PR'
	suppress_output:
	description: 'Dry-run - evaluate but do not post output on the PR'

[jeffhandley]

I am not certain what identity ends up getting used here; I experimented and I think it's one of these but not sure which. 😄

• copilot
• copilot[bot]
• app/copilot-swe-agent
• copilot-swe-agent
• copilot-swe-agent[bot]

[jeffhandley]

I always guard against forks as well, preventing the workflow from running on forks except for the workflow_dispatch event. Otherwise, PRs within a fork will result in failing workflow runs (vs. starting the workflow and skipping all jobs).

Simple case that needs adapting to your scenario: if: (!github.event.repository.fork) || github.event_name == 'workflow_dispatch'.

[jeffhandley]

These expressions get replaced on their way to the model so this would end up embedding the true or false value into the opening statement. I think you need something more like this (but I recommend validating my understanding here). Note I also reflected my suggested input rename from above.

Suggested change

	When triggered via `workflow_dispatch` with `suppress_comment` = `${{ inputs.suppress_comment }}`:
	- If **true**, perform the full evaluation but **do not** post a comment on the PR. Write the evaluation to the workflow log only. This is useful for testing the skill without spamming the PR.
	- If **false** (default), post the comment as normal.
	When triggered via `workflow_dispatch`, the `suppress_output` input controls behavior.
	- If `${{ inputs.suppress_output }}` == **true**, perform the full evaluation but **do not** post a comment on the PR. Write the evaluation to the workflow log only. This is useful for testing the skill without spamming the PR.
	- If `${{ inputs.suppress_output }}` == **false** (default), post the comment as normal.

[jeffhandley]

Suggested change

	If dry-run mode is active (`suppress_comment` is true), log the evaluation report to stdout and stop — do **not** call `add_comment`.
	If dry-run mode is active (`suppress_output` is true), log the evaluation report to stdout and stop — do **not** call `add_comment`.

[PureWeen]

Addressing Jeff's Review Feedback

Thanks for the thorough review @jeffhandley! Here's what we applied:

✅ Applied

1. Renamed suppress_comment → suppress_output — Future-proofs if output changes to a PR review or other format. (commit f3dc6a9)

2. Disabled no-op run report issues — Added report-as-issue: false under noop: in safe-outputs. Note: the docs suggest this goes under safe-outputs: noop:, not as a top-level property. (commit f3dc6a9)

3. Improved dry-run mode wording — Clarified how ${{ inputs.suppress_output }} expression replacement works in the agent prompt. (commit f3dc6a9)

4. Fixed bot identity — Changed bots: from copilot[bot] to copilot-swe-agent[bot]. Git log shows 882 commits from copilot-swe-agent[bot] and 0 from copilot[bot] in this repo. (commit 7bcc980)

🐛 Additional fix discovered

6. Fixed gate step for large PRs (300+ files) — gh pr diff fails with HTTP 406 for PRs with 300+ changed files (e.g., PR March 30th, Inflight Candidate #34617 inflight candidate). Now falls back to the paginated pulls/files REST API. (commit 04bf387)

📝 Acknowledged (no change needed)

7. "Approve and run" concern — You're right this shows up for fork contributors with pull_request_target. However, dotnet/maui already has two pull_request_target workflows (dogfood-comment.yml and bump-global-json.yml) — both self-gate to avoid fork triggers. Our workflow is different in that we want it to run on fork PRs, which is why the approval button appears. Fork contributors can alternatively use /evaluate-tests comment or maintainers can use workflow_dispatch to bypass this. We understand your broader direction toward issue_comment-only for org-wide patterns and will track that.

All changes compiled clean with gh aw compile. Running validation now via workflow_dispatch against PR #34882 (small, with tests) and PR #34617 (300+ files, gate fallback test).

[jeffhandley]

Today I learned about the slash_command trigger at Command Triggers | GitHub Agentic Workflows. I suggest trying that out here. This could then change the if condition to use needs.activation.outputs.slash_command instead of the comment body.

[PureWeen]

Great catch -- implemented in e7fdf22. Replaced issue_comment + startsWith with slash_command: evaluate-tests (scoped to events: [pull_request_comment, issue_comment]). Simplified the if: condition since the platform handles command matching now. Also added an anti-patterns table to the gh-aw instructions file so we don't miss built-in features like this in the future.

[PureWeen]

🔍 Multi-Model Code Review — PR #34678

Reviewed by: 3 independent reviewers + adversarial consensus rounds (3 review cycles)
Files: 4 changed (workflow source .md, compiled .lock.yml, PowerShell script, instruction docs)
CI Status: ⚠️ maui-pr skipping (expected — no runtime code changes), license/cla ✅, dogfood-comment ✅

---

Final Review (commits d142b43 → 72356a7)

All findings from previous review cycles have been addressed. This is the third and final pass.

---

All Findings — Final Status

#	Finding	Severity	Status
1	isFork not enforced in Checkout-GhAwPr.ps1	🟡	✅ FIXED — fork gate at line 64
2	Access gate dead code / stale docs	🟡	✅ FIXED — docs updated for slash_command
3	Fork SKILL.md risk via checkout_pr_branch.cjs	🟡	✅ DOCUMENTED — accepted residual risk (gh-aw#18481)
4	Restore failure silently swallowed	🟢	✅ FIXED — fatal exit 1 on failure
5	Gate exit 1 noise for no-ops	🟢	✅ BY DESIGN — documented as intentional
6	REST fallback fires on "no matches"	🟢	✅ FIXED — if/else on gh pr diff exit code
7	if: issue_comment tautological	🟢	✅ FIXED — removed; compiler generates proper filtering
8	ConvertFrom-Json before LASTEXITCODE	🟢	✅ FIXED — try/catch with separate exit code check
9	Docs reference pull_request_target as active	🟢	✅ FIXED — updated to slash_command
10	API errors masked as "PR not OPEN"	🟢	✅ FIXED — exit code checked separately
11	REST API fallback masks errors as "no test files"	🟢	✅ FIXED — fallback errors surfaced with exit 1

Score: 11/11 resolved (3/3 reviewers confirm on final pass)

---

Non-Issues Verified ✅

• COPILOT_TOKEN exposure: Documented with layered mitigations (firewall, redaction, threat detection)
• Concurrency group: Correct for current issue_comment-only trigger
• hide-older-comments: true: Platform feature, correctly used
• REST API pagination: --paginate with proper error handling
• Permission check: Collaborator API approach is correct
• noop report-as-issue: false: Correctly set
• PR_NUMBER injection: Integer from github.event.issue.number — safe
• Security documentation: Anti-patterns table and defense layers are high-quality additions
• Fork gate ordering: Runs before checkout — correct
• slash_command compiler output: Proper /evaluate-tests matching + PR-only filtering

Test Coverage

Workflow infrastructure — no automated tests available. Validation via /evaluate-tests slash command invocations is appropriate.

---

Recommendation

✅ Approve — All 11 findings across 3 review cycles are resolved. No new issues found on final pass (3/3 reviewers agree). PR is ready to merge.