[CI] Add parity auto-trigger workflow by ethanwee1 · Pull Request #3231 · ROCm/pytorch

ethanwee1 · 2026-05-18T14:26:03Z

Summary

Add a scheduled parity auto-trigger that scans completed pytorch/pytorch main trunk.yml pushes and dispatches parity.yml once per ready upstream SHA.
Gate dispatch on the ROCm arch workflows that actually ran for a SHA, plus the CUDA jobs consumed by parity, so partial reports are avoided.
Add a pull_request dry-run path with a smaller scan window to validate the scanner without creating parity reports from PR CI.

How it works

The workflow runs every 10 minutes and queries recent completed pytorch/pytorch trunk.yml push runs on main. Those trunk runs provide the candidate upstream SHAs to evaluate.
For each candidate SHA, it first checks recent ROCm/pytorch parity.yml run titles. If any existing parity run already contains that SHA, the SHA is skipped so we keep one report per upstream commit.
Maximum number of dispatches of parity.yml are 50, which is comfortably above the maximum number of commits to main branch of pytorch/pytorch in any 10-minute interval
It then lists all upstream workflow runs for that SHA and determines which ROCm arches actually ran. Missing periodic arch workflows are not treated as pending work; only arches with matching workflow files are expected in that report.
For the arches that did run, it lists upstream check-runs and waits for the matching ROCm test shards to reach status=completed. It also waits for the CUDA default, distributed, and inductor check-runs consumed by parity.
Auxiliary shards such as mem_leak_check and rerun_disabled_tests are ignored because the parity report does not consume them.
Once all relevant ROCm and CUDA check-runs are complete, it dispatches parity.yml with the ready arch list and a CSV prefix containing the upstream SHA, for example autoparity-YYYYMMDD-<sha>.
Pull request runs are forced to dry_run=true, so they exercise the scanner and log would-be dispatches without creating reports. Scheduled and manually dispatched runs can create real parity reports.

Test plan

Validated workflow YAML and embedded shell locally with yaml.BaseLoader and bash -n.
PR dry-run workflow succeeded: https://github.com/ROCm/pytorch/actions/runs/26039732579
Full non-dry-run workflow_dispatch succeeded: https://github.com/ROCm/pytorch/actions/runs/26041358738
The full run used dry_run=false, scanned 20 recent upstream trunk runs, skipped SHAs with pending parity check-runs, dispatched 5 ready SHAs, and stopped at max_dispatches=5.
Dispatched parity reports all completed successfully:
- d76e83ef / mi355: https://github.com/ROCm/pytorch/actions/runs/26041518406
- 457e1890 / mi355: https://github.com/ROCm/pytorch/actions/runs/26041528996
- 60f38508 / mi355: https://github.com/ROCm/pytorch/actions/runs/26041541647
- d1d96569 / mi355: https://github.com/ROCm/pytorch/actions/runs/26041551854
- 6e3cf2e4 / mi355, mi300, mi200: https://github.com/ROCm/pytorch/actions/runs/26041618237

Dispatch cadence note

The full validation run used max_dispatches=5 only to avoid flooding ROCm/pytorch during manual testing.
The production scheduled workflow runs every 10 minutes and defaults to max_dispatches=50, max_commits=200, and max_age_hours=72 unless manually overridden.

rocm-repo-management-api · 2026-05-18T14:37:05Z

Jenkins build for f4dfbd8845f2d05dd28225ca78af48c1926d9e31 commit finished as FAILURE
Links: Pipeline Overview / Build artifacts / Test Results

Copilot

Pull request overview

Adds a new GitHub Actions workflow to automatically scan recent completed trunk.yml push runs on main, determine when the relevant ROCm + CUDA check-runs for a given upstream SHA are fully complete, and then dispatch parity.yml (with a PR-only dry-run mode).

Changes:

Introduces a scheduled (every 10 minutes) + manual + PR dry-run “parity auto-trigger” workflow.
Implements SHA deduplication by checking existing parity.yml run titles in the current repo.
Gates dispatch on completion of ROCm arch shard check-runs (only for arch workflows detected as having run) plus specific CUDA check-runs used by parity.

jithunnair-amd · 2026-05-26T11:25:31Z

+          COMMITS=$(gh api \
+            "repos/$UPSTREAM/actions/workflows/trunk.yml/runs?branch=$BRANCH&event=push&status=completed&per_page=$MAX_COMMITS" \
+            --jq '
+              reduce .workflow_runs[] as $run ({seen:{}, rows:[]};
+                if .seen[$run.head_sha] then .
+                else .seen[$run.head_sha] = true | .rows += [$run]
+                end
+              )
+              | .rows[]
+              | "\(.head_sha) \(.created_at)"
+            ')


This is a valid point. per_page is capped at 100.

jithunnair-amd · 2026-05-26T11:31:13Z

+        description: 'JSON: arch -> PCRE regex that matches the check-run names of that arch''s ROCm test shards on pytorch/pytorch. An arch is considered "ready" only when every check-run whose name matches has status=completed (so we wait for all test shards, not just workflow completion).'
+        required: false
+        default: '{"mi355":"rocm.*mi355.*/ test [(](default|distributed|inductor),","mi300":"rocm.*mi300.*/ test [(](default|distributed|inductor),","mi200":"(rocm.*(mi200|mi210).*/ test [(](default|distributed|inductor),|linux-jammy-rocm-py3[.]10 / test [(](default|distributed|inductor),)","navi31":"rocm.*navi31.*/ test [(]default,","nightly":"rocm-nightly.*/ test [(](default|distributed|inductor),"}'
+        type: string
+      arch_workflow_regex_map:
+        description: 'JSON: arch -> PCRE regex that matches workflow file paths for upstream ROCm workflows that mean this arch ran on the SHA. Missing workflows mean the arch is not expected for that commit.'
+        required: false
+        default: '{"mi355":"(^|/)(trunk|rocm-mi355|periodic-rocm-mi355|inductor-rocm-mi355)[.]yml$","mi300":"(^|/)(rocm-mi300|periodic-rocm-mi300|inductor-rocm-mi300)[.]yml$","mi200":"(^|/)(trunk-rocm-sandbox|rocm-mi200|periodic-rocm-mi200|inductor-rocm-mi200)[.]yml$","navi31":"(^|/)(rocm-navi31|periodic-rocm-navi31|inductor-rocm-navi31)[.]yml$","nightly":"(^|/)rocm-nightly[.]yml$"}'
+        type: string


Same thoughts as https://github.com/ROCm/pytorch/pull/3231/changes#r3303391052

jithunnair-amd · 2026-05-21T06:12:40Z

+          # Pull recent parity runs. Run titles look like:
+          #   "<csv_name or SHA> · mi355, mi300, mi200"
+          # Once any parity run exists for a SHA, we do not dispatch another
+          # report for that SHA. This keeps the dashboard to one report per
+          # upstream commit.
+          EXISTING=$(gh run list \
+            --repo "$GITHUB_REPOSITORY" \
+            --workflow parity.yml \
+            --limit 1000 \
+            --json displayTitle 2>/dev/null || echo '[]')
+
+          sha_already_dispatched() {
+            local sha="$1"
+            echo "$EXISTING" | jq -e --arg sha "$sha" \
+              'any(.[]; .displayTitle | contains($sha))' >/dev/null


Since the same parity.yml can be triggered outside of the auto-parity, we should narrow this down to consider only the auto-parity-triggered runs of parity.yml. In addition to SHA, one more "variable" is the arch list. For simplicity, let us get rid of the arch list workflow input entirely in auto-parity, so that we don't have to worry about whether a particular run covered all the archs we wanted.

That being said, I do prefer the suggested version by Copilot review above, since it gets rid of the magic number of 1000 and seems to make the window logic more robust by using the right API parameters.

jithunnair-amd

@ethanwee1 I may not be able to help get this merged once you address the review comments, so please take Pruthvi's help for that. However, please do fix some of the issues pointed out in the review comments and make the inputs interface a bit simpler for the average user (please work with @pablo-garay to iterate on this).
I'm also okay with having the cron-scheduled runs only covering trunk in this PR, since that's our top requirement, and handle the complexity of other archs/workflows in a follow-up PR.

Thanks!

jithunnair-amd · 2026-05-20T00:28:09Z

+        default: 'mi355, mi300, mi200'
+        type: string
+      arch_jobname_regex_map:
+        description: 'JSON: arch -> PCRE regex that matches the check-run names of that arch''s ROCm test shards on pytorch/pytorch. An arch is considered "ready" only when every check-run whose name matches has status=completed (so we wait for all test shards, not just workflow completion).'


"not just workflow completion"... why not wait for workflow completion?

jithunnair-amd · 2026-05-20T00:30:23Z

+      arch_jobname_regex_map:
+        description: 'JSON: arch -> PCRE regex that matches the check-run names of that arch''s ROCm test shards on pytorch/pytorch. An arch is considered "ready" only when every check-run whose name matches has status=completed (so we wait for all test shards, not just workflow completion).'
+        required: false
+        default: '{"mi355":"rocm.*mi355.*/ test [(](default|distributed|inductor),","mi300":"rocm.*mi300.*/ test [(](default|distributed|inductor),","mi200":"(rocm.*(mi200|mi210).*/ test [(](default|distributed|inductor),|linux-jammy-rocm-py3[.]10 / test [(](default|distributed|inductor),)","navi31":"rocm.*navi31.*/ test [(]default,","nightly":"rocm-nightly.*/ test [(](default|distributed|inductor),"}'


This logic seems to repeat (at the risk of conflicting or being out-of-sync) with the download_testlogs logic that determines which workflows and jobs to download. We should avoid that.

Also, this being a user input for workflow_dispatch seems very untenable, for a user to provide it. I'd recommend we remove these inputs, as the default value is set later anyway. And then we need to iterate to figure out how to avoid the apparent duplication vis-a-vis download_testlogs

jithunnair-amd · 2026-05-21T06:12:40Z

+          # Pull recent parity runs. Run titles look like:
+          #   "<csv_name or SHA> · mi355, mi300, mi200"
+          # Once any parity run exists for a SHA, we do not dispatch another
+          # report for that SHA. This keeps the dashboard to one report per
+          # upstream commit.
+          EXISTING=$(gh run list \
+            --repo "$GITHUB_REPOSITORY" \
+            --workflow parity.yml \
+            --limit 1000 \
+            --json displayTitle 2>/dev/null || echo '[]')
+
+          sha_already_dispatched() {
+            local sha="$1"
+            echo "$EXISTING" | jq -e --arg sha "$sha" \
+              'any(.[]; .displayTitle | contains($sha))' >/dev/null


Since the same parity.yml can be triggered outside of the auto-parity, we should narrow this down to consider only the auto-parity-triggered runs of parity.yml. In addition to SHA, one more "variable" is the arch list. For simplicity, let us get rid of the arch list workflow input entirely in auto-parity, so that we don't have to worry about whether a particular run covered all the archs we wanted.

jithunnair-amd · 2026-05-26T11:25:31Z

+          COMMITS=$(gh api \
+            "repos/$UPSTREAM/actions/workflows/trunk.yml/runs?branch=$BRANCH&event=push&status=completed&per_page=$MAX_COMMITS" \
+            --jq '
+              reduce .workflow_runs[] as $run ({seen:{}, rows:[]};
+                if .seen[$run.head_sha] then .
+                else .seen[$run.head_sha] = true | .rows += [$run]
+                end
+              )
+              | .rows[]
+              | "\(.head_sha) \(.created_at)"
+            ')


This is a valid point. per_page is capped at 100.

jithunnair-amd · 2026-05-26T11:31:13Z

+        description: 'JSON: arch -> PCRE regex that matches the check-run names of that arch''s ROCm test shards on pytorch/pytorch. An arch is considered "ready" only when every check-run whose name matches has status=completed (so we wait for all test shards, not just workflow completion).'
+        required: false
+        default: '{"mi355":"rocm.*mi355.*/ test [(](default|distributed|inductor),","mi300":"rocm.*mi300.*/ test [(](default|distributed|inductor),","mi200":"(rocm.*(mi200|mi210).*/ test [(](default|distributed|inductor),|linux-jammy-rocm-py3[.]10 / test [(](default|distributed|inductor),)","navi31":"rocm.*navi31.*/ test [(]default,","nightly":"rocm-nightly.*/ test [(](default|distributed|inductor),"}'
+        type: string
+      arch_workflow_regex_map:
+        description: 'JSON: arch -> PCRE regex that matches workflow file paths for upstream ROCm workflows that mean this arch ran on the SHA. Missing workflows mean the arch is not expected for that commit.'
+        required: false
+        default: '{"mi355":"(^|/)(trunk|rocm-mi355|periodic-rocm-mi355|inductor-rocm-mi355)[.]yml$","mi300":"(^|/)(rocm-mi300|periodic-rocm-mi300|inductor-rocm-mi300)[.]yml$","mi200":"(^|/)(trunk-rocm-sandbox|rocm-mi200|periodic-rocm-mi200|inductor-rocm-mi200)[.]yml$","navi31":"(^|/)(rocm-navi31|periodic-rocm-navi31|inductor-rocm-navi31)[.]yml$","nightly":"(^|/)rocm-nightly[.]yml$"}'
+        type: string


Same thoughts as https://github.com/ROCm/pytorch/pull/3231/changes#r3303391052

jithunnair-amd · 2026-05-26T11:35:34Z

+          # Pull recent parity runs. Run titles look like:
+          #   "<csv_name or SHA> · mi355, mi300, mi200"
+          # Once any parity run exists for a SHA, we do not dispatch another
+          # report for that SHA. This keeps the dashboard to one report per
+          # upstream commit.
+          EXISTING=$(gh run list \
+            --repo "$GITHUB_REPOSITORY" \
+            --workflow parity.yml \
+            --limit 1000 \
+            --json displayTitle 2>/dev/null || echo '[]')
+
+          sha_already_dispatched() {
+            local sha="$1"
+            echo "$EXISTING" | jq -e --arg sha "$sha" \
+              'any(.[]; .displayTitle | contains($sha))' >/dev/null


That being said, I do prefer the suggested version by Copilot review above, since it gets rid of the magic number of 1000 and seems to make the window logic more robust by using the right API parameters.

rocm-repo-management-api · 2026-05-27T14:37:15Z

Jenkins build for 1375a6adeea743262fe99f276cb8afb5c511af86 commit finished as NOT_BUILT
Links: Pipeline Overview / Build artifacts / Test Results

ethanwee1 · 2026-05-27T14:43:08Z

Addressed the latest review feedback from Jithun:

Simplified the workflow_dispatch interface by removing the user-facing arch and regex-map inputs.
Scoped this PR to completed upstream trunk.yml pushes and mi355 default parity only; broader arch/workflow coverage can follow separately.
Removed the check-run/workflow regex duplication with download_testlogs. Readiness is now based on completed upstream trunk workflow runs for the trunk-only report.
Narrowed dedupe to auto-parity-created parity runs by using the autoparity-* run title prefix, so manual parity.yml runs do not suppress automation.
Reworked trunk run scanning to page until max_commits unique SHAs are collected, so values above GitHub's per_page=100 cap work without scanning the entire workflow history.

Validation:

YAML parse + extracted shell bash -n passed locally.
Local dry-run scan passed.
Remote workflow_dispatch dry-run passed: https://github.com/ROCm/pytorch/actions/runs/26518294404

rocm-repo-management-api · 2026-05-27T14:52:11Z

Jenkins build for 1375a6adeea743262fe99f276cb8afb5c511af86 commit finished as FAILURE
Links: Pipeline Overview / Build artifacts / Test Results

Add a scheduled scanner that dispatches one parity report per ready upstream PyTorch main commit, with PR dry-runs to validate readiness without creating reports.

Scope auto parity to completed trunk mi355 default reports, remove user-facing regex inputs, and use paginated workflow APIs for candidate and dedupe windows.

Fetch completed trunk runs page by page only until the configured unique SHA limit is reached, avoiding full workflow-history scans.

rocm-repo-management-api · 2026-05-28T18:07:09Z

Jenkins build for d7ff112ed2257d50c0031bdd24d25675b3f89e52 commit finished as NOT_BUILT
Links: Pipeline Overview / Build artifacts / Test Results

Stop passing csv_name from parity-auto so auto-dispatched parity reports use the same output names as direct parity.yml runs.

Restore the auto parity ready-arch dispatch behavior while preserving parity.yml's default CSV naming by omitting csv_name.

rocm-repo-management-api · 2026-05-28T18:21:53Z

Jenkins build for d7ff112ed2257d50c0031bdd24d25675b3f89e52 commit finished as FAILURE
Links: Pipeline Overview / Build artifacts / Test Results

jithunnair-amd requested a review from Copilot May 20, 2026 00:23

Copilot started reviewing on behalf of jithunnair-amd May 20, 2026 00:25 View session

Copilot AI reviewed May 20, 2026

View reviewed changes

jithunnair-amd requested changes May 26, 2026

View reviewed changes

ethanwee1 requested a review from pablo-garay May 26, 2026 18:47

ethanwee1 added 3 commits May 28, 2026 18:02

[CI] Add parity auto-trigger workflow

061999a

Add a scheduled scanner that dispatches one parity report per ready upstream PyTorch main commit, with PR dry-runs to validate readiness without creating reports.

[CI] Simplify parity auto trigger scope

1945a34

Scope auto parity to completed trunk mi355 default reports, remove user-facing regex inputs, and use paginated workflow APIs for candidate and dedupe windows.

[CI] Limit parity auto trunk pagination

a2b268a

Fetch completed trunk runs page by page only until the configured unique SHA limit is reached, avoiding full workflow-history scans.

ethanwee1 force-pushed the ethanwee/parity-auto-every-commit branch from 1375a6a to a2b268a Compare May 28, 2026 18:03

ethanwee1 added 2 commits May 28, 2026 18:09

[CI] Preserve parity default CSV naming

1495013

Stop passing csv_name from parity-auto so auto-dispatched parity reports use the same output names as direct parity.yml runs.

[CI] Keep multi-arch auto parity dispatch

d7ff112

Restore the auto parity ready-arch dispatch behavior while preserving parity.yml's default CSV naming by omitting csv_name.

Conversation

ethanwee1 commented May 18, 2026 • edited by jithunnair-amd Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

How it works

Test plan

Dispatch cadence note

Uh oh!

rocm-repo-management-api Bot commented May 18, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jithunnair-amd left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rocm-repo-management-api Bot commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ethanwee1 commented May 27, 2026

Uh oh!

rocm-repo-management-api Bot commented May 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rocm-repo-management-api Bot commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rocm-repo-management-api Bot commented May 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ethanwee1 commented May 18, 2026 •

edited by jithunnair-amd

Loading

rocm-repo-management-api Bot commented May 18, 2026 •

edited

Loading

rocm-repo-management-api Bot commented May 27, 2026 •

edited

Loading

rocm-repo-management-api Bot commented May 27, 2026 •

edited

Loading

rocm-repo-management-api Bot commented May 28, 2026 •

edited

Loading

rocm-repo-management-api Bot commented May 28, 2026 •

edited

Loading