ci: self-hosted reliability — case-opt timeout/clean, monitor crash, compiler labels

[sbryngelson]

CI-infrastructure only — no solver or toolchain code touched. Four commits, each independent.

1. frontier_amd case-opt timeout (commit 4fcaaac6)

The frontier_amd case-optimization Pre-Build times out at the Frontier batch partition's 01:59:00 (~2h) cap, even 2-way sharded (dies ~120 min, run 29198726737).

Root cause, from the job log:

• AMD flang case-opt simulation builds run ~30 min/case (measured: 5eq_hllc = 32 min).
• The pre-build globbed all 7 benchmarks/*/case.py, but the run only tests 5 — two built for nothing.
• 2-way → heavy shard = 4 cases × ~30 min = ~120 min = the timeout.

Fix: new .github/scripts/case-optimization-benchmarks.sh (the 5 run-tested cases as a single source of truth + shared shard helpers, sourced by both scripts so binaries and run cases can't drift); pre-build and run both shard 3-way (heavy shard ≈2 cases ≈60 min); phoenix unsharded; log/archive globs shard-agnostic.

2. Dead staging-clean regex (commit 04b06882) — reliability fix

The stale-staging cleanup used -regex '.*/[0-9a-f]+', matching pure-hex names only. But MFC staging slugs are <prefix>-<hash> with prefix ∈ {gpu-acc,gpu-mp,cpu} (+ optional -debug/-chem/…), e.g. gpu-mp-b86d2efce8 (build.py get_slug). So the clean matched nothing and removed no staging at all (all 6 sites: prebuild/run/workflow × staging+install).

Consequence: when a case-opt build is killed mid-configure (e.g. the pre-build hitting its walltime), it leaves a corrupt staging dir the clean fails to remove; the next run on that self-hosted runner reuses it and CMake configure fails deterministically — CMAKE_Fortran_COMPILER not set in FindOpenMP's try_compile, plus Directory not empty on the leftover CMakeFiles. This is exactly the failure seen on the retry of run 29198726737 (identical across all 3 sbatch attempts — not a race). Now matches <prefix>-*, removing target dirs while preserving dep dirs (fftw, hipfort, …).

3–4. Correct compiler labels on self-hosted lanes (commits d36d8add, 81fcad99)

Check names didn't match the compiler they run; verified each against toolchain/modules:

• Frontier: ran CCE (cpe/25.03) but displayed bare Oak Ridge | Frontier while its AMD sibling was (AMD) and bench.yml already used (CCE). Now (CCE).
• Phoenix: split by device — p-gpu = nvfortran, p-cpu = gcc/12.3.0. Lanes were bare, and bench.yml's Phoenix CPU benchmark was mistagged (NVHPC) though it builds with GNU. Now tagged per lane: GPU (NVHPC), CPU (GNU).

Validation

• New regex tested: removes every <prefix>-<hash> variant, preserves dep dirs.
• 3-way partition proven exhaustive; malformed shards abort; unsharded clusters inert.
• Scripts pass bash -n, both workflows parse, precheck green on every commit.

5. SLURM monitor crash (commit b958ca4b) — shared harness

monitor_slurm_job.sh streamed job output with read -t 1 over a tail -f process substitution. When that pipe broke, bash could crash in the read-builtin's alarm/longjmp unwind path (SIGSEGV 139, unwind_frame_run: read_builtin: frame not found), and the EXIT trap would then scancel a still-RUNNING, healthy job — a red X that mimics a real test failure (hit on a Phoenix lane mid-suite at 341/596, all golden-matching). Replaced with a plain backgrounded tail -f + the existing status poll (state was already polled separately); the final cat still reprints the whole file, so output stays complete. Removes the crash-prone construct; behavior-preserving otherwise.

Blast radius: this is the monitor used by every self-hosted lane (all Frontier + Phoenix), so it is the one change here that is not case-opt-scoped. It cannot be fully exercised without real SLURM; the change is minimal and behavior-preserving, and this PR run is the live test.

[Copilot]

Pull request overview

This PR adjusts the case-optimization CI on frontier_amd to avoid SLURM batch walltime timeouts by (1) sharding the workload 3-way and (2) ensuring the pre-build and run steps iterate an identical benchmark list so they cannot drift.

Changes:

• Introduces a shared benchmark list + sharding helpers in .github/scripts/case-optimization-benchmarks.sh, sourced by both pre-build and run scripts.
• Updates prebuild-case-optimization.sh and run_case_optimization.sh to filter cases by an optional job_shard (i/N) consistently.
• Updates .github/workflows/test.yml to run frontier_amd pre-build and run steps as three concurrent shard jobs; adjusts log printing and artifact globs to be shard-agnostic.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File	Description
.github/workflows/test.yml	Shards frontier_amd pre-build and run steps 3-way; updates log/archive globs for shard outputs.
.github/scripts/run_case_optimization.sh	Sources shared benchmark list and applies shard filtering during build/run loops.
.github/scripts/prebuild-case-optimization.sh	Sources shared benchmark list, parses shard, and uses shard filtering for dry-run prebuilds.
.github/scripts/case-optimization-benchmarks.sh	New single source of truth for benchmark list and shard parsing/filter helpers.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 60.95%. Comparing base (977fcff) to head (b958ca4).
⚠️ Report is 1 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #1643   +/-   ##
=======================================
  Coverage   60.95%   60.95%           
=======================================
  Files          83       83           
  Lines       20003    20003           
  Branches     2983     2983           
=======================================
  Hits        12193    12193           
  Misses       5782     5782           
  Partials     2028     2028           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.