ci: route merge-queue builds to self-hosted runner tiers with hosted fallback - #7172
ci: route merge-queue builds to self-hosted runner tiers with hosted fallback#7172Yicong-Huang wants to merge 4 commits into
Conversation
…fallback Merge-queue entries default to the self-hosted scale sets (arc-heavy/ medium/light/bench-linux); PR events opt in via the ci:self-hosted label; push and dispatch stay on GitHub-hosted runners. Before routing, precheck probes runner availability and queue congestion (optional ARC_STATUS_TOKEN secret) and falls back to ubuntu-latest per tier, so the pipeline never stalls when the self-hosted fleet is down. The ARC_AVAILABLE repo variable is a manual kill switch for the first jobs that cannot probe before queueing. Step conditions become runner- agnostic (runner.os == 'Linux') so matrix entries work on any Linux runner label. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Automated Reviewer SuggestionsBased on the
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #7172 +/- ##
=========================================
Coverage 80.38% 80.38%
- Complexity 3865 3866 +1
=========================================
Files 1160 1160
Lines 46189 46189
Branches 5145 5145
=========================================
+ Hits 37128 37131 +3
+ Misses 7400 7396 -4
- Partials 1661 1662 +1
☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🔴 | bs=10 sw=10 sl=64 | 459 | 0.28 | 21,579/25,298/25,298 us | 🔴 +43.8% / 🔴 +69.9% |
| 🔴 | bs=100 sw=10 sl=64 | 1,034 | 0.631 | 98,284/101,753/101,753 us | 🔴 +6.1% / 🟢 -14.3% |
| ⚪ | bs=1000 sw=10 sl=64 | 1,209 | 0.738 | 830,733/898,744/898,744 us | ⚪ within ±5% / 🟢 +21.8% |
Baseline details
Latest main d31406e from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 459 tuples/sec | 657 tuples/sec | 765.25 tuples/sec | -30.1% | -40.0% |
| bs=10 sw=10 sl=64 | MB/s | 0.28 MB/s | 0.401 MB/s | 0.467 MB/s | -30.2% | -40.1% |
| bs=10 sw=10 sl=64 | p50 | 21,579 us | 15,005 us | 12,703 us | +43.8% | +69.9% |
| bs=10 sw=10 sl=64 | p95 | 25,298 us | 17,684 us | 16,199 us | +43.1% | +56.2% |
| bs=10 sw=10 sl=64 | p99 | 25,298 us | 17,684 us | 19,557 us | +43.1% | +29.4% |
| bs=100 sw=10 sl=64 | throughput | 1,034 tuples/sec | 1,073 tuples/sec | 965.22 tuples/sec | -3.6% | +7.1% |
| bs=100 sw=10 sl=64 | MB/s | 0.631 MB/s | 0.655 MB/s | 0.589 MB/s | -3.7% | +7.1% |
| bs=100 sw=10 sl=64 | p50 | 98,284 us | 92,613 us | 103,830 us | +6.1% | -5.3% |
| bs=100 sw=10 sl=64 | p95 | 101,753 us | 98,531 us | 110,778 us | +3.3% | -8.1% |
| bs=100 sw=10 sl=64 | p99 | 101,753 us | 98,531 us | 118,713 us | +3.3% | -14.3% |
| bs=1000 sw=10 sl=64 | throughput | 1,209 tuples/sec | 1,201 tuples/sec | 992.33 tuples/sec | +0.7% | +21.8% |
| bs=1000 sw=10 sl=64 | MB/s | 0.738 MB/s | 0.733 MB/s | 0.606 MB/s | +0.7% | +21.8% |
| bs=1000 sw=10 sl=64 | p50 | 830,733 us | 848,833 us | 1,016,959 us | -2.1% | -18.3% |
| bs=1000 sw=10 sl=64 | p95 | 898,744 us | 863,900 us | 1,062,501 us | +4.0% | -15.4% |
| bs=1000 sw=10 sl=64 | p99 | 898,744 us | 863,900 us | 1,082,611 us | +4.0% | -17.0% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,436.05,200,128000,459,0.280,21578.57,25298.43,25298.43
1,100,10,64,20,1934.48,2000,1280000,1034,0.631,98284.47,101752.89,101752.89
2,1000,10,64,20,16536.08,20000,12800000,1209,0.738,830733.17,898744.22,898744.22runner-heartbeat.yml runs a trivial job on the fleet every 15 minutes. precheck consults the last successful heartbeat (plain GITHUB_TOKEN) and the ARC_AVAILABLE variable, which the heartbeat watchdog flips automatically (GITHUB_TOKEN cannot write variables, so the watchdog uses AUTO_MERGE_TOKEN and degrades to log-only without it). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
This PR is ready! cc @chenlica @aglinxinyuan @parshimers @sshiv012 for review |
|
|
||
| jobs: | ||
| beat: | ||
| runs-on: arc-light-linux |
There was a problem hiding this comment.
I’m new to this runner setup, so I may be missing an infrastructure guarantee: does the health of arc-light-linux imply that the heavy, medium, and bench scale sets are also healthy? If these tiers can fail independently, this heartbeat could remain successful while a merge-queue job waits for an unavailable heavy or medium runner. Since the per-tier token is optional, would it be safer to make beat a matrix covering every self-hosted label, or maintain a separate health value per tier?
There was a problem hiding this comment.
Good point. The current setup is they are VMs running on the same physical machine. Thus this light job test can serve the signal for physical machine availability and network issues. I want to use that as the sole indicator. If we found out different VMs can die individually, we will need to find a better way to form the signal.
There was a problem hiding this comment.
We can also manually change the flag. Heartbeat is one automation to get the signal. Unfortunately we don't have more stable method with our limited committer privilege. The runner status check requires admin from asf.
There was a problem hiding this comment.
That's very interesting. Might it be worthwhile in this heartbeat job, to check other characteristics of the node health, e.g. free disk space, tool issues? Kind of the things Jenkins has for node health checks. Those things can cause cascading job failures despite the node being reachable.
There was a problem hiding this comment.
Ah good point! And yes we should add it.
A bit more context:
Currently our setup is classic runners which means they are not deleted between CI runs. And yes they may have disk issues, or die unexpectedly. This is what's currently allowed by asf.
There is another better option to do elastic scaling and replacing of runners, i've tested it on my fork. the runners will be shutdown and recreated between CIs, less of disk or other resource concerns. I am still talking to INFRA to enable that mode. I think that's better for the long term
Meanwhile we should do those health checks by ourselves when using those classic runners. I will add some simple logics.
There was a problem hiding this comment.
I noticed several terms used across different configurations: ubuntu-latest, ubuntu-slim, arc-light-linux, and linux. What do these each refer to? I guess they might refer to different type of runner. It’s a bit confusing because their meanings seem to overlap, and they appear to be used somewhat interchangeably.
Thanks for flagging this. Please also refer to Pr description for their meanings. Basically these are different runner labels:
They are not used interchangeably. The script is doing selection on which runner to use, depending on what's available. For example, when github runners are all queued up and not available, we can switch to self-hosted ones. When self-hosted ones are not available, we fall back to github default ones. |
|
Ok one blocker: Classic runners do not clean up after themselves. We need either elastic ones or have our ci clean after producing artifacts. |
5 participants
What changes were proposed in this PR?
Merge-queue builds now default to four self-hosted runner tiers; PR runs opt in via a
ci:self-hostedlabel, and push/scheduled runs stay on GitHub-hosted runners.arc-heavy-linux: amber, amber-integration, frontendarc-medium-linux: platform, pyamberarc-light-linux: small service jobs plus the precheck/aggregator control planearc-bench-linux: dedicated bench machine, so benchmark numbers stay comparableA
runner-heartbeatworkflow proves fleet liveness every 15 minutes: precheck falls back toubuntu-latestwhen the last heartbeat is stale, and the heartbeat watchdog mirrors the verdict into theARC_AVAILABLErepo variable consulted by first-hopruns-onexpressions (also usable as a manual kill switch). An optionalARC_STATUS_TOKENsecret adds real-time per-tier availability and congestion probing. Step conditions switch frommatrix.os == 'ubuntu-latest'torunner.os == 'Linux', making the matrices runner-label agnostic.Rollout: 25 INFRA-provisioned runners are already registered under these labels; we will add the
ci:self-hostedlabel. No new credentials are required — the watchdog reusesAUTO_MERGE_TOKEN, andARC_STATUS_TOKENis optional. Without a label or a merge-queue entry, behavior is unchanged.Any related issues, documentation, discussions?
Closes #7171
Discussion: #6973
How was this PR tested?
ubuntu-latest.NOTE: This PR itself is using
ci:self-hostedlabel to run on the self-hosted runners!Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Fable 5)