ci: accelerate runtime, clone, and TypeScript E2E checks by UnArbosFive · Pull Request #2882 · RaoFoundation/subtensor

UnArbosFive · 2026-07-13T12:23:42Z

What changes

This stacks the runtime/clone work from #2876 and the TypeScript E2E acceleration on #2875.

A daily producer creates one exact-finalized, manifest-backed try-runtime snapshot for each of devnet, testnet, and mainnet. Consumers select each artifact independently by canonical workflow, default branch, repository, immutable run/artifact ID, producer SHA, age, network/CLI identity, size, and SHA-256 checksum.
Cached try-runtime jobs run on ubuntu-latest without chain RPC calls. The try-runtime wasm and release node build in parallel; try-runtime depends only on its wasm. MultiBlockMigrator = () enables --disable-mbm-checks; a non-unit migrator retains the full path and ambiguous detection fails.
The mainnet clone uses a separate daily, preinitialized chainspec/database checkpoint. It has the same provenance-bound lookup, but not the try-runtime finalized-block manifest contract. It is optional and retains live mainnet fallback.
Four clone-upgrade shards become one turbo-8 job and one checkpoint download. Interval sealing produces a block every 250 ms while advancing logical time by a normal 12-second slot. Issuance runs on pristine state; the checkpoint is restored before the other eight regressions run sequentially. This reduces runner use and duplicate setup at the cost of shard-level failure isolation.
Staking, coldkey-swap, EVM, and subnet state suites use one immediately-finalized validator. Shield still runs all six files and 19 tests on its six-node/three-validator topology, split into three measured-time-balanced environments. Two general workers plus three Shield workers peak at five self-hosted test runners; an aggregate job preserves the required Shield check name.
EVM wallets bypass ethers' 250 ms cache for automatic pending-nonce reads; explicit-nonce replacement tests retain their manual behavior. Frontier-state reads are synchronized for the faster environment. Runtime path filtering executes the trusted base revision, validates pagination completeness, classifies both sides of renames, and enables all checks when it cannot classify safely.
Try-runtime consumers and scheduled/release smoke checks move off fireactions-tryruntime; smoke retains its six-hour cadence on ubuntu-latest. Snapshot production uses shared turbo-8 because assembly exceeded hosted-runner memory, so the dedicated try-runtime pool is no longer used. Scheduled sccache maintenance is fixed at 12:00 UTC.

Operational behavior

Try-runtime snapshots warn after 36 hours and are required to be at most 72 hours old. After three missed daily refreshes, normal PR checks fail closed until the producer is dispatched; fresh-try-runtime-state or manual fresh_state explicitly opts into live scraping.
Clone checkpoints are optional for seven days. A missing, expired, or unusable checkpoint triggers a slower live mainnet scrape instead of blocking the clone job.
The producer uses four clients against the dedicated archive endpoint (up to sixteen value-fetch workers). Try-runtime networks are serialized; the clone producer can run alongside them, so a refresh can occupy two shared turbo-8 runners.
Fork PRs cannot execute on persistent self-hosted runners. Existing required branch-protection check names and all three independent try-runtime network jobs are preserved.

Measured PR impact

Runner time below excludes queueing. “Clone path” means the release-node build plus clone jobs; the combined row covers the optimized clone path and TypeScript E2E jobs, not unrelated workflows.

Measure	Before	After	Change
Critical path across these workflows	30m58s	14m40s	52.6% faster
Clone required gate	30m58s	12m21s	60.1% faster
Clone-path self-hosted time / peak runners	59m43s / 4	~12m04s / 1	79.8% less time
TypeScript trigger-to-completion	25m54s	14m40s	43.4% faster
TypeScript self-hosted time / peak test runners	72m05s / 6	45m53s / 5	36.3% less time
Combined clone + TypeScript self-hosted time	131m48s	~57m57s	56.0% less time

Notable components: clone regression waiting fell from ~44m54s aggregate to ~3m; cached mainnet try-runtime takes 3m45s (6m36s from workflow trigger, including ~27s artifact restore/validation); EVM 17m30s -> 6m24s, staking 17m17s -> 4m35s, and coldkey-swap 5m35s -> 1m39s. Shield's critical pure Vitest shard fell from 517.62s to 369.92s (28.5%) while preserving all tests and the six-node/three-validator topology.

Four RPC clients reduced mainnet try-runtime snapshot creation from 39m54s to 21m14s on the former dedicated runner (benchmark). The final shared turbo-8 topology measured 48m43s for mainnet and ~67m30s for all three networks (final-runner measurement); this is off the PR critical path and is the capacity tradeoff for retiring the dedicated pool.

Verification

Runtime/clone measurement: all three cached try-runtime checks, build, clone restore, both isolated upgrades, pristine issuance invariant, all nine regressions, SDK drift, and required gate passed.
Final TypeScript measurement: all suites and the required Shield aggregate passed. Shield A/B/C completed in 6m22s, 6m45s, and 7m34s; all 19 Shield tests passed. The EVM job passed in 6m24s.
Snapshot/provenance/age/live-bypass and trusted path-classifier fixtures pass; the E2E validator covers all Shield files/topologies and four single-node suites. Workflow YAML/shell checks and the focused pallet clippy path pass.
The rebase onto docs: fix "fill or kill type or order" typo in allow_partial doc comments #2875 was conflict-free; git range-diff reports all 33 PR commits patch-identical.

vercel · 2026-07-13T12:23:48Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
subtensor	Ready	Preview, Comment	Jul 13, 2026 3:03pm

github-actions · 2026-07-13T12:25:12Z

🛡️ AI Review — Skeptic (security review)

VERDICT: SAFE

VERY HIGH account-age scrutiny (11-day-old account, no public repos), mitigated by repository admin permission and substantive merged CI work; no Gittensor association or author/committer mismatch; targets integration branch fix-fill-or-kill-doc-typo, which has an open PR into main.

Static analysis covered all 29 changed files, including the latest E2E setup commit, self-hosted runner gating, trusted base-revision path classification, snapshot provenance, artifact restoration, clone orchestration, and accelerated timestamp handling. The PR does not modify .github/ai-review/* or .github/copilot-instructions.md.

Findings

No findings.

Conclusion

No malicious behavior or actionable security vulnerability was identified.

🔍 AI Review — Auditor (domain review)

VERDICT: 👍

Gittensor association UNKNOWN; very new contributor account, mitigated by repository admin permission and prior merged CI work.

The PR description accurately reflects the three Shield shards and five-runner peak. The overlapping open PRs concern unrelated features, and this integration-branch target has no applicable spec_version check.

Findings

No findings.

Prior-comment reconciliation

cefac506: addressed — The PR body states that three Shield workers plus two general workers peak at five self-hosted test runners.

Conclusion

The artifact provenance checks, clone checkpoint orchestration, accelerated sealing, and E2E restructuring remain consistent with the description and adequately validated. No blocking domain issues remain.

github-actions

AI review — see the sticky summary comment for the verdict and the inline comments below for specific findings.

github-actions · 2026-07-13T12:27:10Z

                "type": "zombie",
                "zombieSpec": {
-                    "configPath": "./configs/zombie_node.json",
+                    "configPath": "./configs/zombie_single_node.json",


[HIGH] Accelerated EVM environment has repeatable nonce races

The PR body reports that the exact combined commit failed this suite twice with nonce too low, in different tests after 75 and 87 passes. This suite still performs concurrent transactions through shared raw wallets (not every sender is protected by a single NonceManager), so moving it to the immediately sealing single-node environment has produced a reproducible required-check regression. Serialize the EVM cases/files or coordinate every shared sender's nonce, then demonstrate a clean exact-head run.

github-actions · 2026-07-13T12:27:20Z

🔄 AI review updated — Skeptic: SAFE Auditor: 👎

github-actions

AI review — see the sticky summary comment for the verdict and the inline comments below for specific findings.

github-actions · 2026-07-13T12:42:44Z


        beforeAll(async () => {
            api = context.papi("Node").getTypedApi(subtensor);
            provider = context.ethers("EVM").provider as ethers.JsonRpcProvider;


[HIGH] Nonce manager does not resolve the cross-file race

This manager only coordinates transactions within 01-contract-deploy-call.test.ts. 00-evm-substrate-transfer.test.ts concurrently creates the same deterministic wallet and submits many transactions through its own signer, so both files can still select the same nonce—the failure mode already observed twice on the exact PR head. Serialize the EVM test files or give each file a distinct funded account, then demonstrate a clean full EVM run.

github-actions · 2026-07-13T12:42:52Z

🔄 AI review updated — Skeptic: SAFE Auditor: 👎

github-actions

AI review — see the sticky summary comment for the verdict and the inline comments below for specific findings.

github-actions · 2026-07-13T12:59:16Z

-            binary: release
-          - test: zombienet_shield
-            binary: release
+          - test: zombienet_evm


[HIGH] Accelerated EVM job remains demonstrably flaky

The PR body reports that this exact integrated configuration failed twice with nonce too low, at different tests after 75 and 87 passes. The added NonceManager only covers 01-contract-deploy-call.test.ts; 00-evm-substrate-transfer.test.ts still shares raw wallets among its test cases. A required CI job cannot be merged with a repeatable known race. Serialize the EVM test files/cases or provide shared nonce coordination, then demonstrate a clean exact-head run.

github-actions · 2026-07-13T12:59:27Z

🔄 AI review updated — Skeptic: SAFE Auditor: 👎

github-actions

AI review — see the sticky summary comment for the verdict and the inline comments below for specific findings.

github-actions · 2026-07-13T13:28:04Z

-            binary: release
-          - test: zombienet_shield
-            binary: release
+          - test: zombienet_evm


[HIGH] Accelerated EVM gate remains unproven and documented as flaky

The PR body still reports intermittent nonce too low failures in this accelerated EVM job and explicitly calls them a merge-readiness risk. The new uncached nonce read is plausible, but no post-fix run demonstrates stability. Repeatedly validate this exact commit or serialize the EVM tests before making this configuration a required gate.

github-actions · 2026-07-13T13:32:00Z

🔄 AI review updated — Skeptic: SAFE Auditor: 👎

github-actions · 2026-07-13T14:02:40Z

🔄 AI review updated — Skeptic: SAFE Auditor: 👍

github-actions · 2026-07-13T14:29:58Z

🔄 AI review updated — Skeptic: SAFE Auditor: 👍

github-actions · 2026-07-13T14:51:28Z

🔄 AI review updated — Skeptic: SAFE Auditor: 👍

github-actions · 2026-07-13T15:05:02Z

🔄 AI review updated — Skeptic: SAFE Auditor: 👍

UnArbosFive mentioned this pull request Jul 13, 2026

ci: accelerate runtime, clone, and TypeScript E2E checks #2880

Closed

github-actions Bot reviewed Jul 13, 2026

View reviewed changes

vercel Bot deployed to Preview July 13, 2026 12:40 View deployment

github-actions Bot reviewed Jul 13, 2026

View reviewed changes

UnArbosFive force-pushed the codex/accelerate-runtime-and-e2e-ci branch from 81b1041 to 4311cf0 Compare July 13, 2026 12:56

vercel Bot deployed to Preview July 13, 2026 12:57 View deployment

github-actions Bot reviewed Jul 13, 2026

View reviewed changes

vercel Bot deployed to Preview July 13, 2026 13:26 View deployment

github-actions Bot reviewed Jul 13, 2026

View reviewed changes

vercel Bot deployed to Preview July 13, 2026 14:00 View deployment

vercel Bot deployed to Preview July 13, 2026 14:27 View deployment

UnArbosFive added 11 commits July 13, 2026 16:47

ci: clean up snapshot cache controls

098dd84

ci: move scheduled smoke checks to hosted runners

b2311f2

ci: increase snapshot RPC concurrency

b2ebcf1

ci: speed up cached snapshot downloads

a51cec7

revert: retain faster snapshot download path

033e48b

ci: benchmark parallel snapshot downloads

62868fe

ci: benchmark mainnet restore across runner pools

6af747f

ci: parallelize cached snapshot downloads

435766f

ci: benchmark try-runtime on turbo16

f47e986

ci: run try-runtime on turbo16

df09700

ci: run try-runtime on GitHub-hosted runners

f9ffd00

UnArbosFive added 21 commits July 13, 2026 16:47

ci: consolidate and accelerate clone regressions

5b0f2eb

ci: trust base revision for runtime path filtering

2fbd814

ci: test snapshot producer on hosted runners

f70fa9e

ci: retire dedicated try-runtime runners

650ccd4

ci: drop hosted-only snapshot telemetry

35aee69

ci: cache SDK core builds in clone checks

9b2e973

ci: drop ineffective SDK compiler cache

b7e089b

ci: respect clone RPC page limit

1687152

ci: simplify clone runtime orchestration

0fffc88

ci: benchmark single-node e2e finality

1a830d1

ci: tune single-node e2e block waits

c361271

test: synchronize accelerated evm nonces

7a05c04

ci: validate balanced shield shards

7ab7d2d

ci: finalize accelerated typescript e2e suite

ad25548

test: wait for frontier after substrate transfer

089f96c

test: poll frontier balance after substrate transfer

7525fc9

test: bypass ethers balance cache

e037eda

ci: restore six-hour smoke cadence

d890617

test: bypass cached EVM nonce reads

e45c8a7

ci: shorten typescript e2e critical path

6318e45

ci: keep faster e2e linker path

c83cdba

UnArbosFive changed the base branch from main to fix-fill-or-kill-doc-typo July 13, 2026 14:48

UnArbosFive force-pushed the codex/accelerate-runtime-and-e2e-ci branch from c7dd675 to c83cdba Compare July 13, 2026 14:48

vercel Bot deployed to Preview July 13, 2026 14:49 View deployment

ci: centralize e2e setup and validate shield shards

d823013

vercel Bot deployed to Preview July 13, 2026 15:03 View deployment

unarbos merged commit c87ac28 into fix-fill-or-kill-doc-typo Jul 13, 2026
176 checks passed

UnArbosFive deleted the codex/accelerate-runtime-and-e2e-ci branch July 13, 2026 15:23

Uh oh!

Conversation

UnArbosFive commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes

Operational behavior

Measured PR impact

Verification

Uh oh!

vercel Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Jul 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🛡️ AI Review — Skeptic (security review)

Findings

Conclusion

🔍 AI Review — Auditor (domain review)

Findings

Prior-comment reconciliation

Conclusion

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 13, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 13, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 13, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

Uh oh!

github-actions Bot Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Jul 13, 2026

Uh oh!

github-actions Bot commented Jul 13, 2026

Uh oh!

github-actions Bot commented Jul 13, 2026

Uh oh!

github-actions Bot commented Jul 13, 2026

Uh oh!

github-actions Bot commented Jul 13, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

UnArbosFive commented Jul 13, 2026 •

edited

Loading

vercel Bot commented Jul 13, 2026 •

edited

Loading

github-actions Bot commented Jul 13, 2026 •

edited

Loading