Skip to content

ci: use host-local Fireactions caches#2926

Draft
UnArbosFive wants to merge 49 commits into
stagingfrom
codex/fireactions-host-cache-pilot
Draft

ci: use host-local Fireactions caches#2926
UnArbosFive wants to merge 49 commits into
stagingfrom
codex/fireactions-host-cache-pilot

Conversation

@UnArbosFive

@UnArbosFive UnArbosFive commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

What changed

  • discovers the host-local sccache tier and fails open to direct R2
  • routes trusted clone snapshot downloads through an authenticated host-local gateway when available
  • refreshes exact remote compiler keys every six hours
  • mirrors each successful scheduled clone snapshot to an immutable R2 object and publishes its manifest last for per-host polling
  • verifies the exact node-subtensor-runtime Cargo feature graph after cargo clean
  • keeps paired direct-origin and local-cache benchmarks available through manual workflow dispatch
  • uses the original turbo runner label for builds, clone testing, and diagnostics

Why

Fireactions VMs are ephemeral while their physical hosts persist. Immutable compiler objects and large snapshot artifacts can therefore be reused locally instead of being transferred again for every VM, while R2 and GitHub Actions remain authoritative origins.

A GitHub Actions maintenance job can warm only the physical host selected for that job, so it cannot maintain a fleet sharing one production label. Snapshot freshness is now published through R2 for each host to poll. Compiler objects remain content-addressed: a cold local lookup falls through to a remote cache hit and then remains local for later VMs.

The former warm workflow compiled the whole workspace, but representative CI selected only node-subtensor-runtime. Cargo feature unification produced different rustc/sccache keys. The exact-key warmer now cleans the job-local target directory and runs the same command as CI.

Security and failure behavior

  • local endpoints are reachable only from the isolated runner bridge
  • sccache local credentials are bucket-scoped and read-only
  • writer credentials remain restricted to protected producer and prewarm jobs
  • the mirror accepts one exact R2 endpoint and content-addressed object layout, and publishes the manifest only after the archive upload succeeds
  • artifact requests require the short-lived GitHub job token and are checked against repository, workflow, branch, artifact name, digest, and size
  • clients fail open to direct R2 or GitHub Actions
  • fork PRs cannot execute the protected writer prewarm or self-hosted benchmarks

Validation

  • exact runtime verification reached 597 Rust hits and 0 misses after a clean build
  • release compilation improved from 225s on local cold fill to 145s warm
  • a 2 GiB snapshot took 89s from GitHub and 8–9s from the local cache
  • clone-upgrade completed in 365s on cold fill and 309s warm
  • paired same-VM small-object reads showed no material regression versus direct R2
  • mirror signing and manifest boundary tests pass
  • sccache configuration, immutable snapshot selection, artifact boundary, runtime classifier, YAML, and diff checks pass

@UnArbosFive UnArbosFive added the fireactions-cache-benchmark Route selected PR cache benchmarks to the isolated turbo-8 canary label Jul 15, 2026
@vercel

vercel Bot commented Jul 15, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
subtensor Ready Ready Preview, Comment Jul 17, 2026 1:23am

Request Review

@github-actions

github-actions Bot commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

🛡️ AI Review — Skeptic (security review)

VERDICT: SAFE

VERY HIGH account-age scrutiny, mitigated by admin repository permission, substantive merged contributions, matching commit authorship, and no known Gittensor association; codex/fireactions-host-cache-pilot → staging.

Trusted AI-review instructions are unchanged. Static review found fixed cache endpoints, separated reader/writer credentials, protected writer workflows, authenticated artifact access, provenance and digest validation, bounded extraction, trusted-base CI classification, and authoritative-origin fallbacks. The latest runtime-derived type-generation changes operate on the locally built chain specification and introduce no additional credential or production-runtime boundary.

Findings

No findings.

Conclusion

No malicious behavior or security vulnerability was found in the current diff.


🔍 AI Review — Auditor (domain review)

VERDICT: 👍

Gittensor association UNKNOWN; the account is new, but has repository admin permission and substantial recent Subtensor contributions.

The substantive PR description matches the implementation. Overlaps with #2853 and #2804 are incidental file overlap, not competing implementations. The required-check routing remains fail-closed, and the cache paths validate their configuration and fall back safely; static review was sufficient, with no auto-fix needed.

Findings

No findings.

Conclusion

The CI cache and E2E routing changes are well-scoped and have appropriate regression coverage. No substantive domain issues remain.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review — see the sticky summary comment for the verdict and the inline comments below for specific findings.

payload="$RUNNER_TEMP/artifacts.json"
for attempt in 1 2 3 4 5; do
gh api "repos/$GITHUB_REPOSITORY/actions/runs/$PRODUCER_RUN_ID/artifacts?per_page=100" > "$payload"
count=$(jq '[.artifacts[] | select(.name == "mainnet-snapshot" or (.name | startswith("try-runtime-snap-v0.10.1-")))] | length' "$payload")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[LOW] Count only artifacts the warming loop can process

This readiness count includes expired artifacts, but the processing query below excludes them. If a run contains only expired matching artifacts, the retry loop exits and the job succeeds after warming nothing. Apply the same expiry filter here so maintenance fails or retries instead of reporting a false success.

Suggested change
count=$(jq '[.artifacts[] | select(.name == "mainnet-snapshot" or (.name | startswith("try-runtime-snap-v0.10.1-")))] | length' "$payload")
count=$(jq '[.artifacts[] | select(.expired == false) | select(.name == "mainnet-snapshot" or (.name | startswith("try-runtime-snap-v0.10.1-")))] | length' "$payload")

@github-actions

Copy link
Copy Markdown
Contributor

🔄 AI review updated — Skeptic: SAFE Auditor: 👍

@UnArbosFive

UnArbosFive commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

Benchmark results on the standard 8 vCPU / 28 GiB turbo runner shape:

  • Exact runtime prewarm: first clean check 155s with 213 Rust hits / 384 misses; second clean check 49s with 596 hits / 1 miss (99.83% Rust hit rate).
  • Paired same-VM runtime reads: direct R2 34s and 33s (33.5s mean); warm host-local 38s and 31s (34.5s mean). This small-object check is effectively tied.
  • Real PR release build: recent turbo job median 332s; local cold-fill job 280s (225s compile); warm job 206s (145s compile). Both cache-enabled compiles had 1,890 Rust hits / 0 misses.
  • 2.0 GiB snapshot, same VM: direct GitHub 89s; host-local 9s and 8s. The real clone cold fill took 67s; the warm rerun took 19s.
  • Full clone-upgrade: recent turbo median 601s; local cold fill 365s; warm 309s. Both passed.

Exact remote compiler keys refresh every six hours. Scheduled clone snapshots are produced daily with a 36-hour freshness gate and now publish an immutable R2 mirror manifest for per-host polling. There is no random-runner local maintenance job because one selected VM cannot warm a shared-label fleet.

Runs: exact-key verification https://github.com/RaoFoundation/subtensor/actions/runs/29456620982 ; paired/artifact benchmark https://github.com/RaoFoundation/subtensor/actions/runs/29456911456 ; cold + warm PR attempts https://github.com/RaoFoundation/subtensor/actions/runs/29456621901

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review — see the sticky summary comment for the verdict and the inline comments below for specific findings.

payload="$RUNNER_TEMP/artifacts.json"
for attempt in 1 2 3 4 5; do
gh api "repos/$GITHUB_REPOSITORY/actions/runs/$PRODUCER_RUN_ID/artifacts?per_page=100" > "$payload"
count=$(jq '[.artifacts[] | select(.name == "mainnet-snapshot" or (.name | startswith("try-runtime-snap-v0.10.1-")))] | length' "$payload")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[LOW] Count only artifacts the warming loop can process

This count includes expired artifacts and any name with the try-runtime prefix. Such an artifact can end polling even though the loop skips it or download-artifact.sh rejects its name, producing misleading maintenance results. Match the downloader’s exact allowlist and the loop’s expiry filter.

Suggested change
count=$(jq '[.artifacts[] | select(.name == "mainnet-snapshot" or (.name | startswith("try-runtime-snap-v0.10.1-")))] | length' "$payload")
count=$(jq '[.artifacts[] | select(.expired == false) | select(.name == "mainnet-snapshot" or .name == "try-runtime-snap-v0.10.1-mainnet" or .name == "try-runtime-snap-v0.10.1-testnet" or .name == "try-runtime-snap-v0.10.1-devnet")] | length' "$payload")

@github-actions

Copy link
Copy Markdown
Contributor

🔄 AI review updated — Skeptic: SAFE Auditor: 👍

@UnArbosFive

Copy link
Copy Markdown
Contributor Author

Final validation note: a subsequent run reached 597 Rust hits / 0 misses, which exposed that sccache omits the language-specific miss row at zero. The gate initially parsed that omission as empty and failed despite a perfect cache result. Commit e339b14 normalizes omitted counters to zero; the final validation is green with 596 hits / 1 miss on the fill pass and 597 hits / 0 misses (100%) after the second cargo clean.

Final validation: https://github.com/RaoFoundation/subtensor/actions/runs/29458291945

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review — see the sticky summary comment for the verdict and the inline comments below for specific findings.

payload="$RUNNER_TEMP/artifacts.json"
for attempt in 1 2 3 4 5; do
gh api "repos/$GITHUB_REPOSITORY/actions/runs/$PRODUCER_RUN_ID/artifacts?per_page=100" > "$payload"
count=$(jq '[.artifacts[] | select(.name == "mainnet-snapshot" or (.name | startswith("try-runtime-snap-v0.10.1-")))] | length' "$payload")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[LOW] Count only artifacts the warming loop can process

This readiness check counts expired artifacts and any name beginning with try-runtime-snap-v0.10.1-, while the loop excludes expired entries and download-artifact.sh accepts only three exact network names. An unusable artifact can therefore stop polling before a usable one exists. Apply the processing loop/downloader’s expiry and exact-name filters here.

Suggested change
count=$(jq '[.artifacts[] | select(.name == "mainnet-snapshot" or (.name | startswith("try-runtime-snap-v0.10.1-")))] | length' "$payload")
count=$(jq '[.artifacts[] | select(.expired == false) | select(.name == "mainnet-snapshot" or .name == "try-runtime-snap-v0.10.1-mainnet" or .name == "try-runtime-snap-v0.10.1-testnet" or .name == "try-runtime-snap-v0.10.1-devnet")] | length' "$payload")

@github-actions

Copy link
Copy Markdown
Contributor

🔄 AI review updated — Skeptic: SAFE Auditor: 👍

@UnArbosFive UnArbosFive removed the fireactions-cache-benchmark Route selected PR cache benchmarks to the isolated turbo-8 canary label Jul 16, 2026
@UnArbosFive UnArbosFive changed the title ci: pilot host-local Fireactions caches ci: use host-local Fireactions caches Jul 16, 2026

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review — see the sticky summary comment for the verdict and the inline comments below for specific findings.

payload="$RUNNER_TEMP/artifacts.json"
for attempt in 1 2 3 4 5; do
gh api "repos/$GITHUB_REPOSITORY/actions/runs/$PRODUCER_RUN_ID/artifacts?per_page=100" > "$payload"
count=$(jq '[.artifacts[] | select(.name == "mainnet-snapshot" or (.name | startswith("try-runtime-snap-v0.10.1-")))] | length' "$payload")

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[LOW] Count only artifacts the warming loop can process

This count still includes expired artifacts and any name beginning with the try-runtime prefix, while the loop excludes expired entries and download-artifact.sh accepts only three exact try-runtime names. An expired or similarly prefixed artifact can make the retry loop stop despite having nothing usable, or make the warming step fail. Apply the same expired == false and exact-name allowlist predicates here and in the selection at line 69.

Suggested change
count=$(jq '[.artifacts[] | select(.name == "mainnet-snapshot" or (.name | startswith("try-runtime-snap-v0.10.1-")))] | length' "$payload")
count=$(jq '[.artifacts[] | select(.expired == false) | select(.name == "mainnet-snapshot" or .name == "try-runtime-snap-v0.10.1-mainnet" or .name == "try-runtime-snap-v0.10.1-testnet" or .name == "try-runtime-snap-v0.10.1-devnet")] | length' "$payload")

@github-actions

Copy link
Copy Markdown
Contributor

🔄 AI review updated — Skeptic: SAFE Auditor: 👍

@github-actions

Copy link
Copy Markdown
Contributor

🔄 AI review updated — Skeptic: SAFE Auditor: 👍

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review — see the sticky summary comment for the verdict and the inline comments below for specific findings.

# back to the complete matrix rather than guessing. In particular,
# subtensor staking, subnet, and swap code is exercised across several
# nominally separate suites, so all production changes there stay full.
ts-tests/*|common/*|node/*|pallets/*|primitives/*|runtime/*|support/*|chain-extensions/*|src/*|vendor/*|Cargo.toml|build.rs|rust-toolchain.toml|.github/workflows/typescript-e2e.yml|.github/actions/run-typescript-e2e/*|.github/actions/rust-setup/*|.github/actions/sccache-setup/*|.github/scripts/sccache-configure.sh|.github/scripts/sccache-config.py|.github/scripts/sccache-report.sh|.github/scripts/classify-typescript-e2e-changes.sh|.github/scripts/extract-pull-file-paths.sh|.github/scripts/test-classify-typescript-e2e-changes.sh)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MEDIUM] Route lockfile-only dependency changes through E2E

Cargo.lock is explicitly treated as E2E-irrelevant, but a lockfile-only update can change transitive dependencies compiled into the runtime or node. Such a PR would skip every node build and E2E suite. Include Cargo.lock in this production-input arm and update the test that currently expects zero builds for it.

Suggested change
ts-tests/*|common/*|node/*|pallets/*|primitives/*|runtime/*|support/*|chain-extensions/*|src/*|vendor/*|Cargo.toml|build.rs|rust-toolchain.toml|.github/workflows/typescript-e2e.yml|.github/actions/run-typescript-e2e/*|.github/actions/rust-setup/*|.github/actions/sccache-setup/*|.github/scripts/sccache-configure.sh|.github/scripts/sccache-config.py|.github/scripts/sccache-report.sh|.github/scripts/classify-typescript-e2e-changes.sh|.github/scripts/extract-pull-file-paths.sh|.github/scripts/test-classify-typescript-e2e-changes.sh)
ts-tests/*|common/*|node/*|pallets/*|primitives/*|runtime/*|support/*|chain-extensions/*|src/*|vendor/*|Cargo.toml|Cargo.lock|build.rs|rust-toolchain.toml|.github/workflows/typescript-e2e.yml|.github/actions/run-typescript-e2e/*|.github/actions/rust-setup/*|.github/actions/sccache-setup/*|.github/scripts/sccache-configure.sh|.github/scripts/sccache-config.py|.github/scripts/sccache-report.sh|.github/scripts/classify-typescript-e2e-changes.sh|.github/scripts/extract-pull-file-paths.sh|.github/scripts/test-classify-typescript-e2e-changes.sh)

@github-actions

Copy link
Copy Markdown
Contributor

🔄 AI review updated — Skeptic: SAFE Auditor: 👍

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review — see the sticky summary comment for the verdict and the inline comments below for specific findings.

# back to the complete matrix rather than guessing. In particular,
# subtensor staking, subnet, and swap code is exercised across several
# nominally separate suites, so all production changes there stay full.
ts-tests/*|common/*|node/*|pallets/*|primitives/*|runtime/*|support/*|chain-extensions/*|src/*|vendor/*|Cargo.toml|build.rs|rust-toolchain.toml|.github/workflows/typescript-e2e.yml|.github/actions/run-typescript-e2e/*|.github/actions/rust-setup/*|.github/actions/sccache-setup/*|.github/scripts/sccache-configure.sh|.github/scripts/sccache-config.py|.github/scripts/sccache-report.sh|.github/scripts/classify-typescript-e2e-changes.sh|.github/scripts/extract-pull-file-paths.sh|.github/scripts/test-classify-typescript-e2e-changes.sh)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MEDIUM] Route lockfile-only dependency changes through E2E

Cargo.lock remains outside every classifier branch, and the new test explicitly expects Cargo.lock to produce e2e=false. A lockfile-only dependency update can change the node built and exercised by these suites, so skipping all E2E coverage is unsafe. Include Cargo.lock in this full-matrix branch and update the regression test to assert the full matrix for Cargo.lock separately from README.md.

Suggested change
ts-tests/*|common/*|node/*|pallets/*|primitives/*|runtime/*|support/*|chain-extensions/*|src/*|vendor/*|Cargo.toml|build.rs|rust-toolchain.toml|.github/workflows/typescript-e2e.yml|.github/actions/run-typescript-e2e/*|.github/actions/rust-setup/*|.github/actions/sccache-setup/*|.github/scripts/sccache-configure.sh|.github/scripts/sccache-config.py|.github/scripts/sccache-report.sh|.github/scripts/classify-typescript-e2e-changes.sh|.github/scripts/extract-pull-file-paths.sh|.github/scripts/test-classify-typescript-e2e-changes.sh)
ts-tests/*|common/*|node/*|pallets/*|primitives/*|runtime/*|support/*|chain-extensions/*|src/*|vendor/*|Cargo.toml|Cargo.lock|build.rs|rust-toolchain.toml|.github/workflows/typescript-e2e.yml|.github/actions/run-typescript-e2e/*|.github/actions/rust-setup/*|.github/actions/sccache-setup/*|.github/scripts/sccache-configure.sh|.github/scripts/sccache-config.py|.github/scripts/sccache-report.sh|.github/scripts/classify-typescript-e2e-changes.sh|.github/scripts/extract-pull-file-paths.sh|.github/scripts/test-classify-typescript-e2e-changes.sh)

@github-actions

Copy link
Copy Markdown
Contributor

🔄 AI review updated — Skeptic: SAFE Auditor: 👍

@vercel

vercel Bot commented Jul 16, 2026

Copy link
Copy Markdown

Deployment failed with the following error:

Creating the Deployment Timed Out.

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AI review — see the sticky summary comment for the verdict and the inline comments below for specific findings.

Comment thread .github/workflows/typescript-e2e.yml Outdated
Comment on lines +211 to +214
evm-result:
name: typescript-e2e-zombienet_evm
if: always()
needs: [build, run-e2e-tests]

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[HIGH] Do not pass the required check when routing prerequisites fail

This aggregator considers skipped successful. If trusted-pr or changes fails, build and run-e2e-tests are skipped, after which this historical required context reports success. Depend directly on trusted-pr and changes, fail unless both succeeded, and only accept skipped build/test jobs when the classifier explicitly selected no corresponding work. Apply the same correction to shield-result.

# back to the complete matrix rather than guessing. In particular,
# subtensor staking, subnet, and swap code is exercised across several
# nominally separate suites, so all production changes there stay full.
ts-tests/*|common/*|node/*|pallets/*|primitives/*|runtime/*|support/*|chain-extensions/*|src/*|vendor/*|Cargo.toml|build.rs|rust-toolchain.toml|.github/workflows/typescript-e2e.yml|.github/actions/run-typescript-e2e/*|.github/actions/rust-setup/*|.github/actions/sccache-setup/*|.github/scripts/sccache-configure.sh|.github/scripts/sccache-config.py|.github/scripts/sccache-report.sh|.github/scripts/classify-typescript-e2e-changes.sh|.github/scripts/extract-pull-file-paths.sh|.github/scripts/test-classify-typescript-e2e-changes.sh)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[MEDIUM] Route lockfile-only dependency changes through E2E

Cargo.lock can change the dependencies compiled into node-subtensor, so a lockfile-only update can alter the binary exercised by E2E. The classifier currently skips all E2E, and the regression test explicitly preserves that gap. Add Cargo.lock to the shared-input case and update the test to expect the full matrix.

Suggested change
ts-tests/*|common/*|node/*|pallets/*|primitives/*|runtime/*|support/*|chain-extensions/*|src/*|vendor/*|Cargo.toml|build.rs|rust-toolchain.toml|.github/workflows/typescript-e2e.yml|.github/actions/run-typescript-e2e/*|.github/actions/rust-setup/*|.github/actions/sccache-setup/*|.github/scripts/sccache-configure.sh|.github/scripts/sccache-config.py|.github/scripts/sccache-report.sh|.github/scripts/classify-typescript-e2e-changes.sh|.github/scripts/extract-pull-file-paths.sh|.github/scripts/test-classify-typescript-e2e-changes.sh)
ts-tests/*|common/*|node/*|pallets/*|primitives/*|runtime/*|support/*|chain-extensions/*|src/*|vendor/*|Cargo.toml|Cargo.lock|build.rs|rust-toolchain.toml|.github/workflows/typescript-e2e.yml|.github/actions/run-typescript-e2e/*|.github/actions/rust-setup/*|.github/actions/sccache-setup/*|.github/scripts/sccache-configure.sh|.github/scripts/sccache-config.py|.github/scripts/sccache-report.sh|.github/scripts/classify-typescript-e2e-changes.sh|.github/scripts/extract-pull-file-paths.sh|.github/scripts/test-classify-typescript-e2e-changes.sh)

@github-actions

Copy link
Copy Markdown
Contributor

🔄 AI review updated — Skeptic: SAFE Auditor: 👎

@github-actions

Copy link
Copy Markdown
Contributor

🔄 AI review updated — Skeptic: SAFE Auditor: 👍

@github-actions

Copy link
Copy Markdown
Contributor

🔄 AI review updated — Skeptic: SAFE Auditor: 👍

@github-actions

Copy link
Copy Markdown
Contributor

🔄 AI review updated — Skeptic: SAFE Auditor: 👍

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

no-spec-version-bump PR does not contain changes that requires bumping the spec version

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant