[SPARK-57144][INFRA] Unify Coursier cache to a single key across all jobs

[zhengruifeng]

What changes were proposed in this pull request?

Replace 8 distinct per-job Coursier cache keys with a single coursier-<hash> key in .github/workflows/build_and_test.yml and python_hosted_runner_test.yml:

• precompile and build (Scala test matrix): actions/cache@v5 — both can write coursier-<hash>. precompile is the primary writer (runs first, full dependency superset via all -P profiles). build is the fallback writer — when precompile is absent or its save fails, the first build matrix entry seeds the cache. When precompile did save it, build gets an exact key hit and GHA automatically skips the post-save (caches are immutable).
• All other consumers (pyspark ×9, sparkr, lint, docs, tpcds-1g, docker-integration-tests, k8s-integration-tests): converted to actions/cache/restore@v5 — restore-only, never write. tpcds-1g in particular only fires when SQL code changes and is skipped on the vast majority of runs, so its own Coursier cache entry would typically be LRU-evicted before the next run anyway.

Why are the changes needed?

1. Same-commit duplicates — ~0.01% apart by bytes.

Per-job keys let every consumer job re-save its own copy of effectively the same content. Measured on master:

precompile-coursier-4f7e6f95   1,469,711,354 B   current superset
25-hadoop3-coursier-4f7e6f95   1,469,562,712 B   145 KB different  ← 0.01% apart
precompile-coursier-03ca361a   1,624,890,072 B   previous-hash superset (~90% same)
────────────────────────────────────────────────────────────────────
total:                        ~4.56 GB            distinct content: ~1.47 GB

The 145 KB delta exists because Coursier doesn't prune: on a cold run the test-matrix job restores the precompile superset via restore-key, runs tests (which resolve nothing beyond it), and its post-step re-saves a byte-for-byte copy under its own key. The per-module keys are not holding different dependency sets — they are holding copies of the same superset.

2. Repo-wide 10 GB budget consumed by duplicates.

Duplicates from just two branches left no room for any other branch:

branch-4.x:  tpcds-coursier               1895 MB
             21-hadoop3-coursier           1437 MB
             docker-integration-coursier   1437 MB   → 4770 MB

master:      precompile-coursier (hash A)  1549 MB
             precompile-coursier (hash B)  1401 MB
             25-hadoop3-coursier (hash B)  1401 MB   → 4351 MB

total: ~9.1 GB used, 10 GB budget

Old maintenance branches (branch-4.0, 4.1, 4.2, 3.5) had their caches evicted before their next scheduled CI run and were always cold.

3. Dep-upgrade burst amplifies the problem.

pom.xml/plugins.sbt are touched ~5–6 times per month on average, but upgrades cluster: on 2026-05-28 alone, 5 dependency upgrades merged in a single day (rocksdbjni, joda-time, gson, Jetty, zstd-jni). Each commit rolls the hash, so 5 consecutive CI runs each start with a cold Coursier cache. Under the old design each cold run raced to create ~5 new ~1.4 GB entries (~7 GB), immediately overflowing the budget and evicting the previous run's still-warm caches. Under the new design each cold run creates exactly 1 entry (~1.4 GB), so a burst of 5 dep-upgrade commits creates ~7 GB total — still within budget and without evicting each other.

Summary: with one writer per branch the per-branch footprint drops from ~4.5 GB to ~1.4 GB, fitting ~6 branches in the 10 GB budget simultaneously, and a burst of dep-upgrade commits no longer triggers a cascade of mutual evictions.

Does this PR introduce any user-facing change?

No. CI-only.

How was this patch tested?

YAML validates with python3 -c "import yaml; yaml.safe_load(...)".

The correctness of the one-writer design relies on two GHA cache guarantees verified in prior CI runs:

1. Caches are immutable — an exact key hit skips the post-save step (Cache hit occurred on the primary key …, not saving cache), so multiple jobs using actions/cache@v5 with the same key don't produce duplicates when the cache already exists.
2. The precompile job builds with every profile (-Phadoop-3 -Pyarn -Pspark-ganglia-lgpl -Phadoop-cloud -Phive -Pkubernetes -Pjvm-profiler -Pkinesis-asl -Phive-thriftserver -Pdocker-integration-tests -Pvolcano), so its ~/.cache/coursier is a superset of every consumer job's closure.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (claude-sonnet-4-6)