Adopt the compact index for gem commands

[hsbt]

What was the end-user or developer problem that led to this PR?

To build the cooldown option (#9113) for gem commands, RubyGems needs the compact index: the per-version created_at timestamps only exist there.

What is your fix for the problem, implemented in this PR?

This adds Gem::CompactIndexClient with a disk cache (ETag conditional and ranged requests, Repr-Digest verification) and moves gem install resolution and Gem::Source#load_specs (update/outdated/list/search) onto it, falling back to the Marshal indexes when a source does not provide a compact index.

The client is an independent implementation rather than a reuse of Bundler's. Keeping them separate means this change cannot affect Bundler's behavior, which makes regressions easy to isolate while this side stabilizes; the duplication is accepted for now.

Gem::CompactIndexClient itself is a near-verbatim port of Bundler::CompactIndexClient, differing only in dropping Bundler-specific dependencies and adding a couple of small APIs the gem commands need. It also fixes a few minor bugs that still exist in Bundler's copy, which are worth fixing there separately.

Two limitations are left in on purpose.

• Gem::Source#fetch_spec (/quick Marshal gemspecs) remains for commands that need a full gemspec (gem specification -r, gem list -r -d, --development), since the compact index does not carry that data. Those paths are low-frequency and go through Gem::SafeMarshal.
• When the network is unreachable we fail exactly as before instead of silently serving a stale cached index, which would make gem update and gem outdated quietly wrong; an explicit offline mode can be discussed separately.

Make sure the following tasks are checked

• Describe the problem / feature
• Write tests for features and bug fixes
• Write code to solve the problem
• Make sure you follow the current code style and write meaningful commit messages without tags

[Copilot]

Pull request overview

This PR adopts the Compact Index protocol for RubyGems gem commands by introducing a RubyGems-native Gem::CompactIndexClient (with disk caching, ETag/ranged requests, and Repr-Digest verification) and routing Gem::Source#load_specs and gem install resolution through it, with fallback to Marshal indexes when compact index is unavailable.

Changes:

• Add Gem::CompactIndexClient (cache, parser, HTTP fetcher, updater, cache-file writer) and wire it into Gem::Source and Gem::Resolver::APISet.
• Extend resolver specs to carry created_at metadata from compact index v2 and adjust API spec behavior to avoid fetching Marshal gemspecs for install paths.
• Add/adjust tests and test helpers to stub compact-index endpoints and validate caching, redirect handling, and digest verification behavior.

Reviewed changes

Copilot reviewed 26 out of 26 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
lib/rubygems/source.rb	Prefer compact index for load_specs, add compact-index client + caching directory logic, keep Marshal fallback
lib/rubygems/resolver/specification.rb	Add created_at attribute to resolver specs
lib/rubygems/resolver/api_specification.rb	Populate created_at; build stub Gem::Specification from compact-index data
lib/rubygems/resolver/api_set.rb	Switch version fetching/parsing to use compact-index client fetch_info
lib/rubygems/compact_index_client.rb	New top-level client coordinating cache + parser
lib/rubygems/compact_index_client/cache.rb	New on-disk cache wrapper with per-process fetch deduping
lib/rubygems/compact_index_client/cache_file.rb	New safe cache-file writer with optional digest verification
lib/rubygems/compact_index_client/http_fetcher.rb	New fetcher built on Gem::RemoteFetcher with redirect handling
lib/rubygems/compact_index_client/parser.rb	New parser for names/versions/info files using existing gem-parser logic
lib/rubygems/compact_index_client/updater.rb	New updater implementing ETag + ranged requests + digest verification
Manifest.txt	Add new compact index client files to the manifest
test/rubygems/helper.rb	Add compact-index test helper endpoints and response builder
test/rubygems/test_gem_source.rb	Add compact-index coverage for Gem::Source#load_specs + caching behavior
test/rubygems/test_gem_resolver_best_set.rb	Update resolver tests to use compact-index style responses
test/rubygems/test_gem_resolver_api_set.rb	Update APISet tests to use compact-index style responses across scenarios
test/rubygems/test_gem_resolver_api_specification.rb	Add tests for created_at parsing behavior
test/rubygems/test_gem_dependency_installer.rb	Ensure install resolution uses APISet/compact index and avoids quick gemspec fetches
test/rubygems/test_gem_commands_update_command.rb	Ensure gem update exercises compact-index-backed load_specs
test/rubygems/test_gem_commands_outdated_command.rb	Ensure gem outdated exercises compact-index-backed load_specs
test/rubygems/test_gem_compact_index_stub.rb	Validate the stub helper speaks the compact-index protocol correctly
test/rubygems/test_gem_compact_index_client.rb	Unit tests for compact index client behaviors (names/versions/info/cache)
test/rubygems/test_gem_compact_index_client_updater.rb	Unit tests for updater ETag/range/digest verification logic
test/rubygems/test_gem_compact_index_client_parser.rb	Unit tests for parser (versions deletions, requirements/deps parsing, checksums)
test/rubygems/test_gem_compact_index_client_http_fetcher.rb	Unit tests for HTTP fetcher joining paths, headers, redirects, and errors
test/rubygems/test_gem_compact_index_client_cache.rb	Unit tests for cache directory creation, fetch-once semantics, and checksum gating
test/rubygems/test_gem_compact_index_client_cache_file.rb	Unit tests for cache file atomic writes/appends and digest mismatch behavior

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.