[JuArce]

Why

The private keys of eight eMudhra intermediate CAs were compromised (advisory, Mozilla bug 2083449). They chain to two roots:

• emSign ECC Root CA - G3, through the emSign Root TLS CA - G3 cross-sign
• AAA Certificate Services, through Sectigo cross-signs

The servers' ca-certificates upgrade fixes clients that use the OS trust store. It doesn't fix clients that ship their own root list.

Explorer (exposed now)

EthConverter (the CoinGecko price) and OperatorVersionTracker use HTTPoison. HTTPoison goes through hackney, which always verifies against the roots bundled in Erlang certifi, never the OS store. certifi 2.12.0 has both roots, and there is no fixed release yet (2.17.0 is the latest). I tested against the advisory's no-CT test hosts with the patched Debian bookworm store (ca-certificates 20250419~deb12u2) loaded as the OS store:

Host	HTTPoison (before)	Finch (after)
dv-g3b-noct-emudhra (emSign chain)	trusted	rejected, unknown_ca
dv-g3b-aaa-noct-emudhra (AAA chain)	trusted	rejected, unknown_ca
api.coingecko.com	ok	ok
wrong.host.badssl.com / expired.badssl.com	rejected	rejected

Aggregation mode (defense in depth)

alloy's connect_http builds reqwest::Client::new(). reqwest's default-tls feature is enabled here, so that client uses OpenSSL and the OS store, and the RPC traffic was covered by the server upgrade. The bundled webpki-roots 1.0.4 still trusts the emSign chain for anything that selects rustls. I confirmed that with .use_rustls_tls() against the test hosts.

How

• explorer: EthConverter and OperatorVersionTracker move from HTTPoison to Explorer.Finch, which the rest of the explorer already uses. Mint verifies against :public_key.cacerts_get(), the OS store. httpoison is dropped, which also removes hackney, certifi and their exclusive deps from the lockfile. castore, Mint's fallback bundle, goes from 1.0.8 to 1.0.22, which has neither root.
• aggregation_mode/Cargo.lock: webpki-roots 1.0.4 → 1.0.10 and rustls-webpki 0.103.8 → 0.103.16. 0.103.16 requires aws-lc-rs ^1.18, so aws-lc-rs 1.18.2 / aws-lc-sys 0.46.0 come along.

After merge

1. Every host: check the OS trust store

This step doesn't depend on this PR, so it can run now.

dpkg -l ca-certificates | tail -1                                  # Debian 12: 20250419~deb12u2
ls /etc/ssl/certs | grep -iE 'emSign_ECC_Root_CA_-_G3|Comodo_AAA'  # must print nothing

If the second command prints anything, remove the roots by hand as admin. This happens on Debian 11, which has no patched package.

sudo sed -i -e 's|^mozilla/emSign_ECC_Root_CA_-_G3.crt|!&|' -e 's|^mozilla/Comodo_AAA_Services_root.crt|!&|' /etc/ca-certificates.conf && sudo update-ca-certificates

OpenSSL and OTP load the trust store into memory at startup. Restart any long-running process that started before the store changed:

grep ' ca-certificates ' /var/log/dpkg.log | tail -1
ps -o lstart=,cmd= -p "$(pgrep -d, -f 'gateway|poller|explorer|proof_aggregator')"

2. Explorer

Repeat on each explorer host. The inventory lists aligned-mainnet-explorer and aligned-hoodi-explorer.

Don't rerun infra/ansible/playbooks/explorer.yaml:

• It checks out the old 2075-feat-hoodi-deployment branch.
• It re-runs the whole host setup.
• It ends with state: started, which doesn't restart a service that's already running.

These steps repeat only its build task.

As app:

cd ~/repos/explorer/aligned_layer
git status --short                      # explorer/.env is gitignored; stash anything else
git rev-parse HEAD > ~/explorer.prev    # for rollback
git fetch origin && git checkout testnet && git pull --ff-only
cd explorer
source .env && export MIX_ENV=prod      # .env doesn't set MIX_ENV; TRACKER_API_URL is read at compile time
mix deps.get --only prod
mix compile
pnpm --prefix=assets/ install
mix phx.digest
mix assets.deploy
mix release --overwrite
mix ecto.migrate
ls _build/prod/rel/explorer/lib | grep -E '^(hackney|certifi|httpoison|castore)-'   # expect only castore-1.0.22

As admin, re-grant the permission to bind port 443. mix release --overwrite replaces beam.smp, which drops it:

B=/home/app/repos/explorer/aligned_layer/explorer/_build/prod/rel/explorer
sudo setcap CAP_NET_BIND_SERVICE=+eip $B/erts-*/bin/beam.smp && getcap $B/erts-*/bin/beam.smp

As app, restart:

systemctl --user restart explorer
systemctl --user status explorer --no-pager
journalctl --user -u explorer -n 50 --no-pager

Verify inside the running node, which uses the trust store it loaded at startup:

cd ~/repos/explorer/aligned_layer/explorer
_build/prod/rel/explorer/bin/explorer rpc '
  IO.inspect(Finch.build(:get, "https://dv-g3b-noct-emudhra.we-have-a-problem.com/") |> Finch.request(Explorer.Finch), label: "compromised chain");
  Cachex.del(:eth_price_cache, :eth_price);
  IO.inspect(EthConverter.get_eth_price_usd(), label: "eth price")'

• Expected: compromised chain: {:error, %Mint.TransportError{reason: {:tls_alert, {:unknown_ca, ...}}}} and eth price: {:ok, <number>}.
• If the compromised chain returns {:ok, ...}: the host's OS store still trusts the roots. Go back to step 1.

Then open the explorer and check that the operators page and the USD amounts load.

Rollback: run git checkout $(cat ~/explorer.prev), then redo the build, the setcap and the restart.

3. Aggregation mode

This can wait for the next normal deploy, because RPC traffic already uses the OS store.

• Rebuild and redeploy the gateway, payments poller and proof aggregator.
• Run make agg_mode_install_cli on the task sender host.

Testing

• mix compile in the explorer: no warnings. An end-to-end run of the real modules:
  • EthConverter.get_eth_price_usd() returns {:ok, price} through Finch.
  • OperatorVersionTracker, pointed at the compromised test host, logs unknown_ca and returns %{}.
• SKIP_AGG_PROGRAMS_BUILD=1 cargo check --locked --workspace in aggregation_mode.

🤖 Generated with Claude Code

[claude]

Review: no blocking issues found.

• Lockfile bumps (rustls-webpki 0.103.16, webpki-roots 1.0.10, aws-lc-rs/sys) look consistent.
• The vendored webpki-roots 0.25.4 patch is scoped to the crates/ workspace via [patch.crates-io] and removes the two anchors listed in its README. It is excluded from the workspace members.
• Low: emSign Root CA - C1 and G1 (eMudhra) are still in the vendored bundle. I couldn't verify that only the two removed roots chain to the compromised intermediates. Please confirm against the Mozilla bug, and note that webpki-roots 1.0.10 still ships them if that's intended.
• Low: the vendored copy is only the src/lib.rs with entries deleted, so a CI check that diffs it against upstream 0.25.4 would help catch drift.

[github-actions]

Codex Code Review

No actionable security vulnerabilities, bugs, significant performance issues, or unnecessary complexity found in the PR diff.

Verified the vendored bundle excludes both specified roots and its manifest loads successfully. Full builds were not run.