[anth-volk]

Fixes #199

Incident

On 2026-10-20 it was disclosed that eight eMudhra (emSign) intermediate CA certificates were issued with private keys anyone can recover from the public certificate, so anyone can issue a certificate for any domain that chains to "emSign ECC Root CA - G3" (and, through cross-signatures, to Sectigo's "AAA Certificate Services"). Browsers and current operating systems already block them; what stays exposed is software that carries its own CA list or reads an operating-system CA store that has not been updated. Details: https://emudhra.we-have-a-problem.com

What in this repository trusted the affected roots

The hosted MCP server (Modal app policyengine-macro-mcp, built from integration/modal_app.py) makes outbound HTTPS requests from its containers: Hugging Face microdata downloads for population_reform_impact, and PolicyEngine data fetches. Its image carries two CA lists, and both trusted the compromised root:

• certifi 2026.7.22 (used by requests and httpx). The pip_install layer was last built on 2026-08-12 (deploy run 31608860051: Successfully installed ... certifi-2026.7.22 ... hf-xet-1.6.0 ... requests-2.34.2). Every deploy since has reused that cached layer: the 2026-10-08 deploy (run 37774203311) reports Requirement already satisfied: certifi in ./usr/local/lib/python3.12/site-packages ... (2026.7.22), and the 2026-10-12 deploy built no layers at all. The fixed release is certifi 2026.10.19.
• Debian bookworm ca-certificates (used by urllib, aiohttp, and hf-xet 1.6.0, which reads the operating-system store). Modal's debian_slim base is bookworm (the apt output in the same deploy logs shows bookworm, bookworm-updates, bookworm-security), and no step in the image refreshes ca-certificates. The layers above it were built on 2026-08-12, before Debian published the fixed 20250419~deb12u2 to bookworm-security.

No grpcio is installed in the image (not in the 2026-08-12 install list), and no code in the repository overrides TLS verification or sets a custom CA bundle.

Not affected:

• The Vercel project macromod serves this repository as a static site (no api/ directory, no package.json; the latest production deployment has a single static build and no functions), so there is no server-side code on Vercel.
• The GitHub Actions workflows run on GitHub-hosted runners and install unpinned packages fresh on every run; astral-sh/setup-uv@v8.3.2 has no version input, so it installs the latest uv.

Change

integration/modal_app.py: two layers appended at the end of the shared image chain, after the editable installs:

• .apt_install("ca-certificates"), which Modal runs as apt-get update then apt-get install -y ca-certificates, upgrading the installed package to 20250419~deb12u2 from bookworm-security.
• .pip_install("certifi>=2026.10.19"), which upgrades certifi in place (certifi has no dependencies).

The new steps go at the end on purpose. Modal caches each layer by its definition, so appending leaves every earlier layer cached and only builds the two new ones. Adding certifi to the existing pip_install list would have rebuilt that layer, and that rebuild fails today: policyengine-us==1.764.6 is no longer on PyPI (neither the JSON API nor the simple index lists any 1.764.x file). The same applies to a forced rebuild (MODAL_FORCE_BUILD=1), so a forced rebuild of the existing image definition cannot fix this either.

Checks

• Reproduced the image layers in Docker (python:3.12-slim-bookworm), starting from the unfixed state (ca-certificates 20230311+deb12u1, certifi 2026.7.22), then running the two new steps exactly as Modal does:

  Host	Before: requests + certifi	Before: urllib + OS store	After: requests + certifi	After: urllib + OS store
  dv-g3b-emudhra.we-have-a-problem.com (emSign chain)	HTTP 200	HTTP 200	certificate error	certificate error
  dv-g3b-aaa-emudhra.we-have-a-problem.com (Sectigo cross-signed chain)	certificate error	HTTP 200	certificate error	certificate error
  pypi.org	HTTP 200	HTTP 200	HTTP 200	HTTP 200
  huggingface.co	HTTP 200	HTTP 200	HTTP 200	HTTP 200

  The apt step reported 1 upgraded (ca-certificates 20250419~deb12u2 from bookworm-security) and the pip step Successfully installed certifi-2026.10.19.

• integration/modal_app.py imports with modal==1.6.1 and POLICYENGINE_MACRO_IMAGE_SOURCE=github (the CI configuration), registering serve, run_tool_job and check_overlay_gate. No image was built or deployed.

• python3 site_contract.py: passes (it reads the modal_app.py docstring tool count, which is unchanged).

• pytest integration/tests/test_source_pins.py: 2 passed. pytest tests/test_repo_hygiene.py: 8 passed.

• ruff check --select E,F,W on the changed file: passes.

Operator actions after merge

1. Merging to main changes integration/**, so deploy-mcp.yml runs modal deploy integration/modal_app.py automatically. In that run's Deploy step, confirm that two new layers build (RUN apt-get install -y ca-certificates showing 20250419~deb12u2, and Successfully installed certifi-2026.10.19).
2. The Smoke-test the deployed server step of deploy-mcp.yml has failed on every run since 2026-09-22 while the Deploy step succeeded (on 2026-10-12, test_remote_mcp.py::test_obr_shock_cgg failed with Redirect to https://policyengine--policyengine-macro-mcp-serve.modal.run/mcp not followed). That failure is unrelated to this change; judge the deploy by the Deploy step.

Separate from this incident: because policyengine-us==1.764.6 is no longer on PyPI, any future change to the pip_install list (or a forced rebuild) will fail until that pin is moved to a version that exists and matches the policyengine data-release manifest.

Environment-variable verification

This PR adds no environment variables. The deploy of the changed image definition depends on these, all traced end to end:

Variable	Provider	Wiring	Status
MODAL_TOKEN_ID, MODAL_TOKEN_SECRET	GitHub Actions repository secrets (also defined at organization level; the repository secrets take precedence)	deploy-mcp.yml Deploy step env: from secrets.*	Verified: secret names exist; deploys on 2026-10-08 and 2026-10-12 authenticated and deployed
POLICYENGINE_MACRO_IMAGE_SOURCE	Literal github in the workflow	Deploy step env:; read by modal_app.py at definition time	Verified
HF_HOME, POLICYENGINE_MACRO_PE_DATA_DIR	Literals in modal_app.py	image.env({...}) on each function's image	Verified
POLICYENGINE_MACRO_SOURCE_REVISION_*	Constants in modal_app.py	.env(SOURCE_REVISION_ENV) image layer	Verified
HUGGING_FACE_TOKEN / HF_TOKEN	Modal secret macromod-hf in workspace policyengine	secrets=[modal.Secret.from_name("macromod-hf")] on all three functions	Secret exists and was last used 2026-10-22; which key name it holds was not checked
POLICYENGINE_MACRO_REMOTE_TESTS	Literal "1" in the workflow	Smoke-test step env:	Verified

🤖 Generated with Claude Code

[vercel]

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated
macromod	Ready	Preview	Oct 22, 2026 6:09pm UTC