diff --git a/docs-internal/networking-parity-spec.md b/docs-internal/networking-parity-spec.md new file mode 100644 index 0000000000..861f2ee3d5 --- /dev/null +++ b/docs-internal/networking-parity-spec.md @@ -0,0 +1,219 @@ +# Networking Parity Spec — curl / wget / git full TLS + network support + +Status: spec · Owner: registry/runtime · Last updated: 2026-07-08 + +## Goal + +`curl`, `wget`, and `git` must have **full networking with real TLS/SSL, using +native Linux semantics** — not host-brokered shims. Concretely: + +- `curl https://…` and `wget https://…` verify certificates against a real CA + bundle, honor `--cacert`/`--ca-certificate`, fail with the **real exit codes** + (curl exit 60 = cert verify failure), and support `--compressed` (gzip, brotli, + zstd). +- `git clone/fetch/push` over **HTTPS smart-HTTP** works against GitHub/GitLab. +- Certificate trust comes from a **CA bundle inside the VM** (`/etc/ssl/certs/…`), + the way Debian's `ca-certificates` package + apt/curl/openssl/python all resolve + it — not from the host machine's trust store. + +## Current state (verified) + +Sockets are **already real** and are not the problem. The patched wasi-libc +sysroot implements `socket()/connect()/getaddrinfo()/send()/recv()` over +`host_net` WASM imports (`toolchain/std-patches/wasi-libc/0008-sockets.patch`, +`0023-host-net-read-write-sockets.patch`; Rust mirror `toolchain/crates/wasi-ext`). +The runner forwards them to the sidecar socket table (`crates/execution/assets/ +runners/wasm-runner.mjs`). So curl/wget/git already do their own DNS, TCP and HTTP +byte-for-byte. **Only TLS and decompression are shimmed or missing.** + +| Tool | Networking today | TLS today | Gap | +|---|---|---|---| +| **curl** | real (8.11.1) | host-brokered `wasi_tls.c` → `net_tls_connect` → sidecar rustls + **host** cert store | no real verify, `--cacert`/exit-60/`-v` chain all wrong; no gzip/brotli/zstd | +| **wget** | real (1.24.5) | **none** (`--without-ssl`) — HTTP only | no HTTPS at all; no gzip | +| **git** | real, local-only (2.55.0, `NO_CURL`) | **none** | `git-remote-https` is a dead symlink→git binary; no HTTPS clone/fetch/push | + +The current TLS shim (`software/curl/native/c/overlay/lib/vtls/wasi_tls.c`) does +**no handshake and no verification in-guest**: it hands the TCP fd to the host +(`net_tls_connect(fd, host, flags)`), which terminates TLS with rustls + +`rustls_native_certs::load_native_certs()` — **the host machine's** trust store. +`--cacert`/`--capath`/`CURL_CA_BUNDLE`/`SSL_CERT_FILE` are silently ignored; any +TLS error returns curl exit 35 instead of 60; `curl -v` cannot print the chain. + +## The decision: in-guest TLS (mbedTLS) + a VM CA bundle + +Two directions were researched. **We choose in-guest TLS** because it is the only +one that delivers the acceptance criterion (native Linux semantics + a CA bundle +that mimics apt). + +### ✅ Chosen — in-guest mbedTLS 3.6 LTS + `/etc/ssl/certs/ca-certificates.crt` + +- **mbedTLS** is pure portable C99, zero platform deps, TLS 1.3, and curl has a + first-class maintained `USE_MBEDTLS` backend in 8.11.1. Entropy = `getentropy()` + (already proven in `wasi_tls.c`); clocks/time work; build single-threaded. +- Verification happens **in curl's own code path**: `--cacert` → + `mbedtls_x509_crt_parse_file` via the VFS, `-k` → generic `verifypeer`, verify + failure → `CURLE_PEER_FAILED_VERIFICATION` = **exit 60** with the real message, + `curl -v` prints version/cipher/cert. Exactly Linux curl-with-mbedTLS. +- The sidecar becomes a **dumb ciphertext pipe** — strictly better for the trust + model (the untrusted executor no longer asks the trusted host to authenticate + servers on its behalf), and hermetic (no dependence on the host's cert store). + +### ❌ Rejected for parity — host-brokered TLS (the current `net_tls_connect` path) + +Reusing the existing bridge is minimal work (wget = ~100-line shim, git = build +with the overlaid libcurl), but it **cannot reach native semantics**: it uses the +*host's* cert store (non-hermetic), ignores `--cacert`/`--capath`, returns the +wrong exit-code taxonomy, can't print the cert chain, and every future TLS flag +(client certs, `--pinnedpubkey`, TLS-version pinning) needs a new bridge hop — a +permanent shim treadmill. It also contradicts the "real tool, patch the sysroot" +philosophy the rest of the toolchain follows. Keep the host `net.socket_upgrade_tls` +path only for the **Node/JS runtime**, which is a separate surface. + +> Note: the current sidecar TLS path uses `rustls_native_certs` = the **host +> machine's** trust store (`crates/native-sidecar/src/execution.rs`). That is a +> latent hermeticity bug even for the JS runtime — it should read the VM's +> `/etc/ssl` bundle instead. Tracked here; fix alongside. + +## CA bundle — ship Debian-shaped trust inside the VM + +- Ship the **Mozilla CA bundle** (`curl.se/ca/cacert.pem`, i.e. the set Debian's + `ca-certificates` produces) at **`/etc/ssl/certs/ca-certificates.crt`**, with the + conventional `/etc/ssl/cert.pem` symlink. A `ca-certificates` registry package + owns the payload; VM bootstrap links it into the standard tree (the bootstrap + already seeds `/etc` in the shadow root — `crates/native-sidecar/src/vm.rs`). +- This one file at that one path is what makes the **whole class** of TLS tools + "just work": curl's compile-time `CURL_CA_BUNDLE` default, OpenSSL's `OPENSSLDIR` + (`/usr/lib/ssl` → `/etc/ssl/certs`), apt, python, wget — all resolve there on + Debian. `SSL_CERT_FILE`/`SSL_CERT_DIR`/`--cacert` env overrides then work for + free once the backend does real verification. + +## Per-tool plans + +### curl +1. Vendor + build **mbedTLS 3.6** for `wasm32-wasip1` in `toolchain/c/Makefile` + (same pattern as the existing zlib target). +2. In `toolchain/c/scripts/build-curl-upstream.sh`: drop `--without-ssl` and the + `USE_WASI_TLS` injection; add `--with-mbedtls`, `--with-zlib`, `--with-brotli`, + `--with-zstd`, `--with-ca-bundle=/etc/ssl/certs/ca-certificates.crt`; retire the + `wasi_tls.c/h` + `vtls.c` overlay. +3. Tests: assert **exit 60** + real verify message on self-signed without `-k`; + `--cacert` acceptance of a test CA; `--compressed` gzip/br/zstd round-trips. + +### wget +GNU Wget has no mbedTLS backend, so give it a real TLS backend against the same +mbedTLS + CA bundle (its SSL abstraction is just 4 functions in `src/ssl.h`: +`ssl_init`, `ssl_cleanup`, `ssl_connect_wget`, `ssl_check_certificate`). +1. Add `wasi_ssl.c` implementing those 4 functions over **mbedTLS** (handshake + + `mbedtls_x509_crt_parse_file` on `/etc/ssl/certs/ca-certificates.crt`; + `--no-check-certificate`/`--ca-certificate`/`--ca-directory` map to real + mbedTLS verify config, matching Linux wget). Overlay dir mirrors curl's + (`software/wget/native/c/overlay/…`). +2. In `build-wget-upstream.sh`: keep `./configure --without-ssl` (so it doesn't + probe GnuTLS/OpenSSL), then post-configure patch `src/wget.h`'s `HAVE_SSL` + condition to add `|| defined HAVE_WASI_TLS`, pass `-DHAVE_WASI_TLS`, copy the + overlay, append a `wasi_ssl.o` compile/link rule to the generated `src/Makefile` + (the exact playbook the curl script uses). This lights up the `https` scheme, + all `--secure-protocol/--ca-certificate/--no-check-certificate` options, and + HSTS. +3. Enable **zlib** (`--with-zlib` via `ZLIB_CFLAGS/ZLIB_LIBS` at the existing + `build/zlib/libz.a`). Leave `--without-libpsl` (cosmetic — cookie public-suffix + only; wget's built-in heuristic covers single-URL HTTPS). +4. Tests: mirror curl's HTTPS cases (verify-fail on self-signed, + `--no-check-certificate` success, keep-alive), + gzip via `--compression`. + +### git — HTTPS smart-HTTP (clone/fetch/**push**) +Git's HTTP transport lives entirely in the **`git-remote-https` remote helper**, +which links **libcurl in-process** (git never shells out to a `curl` binary). The +current symlink `git-remote-https → git` is structurally broken (`git`'s +`cmd_main` dies "cannot handle remote-https as a builtin"). +1. **Produce a reusable libcurl** artifact (`make -C lib install` the overlaid, + mbedTLS-linked libcurl into a prefix under `toolchain/c/build/curl-upstream/ + install`). +2. In `toolchain/c/scripts/build-git-upstream.sh`: drop `NO_CURL=1`; add + `CURL_CFLAGS/CURL_LDFLAGS` pointing at that libcurl (defining them skips + `curl-config`); keep `NO_EXPAT=1`; build targets **`git git-remote-http`**; + wasm-opt both. (Expect a handful of small WASI compile fixes in + `http.c`/`remote-curl.c`, same class the curl script already makes; the + existing `git_compat.c` shims cover most.) +3. **Packaging:** install `git-remote-http` as a **real second command** (replace + the wrong symlink loop in `toolchain/c/Makefile`); in + `software/git/agentos-package.json` move `git-remote-http` into `commands` and + re-point `git-remote-https` → `git-remote-http` (keep `git-upload-pack`/ + `git-receive-pack`/`git-upload-archive` → `git`; those *are* builtins). +4. **Push** is the same libcurl POST plumbing as fetch (`remote-curl.c` spawns + `git send-pack` and streams the pack via chunked `Transfer-Encoding` + + `Expect: 100-continue` — all inside libcurl, no new host surface). HTTP Basic + auth (token/`user:pass` URLs, `GIT_ASKPASS`) works. (`credential-cache` + unavailable — `NO_UNIX_SOCKETS`; NTLM/Negotiate unsupported — irrelevant for + GitHub/GitLab.) +5. **Nothing new on the host side** — TLS/DNS/TCP/spawn/permission-tiers are all + in place. Spawn already resolves `git-remote-https` by name on the guest PATH + via the `proc_spawn` broker (proven: `git clone` already spawns + `git upload-pack` locally). Permission tiers already grant `git-remote-http(s)` + tier `full`. +6. **ssh transport (secondary):** `git@host:` spawns `ssh` — needs an in-guest + `ssh` command (its own OpenSSH port, needs a real in-guest crypto stack, unlike + host-brokered TLS). HTTPS stays primary; ssh is a separate future project. +7. Tests: flip `hasGitHttpHelper` in `software/git/test/git.test.ts` (the + smart-HTTP clone/fetch suite is already written but skipped); add a push case + (small + >1 MiB chunked POST). + +## Compression (curl) + +- **zlib 1.3.1 is already vendored** and built against this sysroot (git links it): + `--with-zlib` via `CPPFLAGS/LDFLAGS` at `build/zlib`. +- **brotli** — decoder (`common`+`dec`) is dependency-free portable C; add a + Makefile fetch/build target mirroring zlib; `--with-brotli` (decode only). +- **zstd** — already compiles under this sysroot inside the duckdb build + (`toolchain/c/build/duckdb-cmake/third_party/zstd/`); build libzstd + single-threaded; `--with-zstd`. +- Out of scope here: nghttp2 (HTTP/2), libpsl. + +## Refuted / dead-end theses (documented so we don't re-try them) + +- **Host-brokered TLS for parity** — works but never reaches native semantics + (host cert store, ignored `--cacert`, wrong exit codes, no `-v` chain, shim + treadmill). Rejected as the parity path; kept only for the JS runtime. +- **OpenSSL in-guest** — its config/ENGINE/provider machinery expects `dlopen`, + threads, platform RNG; wasip1 builds are fragile forks. The curl build already + defines `CURL_DISABLE_OPENSSL_AUTO_LOAD_CONFIG` just to dodge it. Unnecessary. +- **GnuTLS in-guest** (wget's default) — drags nettle → GMP (asm) → libtasn1 → + p11-kit (dlopen). Non-starter on WASI. +- **rustls + aws-lc-rs in guest** — aws-lc-rs is C+asm, does **not** build for + wasm32-wasip1. `ring` support is unofficial/spotty. Mixing a Rust staticlib into + the clang-LTO curl link is fiddly. No fidelity gain over mbedTLS. +- **rustls-native-certs (as the trust story)** — needs an OS cert store; the guest + has none, and host-side it depends on the host machine's store (non-hermetic). +- **BearSSL** — curl **removed** BearSSL support in 8.12.0 and it lacks TLS 1.3. + Dead end for any curl upgrade. +- **Reusing curl's `wasi_tls.c` for wget** — impossible; it's written against + libcurl-internal `Curl_cfilter` APIs. The shareable unit is the **host import + contract / the mbedTLS backend**, not the C file. +- **git: symlink `git-remote-https` → git binary** (current state) — git's + `cmd_main` hard-dies for a non-builtin `git-*` argv[0]. Structurally impossible. +- **git: spawn the `curl` CLI from a shell helper** — the remote helper is a + long-lived, stateful, bidirectional protocol-v2 program (auth retry, gzip, + full-duplex chunked POST). The CLI can't provide it. +- **git: gitoxide / libgit2 / reimplement smart protocol in-process** — not real + upstream git; diverges on CLI/config/hooks; would need its own WASI net port + anyway. Rejected. +- **git: dumb-HTTP only** — GitHub/GitLab/Gitea serve smart-HTTP only for fetch + and never support dumb push. Non-starter (dumb fetch rides along free). +- **Anything threaded** — the runner is single-threaded wasip1; all TLS/compression + libs build single-threaded (mbedTLS/zstd support it). + +## Work breakdown & sequencing + +1. **mbedTLS 3.6** target in `toolchain/c/Makefile` (shared by curl, wget, git's + libcurl). ← foundational, do first. +2. **CA bundle**: `ca-certificates` payload at `/etc/ssl/certs/ca-certificates.crt` + + VM bootstrap seeding + sidecar reads the VM bundle (not the host store). ← + foundational. +3. **curl**: mbedTLS + zlib/brotli/zstd + CA bundle; retire `wasi_tls.c`; tests. +4. **libcurl install artifact** (from step 3's build) for git to link. +5. **git**: build-with-curl, real `git-remote-http` helper, packaging, push; tests. +6. **wget**: `wasi_ssl.c` mbedTLS backend + zlib + CA bundle; tests. + +Steps 3/5/6 all sit on 1+2. One jj rev per tool. Real e2e tests against a TLS +server (verify-fail on self-signed, `--cacert` success, real clone/fetch/push) are +the deliverable — no host-store shortcuts. diff --git a/docs-internal/registry-parity-worklist.md b/docs-internal/registry-parity-worklist.md index 02ac4b61a9..9d38bdd5d3 100644 --- a/docs-internal/registry-parity-worklist.md +++ b/docs-internal/registry-parity-worklist.md @@ -119,6 +119,17 @@ external build (tracked separately in #9). **Objective:** replace each ❌ with a real/established implementation built to `wasm32-wasip1` and patched only where WASI forces it. The ✅ rows stay. +**⚠️ Networking is a separate, still-open gap (curl/wget/git BUILD but lack real +TLS).** The tools compile and do real HTTP, but: **curl** uses a host-brokered TLS +shim (no real cert verify, ignores `--cacert`, wrong exit codes) and lacks +gzip/brotli/zstd; **wget** is HTTP-only (`--without-ssl`); **git** is `NO_CURL` +(no HTTPS clone/fetch/push — `git-remote-https` is a dead symlink). Full plan — +in-guest **mbedTLS** + a Debian-shaped **CA bundle at `/etc/ssl/certs/ +ca-certificates.crt`** + real `git-remote-http` helper, with all refuted theses — +is in **`docs-internal/networking-parity-spec.md`**. Acceptance: `curl`/`wget`/`git` +do real HTTPS (verify, `--cacert`, exit 60, clone/fetch/push) with native Linux +semantics. + **Approach:** one command at a time, one jj rev each: swap our custom code for the established source (fetched + pinned like sqlite/duckdb), wire into the toolchain, patch for WASI, prove parity with real e2e tests.