feat(sandbox,gateway): route sandbox egress through corporate HTTP proxy

[feloy]

Summary

• Route sandbox TLS egress through a corporate forward proxy so sandboxes work in proxy-required enterprise networks: the supervisor's policy proxy evaluates SSRF/allowlist rules as before, then chains approved CONNECT tunnels through the corporate proxy instead of dialing destinations directly. Plain-HTTP requests are never proxied and always dial directly (forwarding them would need absolute-form requests rather than CONNECT tunneling; deliberately out of scope).
• CONNECT requests target a validated resolved IP by default, so the corporate proxy performs no DNS resolution and the tunnel stays bound to the address that passed SSRF and allowed_ips validation; the hostname still travels inside the tunnel (TLS SNI, application Host). An explicit proxy_connect_by_hostname opt-in exists for proxies whose ACLs filter on hostnames.
• The proxy configuration is an operator-owned egress boundary: the Podman driver delivers proxy settings on the supervisor argv (not container environment), so sandbox/template environment cannot override them and the conventional HTTPS_PROXY/NO_PROXY variables a sandbox sets have no effect. Reserved variable names are stripped from workload child processes.
• Proxy credentials are delivered via a root-only Podman secret mount (proxy_auth_file), never through the environment, container metadata, or the proxy URL. Because Proxy-Authorization: Basic travels over plain TCP to the http:// proxy, sending credentials additionally requires the explicit proxy_auth_allow_insecure = true acknowledgement.
• Configuration is fail-closed end to end: any present-but-invalid setting (empty value, malformed/unsupported URL, scheme-less or port-less URL, URL with path/query/fragment or inline user:pass@, unreadable or malformed credential, credentials without the insecure-auth acknowledgement, port 0, or any auxiliary setting without a proxy) is fatal at gateway startup or sandbox startup instead of silently degrading to direct or unauthenticated egress. Driver and supervisor share single URL/credential validators in openshell-core, so a value accepted at sandbox-create time can never be rejected in-container or vice versa.
• Proxy-auth file reads are bounded (64 KiB hard limit), and the file is opened non-blocking (O_NONBLOCK) to reject FIFOs and other blocking special files promptly.

Related Issue

Part of #1792

Changes

• openshell-supervisor-network: new upstream_proxy module — supervisor-argv parsing, NO_PROXY matching, CONNECT handshake with optional Proxy-Authorization: Basic from the auth file.
  • decision(host, port, resolved) implements port-aware, resolution-aware NO_PROXY: entries take an optional :port qualifier that limits them to that destination port, and IP/CIDR entries also match hostnames through their validated resolved addresses, with the direct dial restricted to the addresses the entry contains.
  • connect_via takes a ConnectTarget (validated IP by default, hostname on operator opt-in) and returns a PrefixedStream that replays any tunneled bytes received in the same read as the CONNECT response instead of discarding them.
  • CONNECT fallback across validated addresses: when the corporate proxy rejects a CONNECT attempt (non-200 status), the supervisor tries the next validated address for the destination instead of failing immediately; each attempt is capped within the shared per-connection timeout budget.
  • Parses bracketed IPv6 authorities in client CONNECT targets (e.g. [::1]:443).
  • Proxy startup emits OCSF ConfigStateChange events and refuses to start on invalid configuration.
• openshell-core: shared parse_upstream_proxy_url (strict http://host:port grammar — explicit scheme and port required, rejects port 0, rejects bracketed IPv6 with empty port) and parse_upstream_proxy_credential validators (bounded reads, non-blocking open to reject FIFOs); credential errors never carry credential content. Reserved sandbox_env names, including OPENSHELL_UPSTREAM_PROXY_AUTH_ALLOW_INSECURE and OPENSHELL_UPSTREAM_PROXY_CONNECT_BY_HOSTNAME, and the fixed secret mount path.
• openshell-driver-podman: PodmanComputeConfig gains https_proxy, no_proxy, proxy_auth_file, proxy_auth_allow_insecure, and proxy_connect_by_hostname with fail-closed validation mirrored by the supervisor; container creation delivers proxy config on the supervisor argv (not container environment) and stages the credential as a per-sandbox root-only secret (cleaned up with the sandbox).
• openshell-sandbox: main entry point parses proxy settings from supervisor argv and passes them to the network supervisor.
• openshell-supervisor-process: reserved proxy variables added to the supervisor-only strip list so workload children never inherit them.
• tasks/scripts/gateway.sh: dev gateway writes the proxy settings into gateway.toml with TOML-escaped values (quotes, backslashes, control characters cannot corrupt the config or inject keys); boolean flags are written only as genuine TOML booleans, with anything else emitted as a string so the gateway rejects it at startup. Set-but-empty values are preserved so validation rejects them instead of silently dropping them.
• docs/reference/gateway-config.mdx: documents the proxy fields, the strict URL grammar, the NO_PROXY semantics, the CONNECT binding and its hostname opt-out, the cleartext-credential acknowledgement, and the fail-closed contract.
• docs/reference/sandbox-compute-drivers.mdx: lists corporate proxy keys in the Podman compute-driver overview.
• architecture/sandbox.md: corporate proxy data flow, trust boundary, CONNECT-target binding, argv transport, and fail-closed invariants.

Commits

• cc0405c feat(sandbox,gateway): route sandbox egress through corporate HTTP proxy
• 3c43b52 fix(sandbox,podman): make corporate proxy routing operator-owned
• fa0082c feat(sandbox,podman): deliver corporate proxy credentials via secret file
• a82cf26 fix(sandbox,podman): fail closed on invalid upstream proxy configuration
• 1a9bcb0 fix(sandbox,podman): finish the fail-closed upstream proxy credential contract
• 2990c6c fix(sandbox,podman): close remaining fail-open upstream proxy config paths
• 1394b24 fix(sandbox,podman): reject upstream proxy URLs with path, query, or fragment
• c6a92fa fix(sandbox,podman): remove plain-HTTP upstream proxy support
• 1f19d26 fix(sandbox,podman): escape generated TOML and require explicit proxy URL form
• 4389400 fix(sandbox,podman): preserve tunneled bytes read with the CONNECT response
• a1c8b99 fix(sandbox,podman): gate cleartext proxy Basic auth behind an explicit opt-in
• f86d73c fix(sandbox,podman): honor port qualifiers and resolved addresses in NO_PROXY
• 5617bcf fix(sandbox,podman): bind proxied CONNECT tunnels to validated addresses
• 32930c2 fix(sandbox,podman): reject empty port after bracketed IPv6 proxy host
• cdf5d89 fix(sandbox,podman): fall back across validated addresses in proxied CONNECT
• 707a50f fix(sandbox,podman): strip new reserved proxy vars and complete docs/tests
• 2768c08 fix(sandbox,podman): cap each proxied CONNECT attempt within the shared budget
• fcc2651 fix(sandbox,podman): deliver corporate proxy config on the supervisor argv
• 5be9905 docs(sandbox): align proxy comments with the argv transport
• 3192ee8 fix(sandbox): parse bracketed IPv6 authorities in client CONNECT targets
• 19f2369 test(podman): cover proxy-auth secret cleanup across lifecycle failures
• 80e4ac3 docs: list corporate proxy keys in the Podman compute-driver overview
• c8434ad test(sandbox): cover the SSRF-to-TLS composition across the proxy tunnel
• 4524ac9 fix(sandbox,podman): bound proxy-auth reads, reject port 0, fix stale comment
• 1f9fc5b fix(sandbox): open proxy-auth file non-blocking to reject FIFOs promptly

Testing

• Unit tests for reserved-variable parsing and fail-closed rejection: empty values, bad schemes, missing scheme/port, URL components, inline credentials, malformed or unreadable auth files, credentials without the insecure-auth acknowledgement, invalid acknowledgement values, port 0, bracketed IPv6 with empty port, and auxiliary settings without a proxy

• Unit tests for NO_PROXY matching: wildcards, domains, CIDR, loopback bypass, port-qualified entries on domain/IP/CIDR patterns, invalid port qualifiers, resolved-address matching, and split-resolution subset dialing

• Unit tests for the CONNECT handshake: success, auth header, non-200 rejection, malformed response, timeout, IPv6 bracketing in both target modes, validated-IP request lines with hostname non-leakage, hostname opt-in, preservation of tunneled bytes read with the CONNECT response, fallback across validated addresses, and per-attempt timeout budgeting

• Unit tests for the shared URL/credential validators in openshell-core (including bounded reads, non-blocking FIFO rejection) and for Podman config validation, container-spec argv injection/env stripping, and proxy-auth secret cleanup across lifecycle failures; TOML escaping verified against a TOML parser with hostile values

• Integration test covering the full SSRF-to-TLS composition across the proxy tunnel

• Existing proxy and SSRF test suites pass — policy evaluation is unchanged

• mise run pre-commit passes

• Unit tests added/updated

• E2E tests added/updated (deferred pending gator re-review and /ok to test)

Checklist

• Follows Conventional Commits
• Commits are signed off (DCO)
• Architecture docs updated (if applicable)

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[johntmyers]

gator-agent

PR Review Status

Validation: This is project-valid work for the enterprise restricted-egress gap in #1792. Maintainer triage confirmed the need for corporate proxy chaining after OpenShell policy enforcement, and the PR includes the relevant Podman configuration plus Fern and architecture documentation.

Head SHA: ca5c041fca5547c2f659cf6d919db0ba8aaf39b7

Review findings:

• Blocking — bind SSRF validation to the destination actually reached by the corporate proxy (crates/openshell-supervisor-network/src/proxy.rs:2818, upstream_proxy.rs:364). OpenShell validates locally resolved addresses but sends the hostname in CONNECT, allowing proxy-side DNS to resolve to a different private, metadata, or control-plane address. Use a selected validated IP for the CONNECT authority while retaining the hostname for HTTP Host/TLS SNI, or explicitly establish the corporate proxy as the SSRF enforcement boundary.
• Blocking — keep proxy routing operator-owned (crates/openshell-driver-podman/src/container.rs:362, :394; upstream_proxy.rs:357). Sandbox/template environment currently wins over operator values, so a sandbox creator can choose an arbitrary proxy or set NO_PROXY=*. Use reserved supervisor-only variables written after user environment, and do not let ordinary HTTP_PROXY/HTTPS_PROXY/ALL_PROXY/NO_PROXY variants control this boundary.
• Blocking — fail closed on malformed configured proxies (config.rs:211, upstream_proxy.rs:175, :229). Some invalid values pass Podman validation and are then ignored, silently restoring direct egress. Parse once into a typed configuration and reject startup when configured proxy data is invalid.
• Blocking — preserve bytes read past the CONNECT response headers (upstream_proxy.rs:373-412). A coalesced 200 response and server-first payload currently drops the tunneled bytes. Consume exactly the header or return a buffered stream that replays over-read data.
• Required follow-up — implement conventional plain-HTTP forwarding or narrow the advertised scope (proxy.rs:3159, :2818). The current HTTP_PROXY path still uses CONNECT and sends origin-form requests; many forward proxies expect absolute-form requests and deny CONNECT to port 80.
• Required follow-up — reject control characters in decoded proxy credentials (upstream_proxy.rs:254, :365-369) to prevent HTTP header injection.
• Required follow-up — apply the documented host-gateway bypass to HTTPS CONNECT paths too (proxy.rs:1102, :4396; architecture/sandbox.md:84).

Tests should cover the SSRF/DNS binding, CONNECT over-read, invalid-config fail-closed behavior, plain HTTP against a standard forward proxy, combined Podman environment precedence, and HTTPS host-gateway bypass. Runtime proxy behavior also requires test:e2e once these review findings are resolved; E2E has not been authorized on this head because author changes are needed first.

Docs: The existing Fern gateway reference is the correct page and no docs/index.yml navigation change is needed. The architecture claim about all host-gateway aliases bypassing the proxy must match the implementation. Kubernetes propagation remains outside this PR and should stay explicit.

@feloy, please push an updated commit addressing the items above. Gator will re-review the new head before starting the E2E gate.

Next state: gator:in-review

[johntmyers]

gator-agent

PR Review Status

Validation: This remains project-valid work for the corporate restricted-egress gap in #1792. The current scope covers the supervisor proxy path and Podman operator configuration, with the relevant Fern and architecture documentation present.

Head SHA: 7a1f8f3a38a282362f55c326411b9b2e3adcc4ad

The fresh independent code review found these blocking items on the current head:

• Bind SSRF and allowed_ips validation to the destination the corporate proxy actually reaches (crates/openshell-supervisor-network/src/proxy.rs:2800-2824, upstream_proxy.rs:322-325). The code validates local DNS results, then sends the hostname in CONNECT and lets proxy-side DNS choose a potentially different private or control-plane address. Use a selected validated IP in the CONNECT authority while retaining the hostname for TLS SNI/HTTP Host, or require an explicit mode that delegates this security boundary to the corporate proxy.
• Keep upstream routing operator-owned (crates/openshell-driver-podman/src/container.rs:362-404, upstream_proxy.rs:175-208). Sandbox/template environment currently takes precedence and can select an arbitrary proxy, set NO_PROXY=*, or use lowercase/ALL_PROXY variants to bypass operator routing. Use dedicated driver-owned supervisor variables injected at highest priority; arbitrary workload proxy variables must not configure this boundary.
• Fail closed on malformed configured proxies (config.rs:212-233, upstream_proxy.rs:227-253). Values such as http:// or an invalid port pass driver validation, are ignored at supervisor startup, and restore direct dialing. Fully parse once and abort startup for any configured-but-invalid value.
• Implement standard plain-HTTP forward-proxy behavior or narrow the scope (proxy.rs:3159-3200, :4396-4408). The current HTTP_PROXY path issues CONNECT and then sends origin-form requests; many enterprise proxies reject CONNECT to port 80 and require absolute-form requests. Use a separate forwarding path with operator proxy credentials, or remove the advertised plain-HTTP support.
• Preserve tunneled bytes read with the CONNECT response (upstream_proxy.rs:373-412). Any payload coalesced after \r\n\r\n is discarded. Return a buffered stream that replays overflow and cover it with a combined-response/payload test.
• Apply host-gateway bypass to HTTPS CONNECT as documented (proxy.rs:1102-1108, :4396-4400; architecture/sandbox.md:81-89). Only the plain-HTTP path currently bypasses the corporate proxy for host-gateway aliases.

Also tighten port-qualified NO_PROXY matching instead of broadening it to every port, avoid exposing long-lived proxy URL credentials through container environment/metadata, and add coverage for the security and interoperability boundaries above.

The prior control-character concern is not a header-injection blocker because decoded Basic credentials are Base64-encoded before insertion, though rejecting ASCII controls is still sensible input validation.

Docs: docs/reference/gateway-config.mdx is the correct Fern page; no docs/index.yml change is needed because the existing reference folder is already in navigation.

test:e2e and /ok to test are deferred until these review findings are resolved. @feloy, please push an updated commit; gator will review the new head before starting the pipeline gate.

Next state: gator:in-review

[johntmyers]

gator-agent

PR Review Status

Validation: This remains project-valid work for the corporate restricted-egress gap in #1792. The supervisor and Podman scope is coherent, and the relevant Fern and architecture documentation is present.

Head SHA: 9f1b270d576c4fade99a36779661c8990eff1df4

The newest commit resolves the prior proxy-selection ownership finding: reserved OPENSHELL_UPSTREAM_* variables now override or remove sandbox/template values, and conventional workload-controlled proxy variables no longer steer the supervisor boundary.

The fresh independent code review found these remaining blocking items:

• Bind SSRF and allowed_ips validation to the destination the corporate proxy actually reaches (crates/openshell-supervisor-network/src/proxy.rs:2814-2827, upstream_proxy.rs:374-379). The proxied path discards the locally validated addresses and sends the hostname in CONNECT, so proxy-side or split-horizon DNS can reach a different private or control-plane address. Use a selected validated IP for the CONNECT authority while retaining the hostname for TLS SNI/application Host, and add split-horizon regression coverage.
• Fail closed on every configured-but-invalid proxy value (crates/openshell-driver-podman/src/config.rs:217-238, upstream_proxy.rs:194-205). Values such as http:// still pass driver validation and are ignored by supervisor parsing, silently restoring direct egress. Fully parse configuration at the boundary and abort startup for malformed authority, port, IPv6, userinfo, or mixed valid/invalid scheme configuration.
• Keep proxy credentials out of the workload and container metadata (crates/openshell-driver-podman/src/container.rs:402-419, crates/openshell-supervisor-process/src/process.rs:73-90). The reserved proxy URLs are inherited by the initial agent because the child-environment denylist does not strip the new names; embedded credentials are also visible in Podman metadata. Strip all reserved variables from child processes and reject URL userinfo or deliver credentials through a protected file/secret channel.
• Implement conventional plain-HTTP forwarding or narrow the feature scope (crates/openshell-supervisor-network/src/proxy.rs:3364-3369, :4404-4411). The current HTTP path uses CONNECT and then origin-form requests; standard forward proxies commonly require absolute-form requests and may deny CONNECT to port 80.
• Preserve tunneled bytes read with the CONNECT response (crates/openshell-supervisor-network/src/upstream_proxy.rs:388-427). Bytes coalesced after \r\n\r\n are discarded; replay the overflow through a buffered stream and test a combined response/payload write.
• Apply host-gateway bypass consistently to HTTPS CONNECT (crates/openshell-supervisor-network/src/proxy.rs:1104-1110, :4399-4403). Only plain HTTP currently forces a direct dial for host-gateway aliases.
• Honor ports in port-qualified NO_PROXY entries (crates/openshell-supervisor-network/src/upstream_proxy.rs:123-132, :589-594). The current parser strips the port and broadens the bypass to every port and subdomain; carry the destination port through matching and add bracketed IPv6 coverage.
• Align docs/help and extend trust-boundary tests. crates/openshell-driver-podman/README.md:348-355 and CLI help at src/main.rs:107-117 still describe conventional-variable injection or per-sandbox precedence. Tests must also cover validated-address dialing, fail-closed startup, child-environment stripping, HTTP wire form, HTTPS host-gateway bypass, and CONNECT over-read. The Fern page is already navigated, so no docs/index.yml change is needed.

test:e2e and /ok to test remain deferred because author changes are required before pipeline gating. @feloy, please push an updated commit; gator will review the new head.

Next state: gator:in-review

[johntmyers]

gator-agent

PR Review Status

Validation: This remains project-valid work for the corporate restricted-egress gap in #1792. The supervisor and Podman scope is coherent, with the relevant Fern and architecture documentation present.

Head SHA: 42291f2a1b3b1660581dd4fef2f79e3d17e5b7cf

The newest commit resolves the prior child-environment part of the credential finding: reserved OPENSHELL_UPSTREAM_* values are now stripped from workload child processes. The operator-owned routing boundary also remains correctly enforced against sandbox/template overrides.

The fresh independent code review found these remaining blocking items:

• Bind SSRF and allowed_ips validation to the destination the corporate proxy actually reaches (crates/openshell-supervisor-network/src/proxy.rs:2802-2827, upstream_proxy.rs:337-386). The proxied path still sends the hostname in CONNECT after validating local DNS, allowing proxy-side or split-horizon DNS to reach a different internal or non-allowlisted address. Use a selected validated IP for the CONNECT authority while retaining the hostname for TLS SNI/application Host, or otherwise verifiably bind proxy resolution to the validated destination.
• Fail closed on every configured-but-invalid proxy value (crates/openshell-driver-podman/src/config.rs:217-238, upstream_proxy.rs:194-205). Values such as http:// or malformed authorities can pass driver validation, be ignored by supervisor parsing, and silently restore direct egress. Reuse a strict typed parser at both boundaries and abort startup for any present-but-invalid value.
• Keep proxy credentials out of container metadata and generated configuration (crates/openshell-driver-podman/src/container.rs:402-419, tasks/scripts/gateway.sh:311-313, :355-359). URL userinfo is still visible through Podman inspection, and the generated gateway.toml can inherit permissive umask permissions. Use a root-only secret/file channel and mode 0600 for credential-bearing configuration.
• Implement conventional plain-HTTP forwarding or narrow the feature scope (crates/openshell-supervisor-network/src/proxy.rs:3162-3205, :4399-4412). The HTTP path still uses CONNECT and then sends origin-form requests; common enterprise proxies require absolute-form forwarding and may reject CONNECT to port 80.
• Preserve tunneled bytes read with the CONNECT response (crates/openshell-supervisor-network/src/upstream_proxy.rs:388-427). Replay bytes received after the response-header terminator and add combined response/payload coverage.
• Apply host-gateway bypass consistently to HTTPS CONNECT (crates/openshell-supervisor-network/src/proxy.rs:1104-1112, :2814-2827). Plain HTTP bypasses host aliases, while CONNECT still consults the corporate proxy.
• Honor ports in port-qualified NO_PROXY entries (crates/openshell-supervisor-network/src/upstream_proxy.rs:123-163, :214-225). internal.corp:8443 is still broadened to every port and subdomain.
• Align docs/help with the implementation and scope. crates/openshell-driver-podman/README.md:348-355 and CLI help still describe conventional variables or per-sandbox precedence. Also either add Kubernetes Secret-backed propagation or explicitly narrow the feature to Podman. The existing Fern page is correct and needs no navigation change.

Please add regression coverage for the SSRF/DNS binding, fail-closed startup, HTTP wire form, HTTPS host-gateway bypass, CONNECT over-read, port-aware NO_PROXY, and credential delivery boundary.

test:e2e and /ok to test remain deferred because author changes are required before pipeline gating. @feloy, please push an updated commit; gator will review the new head.

Next state: gator:in-review

[johntmyers]

gator-agent

PR Review Status

Validation: This remains project-valid work for the corporate restricted-egress gap in #1792. The supervisor and Podman scope is coherent, and the relevant Fern and architecture documentation is present.

Head SHA: d74c780b4388877d50b20689f0ea6c1f1c1c48ae

The newest commit resolves the prior credential-storage concern substantially: proxy credentials now use a root-only per-sandbox Podman secret, the reserved auth-file path is stripped from workload children, and generated gateway configuration is mode 0600. Operator-owned proxy routing also remains protected from sandbox/template overrides.

The fresh independent code review found these remaining blocking items:

• Bind SSRF and allowed_ips validation to the destination the corporate proxy actually reaches (crates/openshell-supervisor-network/src/proxy.rs:1104-1110, :2821-2826; upstream_proxy.rs:366-369, :403-408). The proxied path still sends the hostname after validating local DNS, so split-horizon or proxy-side DNS can reach a different private, metadata, control-plane, or non-allowlisted address. CONNECT to a selected validated IP while retaining the hostname for policy, logging, TLS SNI, and certificate verification, or define an explicit trusted-proxy enforcement contract.
• Fail closed on every present-but-invalid proxy or auth setting (upstream_proxy.rs:190-224, :281-310, :350-354). Invalid reserved proxy values can be ignored, an unreadable auth file explicitly proceeds without credentials, and malformed credentials silently become unauthenticated. Make any configured-but-invalid URL, auth file, or credential fatal to supervisor startup/readiness and share validation semantics with the driver.
• Implement conventional plain-HTTP forwarding or narrow the feature scope (proxy.rs:3162-3202, :4404-4411; upstream_proxy.rs:403-415). The current http_proxy path uses CONNECT and then origin-form requests; common enterprise proxies expect absolute-form forwarding and may reject CONNECT to port 80.
• Preserve tunneled bytes read with the CONNECT response (upstream_proxy.rs:417-456). Bytes received after \r\n\r\n in the same read are discarded. Replay that overflow and cover a combined response/payload write.
• Apply host-gateway bypass consistently to HTTPS CONNECT (proxy.rs:1104-1110, :4399-4402). Plain HTTP forces a direct dial for host-gateway aliases, but HTTPS still enters generic proxy selection.
• Honor ports in port-qualified NO_PROXY entries (upstream_proxy.rs:123-163, :249-256). internal.corp:8443 currently bypasses the proxy for every port because the qualifier is discarded.
• Make the Basic-auth transport boundary explicit or protect it (upstream_proxy.rs:401-415). Credentials are stored safely, but Proxy-Authorization: Basic is sent over the plain http:// connection to the corporate proxy. Support TLS-to-proxy, restrict authenticated use to a protected link, or document this cleartext transport limitation prominently.

Docs: docs/reference/gateway-config.mdx is the correct Fern page and no docs/index.yml change is needed. Please correct the architecture claim that host-gateway aliases always dial directly, avoid implying propagation beyond Podman, and document the CONNECT-only HTTP and Basic-auth transport limitations unless the implementation changes.

Please add focused regressions for validated-IP binding, fatal invalid config/auth handling, absolute-form HTTP forwarding, CONNECT over-read, HTTPS host-gateway bypass, port-aware NO_PROXY, and authenticated proxy lifecycle/E2E behavior.

test:e2e and /ok to test remain deferred because author changes are required before pipeline gating. @feloy, please push an updated commit; gator will review the new head.

Next state: gator:in-review

[johntmyers]

gator-agent

PR Review Status

Validation: This remains project-valid work for the corporate restricted-egress gap in #1792. The supervisor and Podman scope is coherent, and the existing Fern gateway reference is the right documentation location.

Head SHA: cd9fd6cd84fc5e9f4726542a3770d2508e66f996

The latest commit materially improves the fail-closed configuration path: non-empty malformed or unsupported proxy URLs, inline credentials, and unreadable or malformed auth-file contents now fail closed through shared URL validation. The operator-owned environment boundary, workload stripping, and Podman secret staging also remain sound.

The fresh independent code review found these remaining blocking items:

• Bind SSRF and allowed_ips validation to the destination the corporate proxy actually reaches (crates/openshell-supervisor-network/src/proxy.rs:1123, :2833-2845; upstream_proxy.rs:347-389). The supervisor validates locally resolved addresses, then sends the hostname in CONNECT for proxy-side resolution. Split-horizon DNS or rebinding can therefore reach a different private, metadata, control-plane, or non-allowlisted address. CONNECT to a selected validated IP while retaining the hostname for TLS SNI/application Host, or define a verifiable trusted-proxy resolution contract.
• Implement conventional plain-HTTP forward-proxy behavior or narrow the feature scope (proxy.rs:4423, :4523-4531; upstream_proxy.rs:377-396). http_proxy uses CONNECT and then sends an origin-form request; standard enterprise proxies commonly require absolute-form forwarding and may reject CONNECT to port 80. Add a separate HTTP-forwarding path with trusted proxy auth and representative wire-level coverage.
• Preserve tunneled bytes read with the CONNECT response (upstream_proxy.rs:398-437). Bytes received after \r\n\r\n in the same read are discarded; replay the overflow and add a combined response/payload regression.
• Apply host-gateway bypass consistently to HTTPS CONNECT (proxy.rs:1123, :4418-4425; architecture/sandbox.md:89). Plain HTTP bypasses driver-injected host aliases, while HTTPS can still enter corporate proxy selection despite the documented direct-dial invariant.
• Honor ports in port-qualified NO_PROXY entries (upstream_proxy.rs:129, :266-276, :708-712). internal.corp:8443 is currently broadened to all ports and subdomains; retain the optional port and match it against the requested destination port.
• Finish the fail-closed credential/config contract (upstream_proxy.rs:211-229, :328-339; crates/openshell-driver-podman/src/driver.rs:132-147). Present-but-whitespace reserved values are treated as unset, and driver/supervisor credential validation still differs. Use one shared credential parser, reject present-but-empty values, and enforce the documented credential form consistently.
• Protect or clearly document the Basic-auth transport boundary (upstream_proxy.rs:382-396; docs/reference/gateway-config.mdx:365-378). Only plaintext http:// proxy transport is accepted, so Proxy-Authorization: Basic is exposed on that network link. Support authenticated TLS to the proxy or explicitly require a trusted isolated link and document the limitation.

Docs/tests: No docs/index.yml change is needed because the existing Reference folder already includes the page. Update Fern/architecture claims to match the final HTTP, host-gateway, NO_PROXY, fail-closed, and auth-transport behavior. Add focused regressions for proxy/local DNS divergence, absolute-form HTTP forwarding, HTTPS host-gateway bypass, CONNECT over-read, port-qualified NO_PROXY, and the shared config/credential parser.

test:e2e and /ok to test remain deferred because author changes are still required before pipeline gating. @feloy, please push an updated commit; gator will review the new head.

Next state: gator:in-review

[johntmyers]

gator-agent

PR Review Status

Validation: This remains project-valid work for the corporate restricted-egress gap in #1792. The supervisor and Podman scope is coherent, and the existing Fern gateway reference is the correct documentation location.

Head SHA: fad5b4e42fe8cf574cd345fb70d0df6af012f35f

The latest commit improves the shared fail-closed contract: driver and supervisor now share proxy URL/credential parsing, reject normal present-but-empty reserved values, and validate malformed credentials consistently. Operator-owned routing, workload stripping, root-only credential staging, and cleanup also remain sound.

The fresh independent code review found these remaining blocking items:

• Bind SSRF and allowed_ips validation to the destination reached by the corporate proxy (crates/openshell-supervisor-network/src/proxy.rs:820, :2830-2844; upstream_proxy.rs:397-402). Local addresses are validated, then the original hostname is sent in CONNECT for proxy-side resolution, so split-horizon DNS can reach a different internal, metadata, control-plane, or non-allowlisted address. CONNECT to a selected validated IP while retaining the original hostname for TLS SNI/application Host, or provide an equivalent verifiable binding.
• Bypass the corporate proxy for HTTPS host-gateway aliases too (proxy.rs:1123-1129, :4418-4429; architecture/sandbox.md:88-92). Plain HTTP has a direct-dial branch, but HTTPS still uses generic proxy selection, contradicting the documented invariant.
• Implement standard plain-HTTP forward-proxy semantics or remove http_proxy from this scope (proxy.rs:3181-3221, :4423-4429). The current path CONNECTs to port 80 and then sends origin-form requests; conventional forward proxies commonly require absolute-form requests directly over the proxy connection.
• Preserve tunneled bytes read with the CONNECT response (upstream_proxy.rs:411-450). Bytes received after \r\n\r\n in the same read are discarded; return a buffered stream that replays overflow and add a server-first/combined-write regression.
• Close the remaining fail-open configuration paths (crates/openshell-driver-podman/src/config.rs:253-278, upstream_proxy.rs:235-245, tasks/scripts/gateway.sh:358-368). Reject no_proxy when no proxy is configured, and make the development script distinguish unset from explicitly empty values instead of dropping the latter before validation.
• Protect or explicitly gate Basic authentication over plaintext proxy transport (upstream_proxy.rs:346-352). Only http:// proxy transport is accepted, so Basic credentials are recoverable by observers on that link. Support TLS-to-proxy, require an explicit insecure-auth opt-in, or at minimum document the trusted-link limitation clearly in Fern, architecture, and driver docs.
• Honor ports in port-qualified NO_PROXY entries (upstream_proxy.rs:130-169, :732-737). internal.corp:8443 is currently broadened to every port; retain the optional port and match it against the requested destination port.

Also reject proxy URL paths, queries, and fragments rather than silently ignoring them. Add focused regressions for split-horizon binding, HTTPS host-gateway bypass, absolute-form HTTP forwarding, CONNECT over-read, NO_PROXY port matching, and the remaining fail-closed cases.

Docs: No docs/index.yml change is needed. Update the current host-gateway guarantee and disclose the plaintext Basic-auth transport boundary unless implementation changes remove those limitations.

test:e2e and /ok to test remain deferred because author changes are required before pipeline gating. @feloy, please push an updated commit; gator will review the new head.

Next state: gator:in-review

[johntmyers]

gator-agent

PR Review Status

Validation: This remains project-valid work for the corporate restricted-egress gap in #1792. The supervisor and Podman scope is coherent, and the existing Fern gateway reference is the correct documentation location.

Head SHA: d08f42d9e6366200caf2e07996ede8581c316215

The newest commit resolves two prior fail-open paths: NO_PROXY without any configured proxy is now rejected, and the development script preserves explicitly empty values so validation can reject them. The independent review found no new regression specific to this commit.

The following blocking items remain:

• Bind SSRF and allowed_ips enforcement to the destination reached by the corporate proxy (crates/openshell-supervisor-network/src/proxy.rs:2830-2844, upstream_proxy.rs:362-408). Local addresses are validated, but CONNECT sends the hostname for proxy-side resolution, so split-horizon DNS can reach a different internal, metadata, control-plane, or non-allowlisted address. CONNECT to a selected validated IP while retaining the hostname for TLS SNI/application Host, or define an explicit verifiable trust-transfer contract.
• Implement conventional plain-HTTP forwarding or remove http_proxy from this scope (proxy.rs:3181-3221, :4418-4429). The current path CONNECTs to port 80 and then sends origin-form requests; enterprise forward proxies commonly require absolute-form requests and may reject CONNECT to port 80.
• Bypass the corporate proxy for HTTPS host-gateway aliases (proxy.rs:1123-1129, :4418-4429). Only the plain-HTTP path has the direct-dial special case, contradicting the documented invariant.
• Preserve tunneled bytes read with the CONNECT response (upstream_proxy.rs:418-457). Replay any bytes after \r\n\r\n instead of discarding them, with a combined response/payload regression.
• Honor the full NO_PROXY contract (upstream_proxy.rs:130-169, :286-298, :747-751). Preserve port qualifiers instead of broadening them to all ports, and either match CIDRs against validated DNS results or explicitly narrow the documented behavior.
• Reject proxy URL paths, queries, and fragments (crates/openshell-core/src/driver_utils.rs:137-170) rather than silently discarding them.
• Protect or explicitly gate Basic authentication over plaintext proxy transport (driver_utils.rs:149-153, upstream_proxy.rs:402-415). Support TLS-to-proxy, require an explicit insecure-auth opt-in, or otherwise make this security boundary safe and clear.

Docs: No docs/index.yml change is needed. Update Fern and architecture text to match the final host-gateway, HTTP forwarding, CIDR/port-qualified NO_PROXY, Podman-only propagation, and proxy-auth transport behavior.

Please add focused regressions for proxy/local DNS divergence, HTTPS host-gateway bypass, absolute-form HTTP forwarding, CONNECT over-read, URL-component rejection, port-aware/CIDR NO_PROXY, and proxy-auth lifecycle cleanup.

test:e2e and /ok to test remain deferred because author changes are required before pipeline gating. @feloy, please push an updated commit; gator will review the new head.

Next state: gator:in-review

[johntmyers]

gator-agent

PR Review Status

Validation: This remains project-valid work for the corporate restricted-egress gap in #1792. The supervisor and Podman scope is coherent, and the existing Fern gateway reference is the correct documentation location.

Head SHA: 0dc7030305877e1bd9ebe482b0ae5979ace04c85

The newest commit resolves the prior proxy-URL component finding: path, query, and fragment values are now rejected through the parser shared by the driver and supervisor, with focused coverage.

The fresh independent code review found these remaining blocking items:

• Bind SSRF and allowed_ips enforcement to the destination reached by the corporate proxy (crates/openshell-supervisor-network/src/proxy.rs:2821-2844, upstream_proxy.rs:367-408). The proxied branch ignores the locally validated address set and sends the original hostname for proxy-side resolution, so split-horizon or source-dependent DNS can reach a private, control-plane, metadata, or otherwise non-allowlisted address. CONNECT only to selected validated IPs, or validate through the same trusted resolution boundary that performs the final connection, and add divergence coverage.
• Implement conventional plain-HTTP forwarding or remove http_proxy from this scope (proxy.rs:3181-3221, :4423-4429; upstream_proxy.rs:397-415). The current path CONNECTs to port 80 and then sends origin-form requests; standard enterprise proxies commonly require absolute-form forwarding and may reject CONNECT to port 80.
• Bypass the corporate proxy for HTTPS host-gateway aliases (proxy.rs:820-826, :1123-1129, :4418-4429). Only the plain-HTTP path directly dials these driver-injected aliases, contradicting the documented invariant.
• Preserve tunneled bytes read with the CONNECT response (upstream_proxy.rs:418-451). A read can include data after \r\n\r\n, but the current code returns the raw stream and discards that suffix. Replay the overflow and add a combined response/payload regression.
• Honor the full NO_PROXY contract (upstream_proxy.rs:130-169, :287-297, :756-760). Port-qualified entries are broadened to every port, and CIDRs only match IP-literal hostnames instead of the already validated resolved addresses. Preserve optional ports, evaluate IP/CIDR entries against validated addresses, and cover hostname-to-IPv4/IPv6 CIDR cases.
• Protect or explicitly gate Basic authentication over plaintext proxy transport (crates/openshell-core/src/driver_utils.rs:111-117, upstream_proxy.rs:397-415). The secret mount protects storage, but reusable Basic credentials are still sent over the only supported http:// proxy link. Support certificate-validated HTTPS proxies or require an explicit, clearly documented insecure-auth opt-in.

Also escape the proxy values written into generated TOML at tasks/scripts/gateway.sh:362-372, and align docs/help with the parser: the docs say scheme://host:port is required while the implementation accepts a bare host, omitted port, and trailing slash.

Docs: No docs/index.yml change is needed because the existing gateway configuration page is already navigated. Update Fern and architecture text to match the final proxy syntax, HTTP behavior, host-gateway bypass, NO_PROXY, and authentication transport boundary.

test:e2e and /ok to test remain deferred because author changes are required before pipeline gating. @feloy, please push an updated commit; gator will review the new head.

Next state: gator:in-review

[johntmyers]

gator-agent

PR Review Status

Validation: This remains project-valid work for the corporate restricted-egress gap in #1792. The supervisor and Podman scope is coherent, and the existing Fern gateway reference is the correct documentation location.

Head SHA: 328a0847e8a61af0ae8bd452d323e8dcdc83a3ce

Thanks @feloy. I checked the newest commit's decision to remove plain-HTTP upstream proxying. That resolves the prior forwarding-scope finding: plain HTTP now explicitly dials directly, and the architecture text no longer promises an automatic HTTPS host-gateway bypass. Operators instead add unreachable host aliases to the reserved NO_PROXY list.

The fresh independent code review found these remaining blocking items:

• Bind SSRF and allowed_ips enforcement to the destination reached by the corporate proxy (crates/openshell-supervisor-network/src/proxy.rs:1122, :2824-2837; upstream_proxy.rs:335-377). The proxied branch still discards the locally validated addresses and sends the original hostname for proxy-side resolution, so split-horizon DNS or rebinding can reach an internal or otherwise unapproved destination. CONNECT to selected validated addresses while retaining the hostname for inner TLS SNI/application Host, or provide an equivalent trustworthy binding.
• Protect or explicitly gate Basic authentication over plaintext proxy transport (crates/openshell-supervisor-network/src/upstream_proxy.rs:370-384). The secret mount protects credentials at rest, but Proxy-Authorization: Basic is sent over plain TCP. Support certificate-validated https:// proxies, or require an explicit insecure-auth opt-in with prominent documentation.
• Honor the documented NO_PROXY contract (upstream_proxy.rs:125-164; proxy.rs:2831-2839). Port-qualified entries are broadened to every port, while IP/CIDR entries do not match a hostname's already validated resolved addresses. Preserve optional ports and evaluate matching against (host, port, validated_addrs), with direct dialing limited to matching validated addresses.
• Preserve tunneled bytes read with the CONNECT response (upstream_proxy.rs:386-425). A read may contain payload after \r\n\r\n, but the implementation returns the socket after discarding that suffix. Replay the overflow through the downstream reader and add combined response/payload coverage.
• Escape generated TOML and align the documented proxy URL grammar (tasks/scripts/gateway.sh:362-369; crates/openshell-core/src/driver_utils.rs:135-154; docs/reference/gateway-config.mdx:360-362). Environment values containing quotes, backslashes, or newlines can corrupt or inject configuration. Use a TOML serializer or tested encoder. Then either reject the bare-host/default-port form or document it consistently across Fern, architecture, README, and CLI help.

Please add focused regressions for DNS/SSRF divergence, port-aware and resolved-address NO_PROXY, CONNECT over-read, insecure-auth rejection or opt-in, and TOML metacharacters. Runtime proxy behavior still requires test:e2e after the code-review findings are resolved; the E2E label and /ok to test remain deferred on this head.

Docs: No docs/index.yml change is needed because the existing gateway configuration page is already navigated. Update it to match the final URL, NO_PROXY, host-alias, and authentication transport contracts.

@feloy, please push an updated commit; gator will re-review the new head.

Next state: gator:in-review

[feloy]

@johntmyers the PR is now rebased (still some rebase problems)

[feloy]

@johntmyers the PR is now rebased, and pre-commit passes

[johntmyers]

/ok to test f0504e9