feat(kubernetes): add DNS-routed proxy mode without NET_ADMIN (draft RFC)

[dhirajsb]

Problem Statement

OpenShell's current in-sandbox network path constructs a network namespace and packet-routing rules so workload traffic is forced through the OpenShell proxy. On Kubernetes, that path requires elevated networking capabilities. Platforms that enforce a restricted pod security profile may therefore require a custom admission exception before they can run sandboxes.

Issue #899 proposes a platform mode that retains a loopback proxy for cooperative clients and uses Kubernetes NetworkPolicy as the L3/L4 boundary. Issue #1737 proposes a general isolation-backend interface. This draft explores a complementary routing mechanism: resolve governed destination names to the OpenShell proxy by using pod hostAliases for fixed names and a policy-aware DNS resolver for dynamic names. The workload pod would not create a network namespace or program packet tables and would not require NET_ADMIN.

The design goal is an explicit deployment option for restricted Kubernetes environments. It should preserve policy evaluation, L7 inspection, inference routing, and credential injection while keeping the security difference from the current namespace-based mode visible to operators.

Proposed Design

Scope and configuration

Add an operator-selected Kubernetes proxy-routing mode. The exact configuration surface is open for review, but one possible shape is:

[openshell.drivers.kubernetes.proxy_routing]
mode = "dns"
resolver_service = "openshell-proxy-resolver.openshell-system.svc"
use_host_aliases = true
network_policy_backstop = true

The mode would be opt-in. The existing namespace-based routing mode would remain the default. Workloads could not select or weaken the mode themselves.

This proposal assumes that the proxy enforcing egress runs in a separate trusted network workload from the sandbox workload. The first implementation could use one proxy workload per sandbox. A shared multi-tenant proxy is a separate design question and is intentionally not required by this proposal.

Name-to-proxy routing

At sandbox admission, the gateway compiles the effective network policy into a routing snapshot containing allowed host patterns, ports, protocols, and a monotonically increasing revision.

• Fixed synthetic names and stable exact hostnames may be placed in the pod's hostAliases, pointing to the proxy service address. This provides deterministic bootstrap behavior and does not depend on cluster DNS search rules.
• Other DNS lookups are sent to a resolver managed by the OpenShell deployment. For governed names, the resolver returns the proxy service address rather than the upstream address. The proxy resolves the actual upstream name using its own trusted resolver after policy authorization.
• The original destination port is preserved. For TLS, SNI remains available to the proxy; for HTTP, the Host header remains available. For protocols that provide neither, the resolver and proxy need a policy-revision-scoped virtual-address mapping so the proxy can recover the intended hostname from the destination address and port.
• Queries that cannot be associated with a valid sandbox routing snapshot fail closed. The resolver must not return an upstream address that lets the workload bypass the proxy.

hostAliases are immutable for a running pod, so they are suitable only for bootstrap and stable entries. Dynamic policy updates use the resolver snapshot and do not require recreating the sandbox pod.

flowchart LR
    W[Sandbox workload] -->|DNS query| R[Policy-aware resolver]
    R -->|Proxy or virtual address| W
    W -->|Connection with original host context| P[OpenShell proxy]
    P -->|Authorize against sandbox policy revision| E[Policy engine]
    E -->|Allow| U[Upstream service]
    E -->|Deny| D[Denied and audited]

Loading

Request flow

The proxy must derive the sandbox identity from an authenticated transport or an unforgeable routing binding, not from a caller-controlled header. The proxy then applies the policy revision bound to that identity before resolving or dialing the upstream destination.

sequenceDiagram
    participant W as Sandbox workload
    participant R as DNS resolver
    participant P as OpenShell proxy
    participant G as Gateway and policy authority
    participant U as Upstream

    W->>R: Resolve api.example.test
    R->>G: Read sandbox routing snapshot
    R-->>W: Proxy or virtual address
    W->>P: Connect on original destination port
    P->>G: Authorize sandbox, host, port, protocol, revision
    alt allowed
        G-->>P: Authorized route and credential handle
        P->>U: Resolve and connect upstream
        P-->>W: Proxied response
    else denied or stale
        G-->>P: Deny
        P-->>W: Fail closed
    end

Loading

Enforcement boundary

DNS and aliases steer traffic; they are not sufficient on their own to prevent a workload from dialing an IP address directly. The Kubernetes driver therefore creates a default-deny egress NetworkPolicy for each sandbox workload. It allows only the DNS resolver, the sandbox's proxy endpoint, and the minimum control-plane endpoints required by the supervisor. The proxy workload has a separate identity and egress policy, so permission for the proxy to reach upstream services does not grant the sandbox workload the same permission.

This separation is an important invariant. If the sandbox process and the upstream-dialing proxy share one pod network identity, ordinary NetworkPolicy cannot distinguish their egress. Such a local-only variant may still be useful for cooperative clients, but it must be documented as a weaker mode and must not be described as equivalent to enforced proxy routing.

Additional invariants:

• A sandbox cannot choose its routing mode, resolver view, proxy identity, or policy revision.
• Unknown names, stale mappings, resolver failures, and proxy-authentication failures deny by default.
• The proxy re-authorizes the recovered hostname and port; a successful DNS answer is not an authorization decision.
• Credentials are resolved only after sandbox identity and route authorization succeed, and are never returned to the resolver.
• Resolver and proxy logs record sandbox ID, policy revision, decision, and destination without recording secrets or sensitive query values.

Compatibility and rollout

The existing routing mode remains unchanged. The new mode is enabled per Kubernetes driver configuration and advertised in sandbox status so operators and support tooling can see which enforcement model is active.

A phased rollout could be:

1. Support exact hostnames on HTTP and TLS ports with a per-sandbox proxy workload, static aliases, and default-deny NetworkPolicy.
2. Add the policy-aware resolver and hot-reload versioning for dynamic policy updates.
3. Add virtual-address routing for non-SNI and non-HTTP protocols where the policy schema can identify a safe mapping.
4. Add conformance tests that attempt DNS, literal-IP, alternate-resolver, stale-revision, and proxy-identity bypasses before the mode is documented as enforced.

Risks and tradeoffs

• DNS is now part of the enforcement path. Resolver unavailability can block new connections, so the deployment needs health checks, redundancy, bounded caching, and observable revision state.
• Some applications use literal IP addresses, custom DNS transports, certificate pinning, or protocols without SNI/Host metadata. Literal and alternate-resolver bypasses must be blocked by NetworkPolicy; protocols without host metadata may require virtual addresses or may remain unsupported.
• Static aliases can become stale and require pod recreation. The design limits them to stable entries and uses versioned DNS state for normal policy changes.
• A separate proxy workload adds a network hop and lifecycle coordination. It also creates a clean network-policy boundary between untrusted workload egress and trusted upstream dialing.
• Binding standard destination ports without elevated privileges depends on the runtime's unprivileged-port configuration or a narrowly permitted NET_BIND_SERVICE capability. The implementation must verify the target restricted profiles and avoid assuming that all clusters configure this identically.
• NetworkPolicy semantics depend on the installed CNI. Admission should reject enforced mode when required policy behavior is unavailable or cannot be verified.

Open questions

• Should this be a concrete backend under the interface proposed in feat: establish the Isolation Backend interface #1737, or part of the platform mode discussed in feat: Support restricted SecurityContextConstraints for managed Kubernetes platforms #899?
• Is a per-sandbox proxy workload the right initial trust boundary, or is another restricted-capability topology preferable?
• Should the resolver allocate a distinct virtual address per hostname, or should the initial scope support only protocols where SNI or Host identifies the destination?
• Which CNI and restricted pod-security profiles should define the initial conformance matrix?
• How should policy-revision changes drain or invalidate existing long-lived connections?

Feedback on the trust boundary, configuration surface, and relationship to #899 and #1737 would be especially useful. This is intended as a starting design, and suggestions for a smaller or safer first increment are welcome.

Alternatives Considered

Keep the current namespace and packet-routing implementation

This preserves the current enforcement model and avoids a new DNS component, but it continues to require elevated networking capabilities on Kubernetes. It remains the appropriate default where those capabilities are available.

Cooperative proxy environment variables only

Setting HTTP_PROXY, HTTPS_PROXY, and related variables is simple and works for many clients. It does not cover applications that ignore proxy variables or dial literal addresses, and therefore needs to be presented as cooperative routing rather than enforced routing.

NetworkPolicy without DNS or alias routing

NetworkPolicy can provide an L3/L4 boundary, but by itself it does not preserve hostname-aware L7 policy, credential injection, or inference routing. Steering governed names to the OpenShell proxy retains those capabilities.

Runtime packet interception supplied by a privileged node component

A node-level component could program routing outside the sandbox pod, keeping capabilities away from the workload. That may provide stronger transparency, but it introduces cluster-wide privileged infrastructure and broader operational scope. It can remain a separate isolation-backend option.

Do nothing

Operators on restricted platforms would continue to need a custom security exception or use a mode with reduced proxy coverage. This remains viable for deployments that accept those constraints.

Agent Investigation

• Reviewed the RFC process and feature-request template on the current default branch.
• Reviewed the gateway and Kubernetes topology documentation and searched current proxy, driver, and networking code paths for NET_ADMIN usage.
• Reviewed related issues feat: Support restricted SecurityContextConstraints for managed Kubernetes platforms #899, Proposal: Split Supervisor and Agent into Separate Pods with gVisor Isolation #981, feat: establish the Isolation Backend interface #1737, cleanup(supervisor-*): Cleanup supervisor crates & prepare for split-pod topology #1810, Supervisor's musl DNS resolver fails for external domains in Kubernetes (ndots:5) #2052, and Supervisor's musl DNS resolver fails for external domains in Kubernetes (ndots:5) #2053.
• Found no existing issue that proposes the combined hostAliases plus policy-aware resolver mechanism. feat: Support restricted SecurityContextConstraints for managed Kubernetes platforms #899 is the closest overlap and should likely be coordinated with or consolidated into this design if maintainers prefer one platform-mode proposal.
• The investigation indicates that DNS steering requires a separate network enforcement boundary to prevent direct-IP bypass. The proposal therefore does not treat DNS behavior alone as a security control.

Checklist

• I've reviewed existing issues and the architecture docs
• This is a design proposal, not a "please build this" request