feat(sandbox): introduce middleware layer for L7 proxy request transformations

[jhjaggars]

PR #1638 (SigV4 credential re-signing) adds AWS-specific fields (credential_signing, signing_service) threaded through the entire L7 proxy stack — policy YAML, proto, OPA data, L7EndpointConfig, RelayRequestOptions, and inline branching in relay_http_request_with_options_guarded. This works, but the approach doesn't scale: every future service-specific behavior (Azure token exchange, GCP IAM, rate limiting, custom header injection) would require adding more fields to the same structs and more if branches to the same relay function.
The review discussion on #1638 (comment) noted that a middleware pattern would fit more cleanly, aligning with the broader provider profile vision in #896.
A middleware is a named, configurable transform that intercepts an HTTP request between L7 policy evaluation and upstream forwarding. Middlewares operate on parsed HTTP requests — they see headers, body framing, and the resolved credential context, and can modify headers, buffer/re-sign bodies, or inject new headers before the request goes upstream.
Middleware configurations are defined as a top-level block (middleware_configs) parallel to network_policies. Each entry declares its middleware type and type-specific settings. Endpoints reference middleware configs by name.

middleware_configs:
  sigv4_bedrock:
    middleware: sigv4
    signing_service: bedrock
  sigv4_s3:
    middleware: sigv4
    signing_service: s3
network_policies:
  aws_bedrock:
    endpoints:
      - host: bedrock-runtime.us-east-1.amazonaws.com
        port: 443
        protocol: rest
        access: full
        middleware:
          - sigv4_bedrock
  aws_s3:
    endpoints:
      - host: s3.us-east-1.amazonaws.com
        port: 443
        protocol: rest
        access: full
        middleware:
          - sigv4_s3

• middleware_configs: top-level map of config name → config object. Each entry has a middleware field naming the middleware type, plus type-specific fields. Validated at policy load time by the middleware type's own validation function.
• middleware (on endpoint): ordered list of middleware config names to apply. Each name must resolve to an entry in middleware_configs. Validated at policy load time.
  This design provides three advantages over nesting configs inside each policy:

1. Shared configs — multiple endpoints across different policies reference the same middleware config by name without duplication.
2. Multiple instances of the same type — e.g. sigv4_bedrock and sigv4_s3 are both sigv4 middlewares with different signing_service values.
3. Clean indirection — the config name carries intent (sigv4_bedrock) while the middleware field inside it identifies the type.

// crates/openshell-sandbox/src/l7/middleware.rs
pub trait L7Middleware: Send + Sync + fmt::Debug {
    fn name(&self) -> &str;
    fn process_request<'a>(
        &'a self,
        ctx: &'a MiddlewareContext<'_>,
        req: &'a L7Request,
        headers: &'a [u8],
        client: &'a mut (dyn AsyncRead + Unpin + Send),
    ) -> Pin<Box<dyn Future<Output = Result<MiddlewareAction>> + Send + 'a>>;
}
pub enum MiddlewareAction {
    Forward { data: Vec<u8> },
    Passthrough,
}

In relay_http_request_with_options_guarded, replace the SigV4 inline block with a middleware chain:

for mw in &options.middleware {
    match mw.process_request(&mw_ctx, &req, &headers, client).await? {
        MiddlewareAction::Forward { data } => {
            upstream.write_all(&data).await?;
            forwarded = true;
            break;
        }
        MiddlewareAction::Passthrough => continue,
    }
}

Move existing SigV4 logic from the inline block in rest.rs into SigV4Middleware. The sigv4.rs utility module stays as-is — pure signing functions used by the middleware.

• L7EndpointConfig and RelayRequestOptions drop credential_signing/signing_service/host and gain middleware: Vec<Arc<dyn L7Middleware>>
• Middleware instances are constructed at policy load time: parse middleware_configs top-level block, then resolve each endpoint's middleware list against it
• Proto: top-level middleware_configs map + repeated string middleware on NetworkEndpoint
• Backward compat: credential_signing: sigv4 on an endpoint expands to a synthetic middleware_configs entry and middleware: [sigv4_<host>] during policy loading

1. Keep adding per-endpoint fields (current approach in feat(sandbox): proxy-side AWS SigV4 credential signing for CONNECT tunnels #1638) — works short-term but every new service-specific behavior adds fields and branches to the core relay path. The policy schema grows unboundedly.
2. Per-policy middleware_config (original proposal in this issue) — nests middleware config inside each network_policy. Works but forces duplication when multiple policies need the same middleware config. The top-level approach suggested in comments is strictly better.
3. Provider-level middleware only (tie to Enhanced Provider Management #896 provider profiles) — cleaner for providers that know their endpoints, but some middleware (rate limiting, custom headers) is per-endpoint, not per-provider. The endpoint-level middleware list handles both cases.
4. Tower-style middleware — more generic but heavier. The L7 proxy already has its own request/response lifecycle (parsed HTTP over raw streams, not hyper Request/Response objects). A simple trait matching the existing relay flow is more practical.
   Explored the L7 proxy architecture across these files:

• l7/mod.rs: L7EndpointConfig struct, parse_l7_config, CredentialSigning enum, policy validation. The PR adds credential_signing and signing_service fields here, parsed from OPA regorus values.
• l7/rest.rs: relay_http_request_with_options_guarded is the core relay function. The PR adds ~175 lines of inline SigV4 branching here (detect payload mode, handle Expect: 100-continue, buffer body, sign, forward). This is the function that would gain the middleware loop.
• l7/relay.rs: relay_rest and relay_with_route_selection thread RelayRequestOptions (including the new SigV4 fields) into the rest.rs relay function.
• l7/provider.rs: L7Provider trait — parse/relay/deny. Middleware sits between parse+evaluate and relay, not inside the provider.
• proxy.rs: CONNECT tunnel handling, TLS termination, routes to relay_with_inspection. Threads CredentialSigning::None through options in ~6 locations.
• sigv4.rs: Pure utility functions (extract region, strip AWS headers, apply signing). No policy coupling — stays as-is.
• openshell-policy/src/lib.rs: NetworkEndpointDef with serde, proto conversion, policy validation. PR adds credential_signing/signing_service string fields.
• proto/sandbox.proto: NetworkEndpoint message. PR adds fields 19-20.
  The PR review comments explicitly discuss the awkwardness of AWS-specific fields in the generic policy schema and reference Enhanced Provider Management #896 as the broader solution space.
• Depends on: feat(sandbox): proxy-side AWS SigV4 credential signing for CONNECT tunnels #1638 (SigV4 signing — provides the logic to be refactored into middleware)
• Related: Enhanced Provider Management #896 (Provider Profiles — would eventually declare middleware per provider type)
• Does NOT change: OPA/Rego evaluation, L7Provider trait, SecretResolver, sigv4.rs utilities
• L7Middleware trait defined with process_request returning MiddlewareAction
• SigV4Middleware implements the trait, moves logic from rest.rs inline block
• relay_http_request_with_options_guarded uses middleware chain instead of inline SigV4 branching
• Policy schema supports top-level middleware_configs block and per-endpoint middleware list referencing configs by name
• Policy validation rejects unknown middleware names, missing config entries, and invalid middleware-specific config
• Backward compat: credential_signing: sigv4 policies expand to middleware form during loading
• Proto updated with top-level middleware configs and per-endpoint middleware references
• Existing SigV4 tests pass through the new middleware path
• mise run pre-commit and mise run test pass

[johntmyers]

Could middleware_config be its own block parallel with network_policies? If an endpoint can reference a named middleware then in the case where multiple endpoints need to use the same middleware, the mw config doesn't need to be redefined on each endpoint.

[jhjaggars]

Do you mean something like this?:

middleware_configs:
   sigv4_bedrock:
       middleware: sigv4
       signing_service: bedrock
network_policies:
  aws_bedrock:
    endpoints:
      - host: bedrock-runtime.us-east-1.amazonaws.com
        port: 443
        protocol: rest
        access: full
        middleware:
          - sigv4_bedrock

if so, then yes and I like that much more. I've updated the issue to reflect that.