Skip to content

feat(supervisor): NFLOG-based bypass detection to drop the CAP_SYSLOG requirement #2382

Description

@gaveezy

Problem Statement

Sandbox pods/containers currently request CAP_SYS_ADMIN, CAP_NET_ADMIN, CAP_SYS_PTRACE, and CAP_SYSLOG. With Kubernetes user namespaces (GA in 1.36, already supported via enable_user_namespaces) the first three become namespaced and lose their host-level power — but CAP_SYSLOG does not. The kernel checks syslog(2)//dev/kmsg access against the initial user namespace (capable(), not ns_capable()), so in a hostUsers: false pod the granted CAP_SYSLOG is a no-op and the bypass detection monitor silently degrades.

The only consumer of CAP_SYSLOG is the bypass detection monitor in openshell-supervisor-process, which tails dmesg --follow for nftables log entries (prefix openshell:bypass:<ns>:) emitted by the per-workload netns REJECT rules. Reading the host-global kernel ring buffer from inside a sandbox is also a mild information leak, and requires the nf_log_all_netns sysctl on the node.

Proposed Design

Switch bypass log delivery from kernel-printk to nfnetlink_log (NFLOG):

  • During netns setup (privileged startup, before supervisor seccomp hardening), bind an AF_NETLINK/NETLINK_NETFILTER socket to an NFLOG group inside the workload netns. Netns affinity is fixed at socket() time; binding needs only CAP_NET_ADMIN over the netns owner, which works under user namespaces.
  • Generate the nftables log rules as log prefix <p> group <N> when the socket bind succeeds; keep the existing log ... flags skuid printk rules as an automatic fallback when it fails (e.g. kernel without nfnetlink_log).
  • The bypass monitor consumes NFLOG packet messages (prefix attribute for filtering, NFULA_UID for workload UID, raw IP payload for dst/ports/proto) and reports through the existing OCSF dual-emit path. The dmesg path remains as fallback (still useful in the VM driver, which runs with full capabilities).
  • Drop CAP_SYSLOG from the Kubernetes, Docker, and Podman driver capability sets.

Enforcement is unaffected: the nftables REJECT rules do not depend on the log delivery mechanism. NFLOG delivery is netns-scoped, so the monitor no longer reads host-global kernel logs, and nf_log_all_netns is no longer needed on the NFLOG path.

Alternatives Considered

  • Keep dmesg, relax kernel.dmesg_restrict on nodes — exposes host kernel logs to every pod on the node; strictly worse.
  • eBPF-based detectionbpf() is also initial-userns-gated (CAP_BPF/CAP_SYS_ADMIN in init ns), so it has the same problem as CAP_SYSLOG, plus a new dependency.
  • Netlink crate dependency (netlink-packet-netfilter / libnetfilter_log bindings) — the required protocol subset (group bind, copy mode, packet attr parsing) is ~stable kernel ABI and small; a hand-rolled libc implementation avoids new dependencies in a security-sensitive crate and matches its existing raw-FFI style.

Agent Investigation

  • CAP_SYSLOG's only consumer is bypass_monitor::spawn (crates/openshell-supervisor-process/src/bypass_monitor/mod.rs); drivers request it in openshell-driver-kubernetes/src/driver.rs, openshell-driver-docker/src/lib.rs, openshell-driver-podman/src/container.rs.
  • No eBPF usage anywhere in the workspace; netns/nftables management shells out to iproute2/nft, so NFLOG covers the whole detection path.
  • The supervisor seccomp prelude does not restrict socket(); the workload runtime filter blocks non-NETLINK_ROUTE netlink sockets, so the sandboxed workload cannot bind the NFLOG group itself.
  • The sidecar topology has no bypass log listener today (no CAP_SYSLOG there either); it keeps printk rules unchanged.
  • Implementation is complete on a branch with unit tests (netlink message layout, datagram parsing, IPv4/IPv6 packet parsing, ruleset generation); Linux cargo check/clippy (pedantic+nursery)/tests pass. PR to follow.

Metadata

Metadata

Assignees

No one assigned

    Labels

    state:triage-neededOpened without agent diagnostics and needs triage

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions