[FIX][CHAOS] Fix disk-failure cgroupv2 filtering and relative-path disruption#1092
Closed
aymericDD wants to merge 20 commits into
Closed
[FIX][CHAOS] Fix disk-failure cgroupv2 filtering and relative-path disruption#1092aymericDD wants to merge 20 commits into
aymericDD wants to merge 20 commits into
Conversation
|
aymericDD
force-pushed
the
aymeric.daurelle/fix/disk-failure
branch
3 times, most recently
from
13255ef to
fa08683
Compare
Adds two capabilities to the eBPF disk-failure injector:
1. Cgroupv2-based process filtering
Expose CgroupV2Path() on cgroup.Manager (returns the absolute
cgroupv2 unified hierarchy path on v2, empty string on v1).
The injector passes this path as -cgroup-path to bpf-disk-failure,
which populates a BPF_MAP_TYPE_CGROUP_ARRAY and uses
bpf_current_task_under_cgroup() to match the target container and
all its sub-cgroups (e.g. kubectl exec creates containerd
exec-<id> sub-cgroups that the old PID filter missed).
Gate disk-failure on HaveCgroupArrayMapType so load failures on
kernels that lack it surface as a clear error.
2. Relative-path disruption (cd /dir && cat file)
Previously only absolute openat paths were matched. Now the
injector stats the filter path inside the container via
/proc/<pid>/root and passes two inode/device pairs to the BPF
program:
- filter_dir_inode/filter_dir_dev: inode of the parent directory
for basename-prefix matching ("cwd=/parent && rtk read dir/file")
- filter_dir_inode2/filter_dir_dev2: inode of the path itself
when it is a directory, for exact-CWD matching
("cwd=/dir && rtk read file")
The BPF check_relative_path helper runs both checks and only fires
when dirfd == AT_FDCWD, avoiding false matches on non-CWD dirfds.
Device IDs are passed in kernel MKDEV encoding (major<<20|minor)
rather than the glibc encode_dev format to match super_block.s_dev.
ProcRoot is injected via config so tests point at a nonexistent
path and remain deterministic across macOS and Linux CI.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
aymericDD
force-pushed
the
aymeric.daurelle/fix/disk-failure
branch
from
fa08683 to
71e2c33
Compare
Fix TID/TGID naming inversion in eBPF (bpf_get_current_pid_tgid lower bits = TID, upper bits = TGID/process-ID). Improve process filter to use TGID for exclude_pid check, covering all threads. Decouple HaveCgroupArrayMapType requirement: only fail when a cgroupv2 path is actually available, allowing cgroupv1 fallback. Add syscall.Stat check on the cgroup path before passing it to the eBPF binary; fall back to PID filter on stale paths. Add bpf_trace_printk after bpf_override_return to confirm the override fires. Mount /sys/kernel/debug/tracing at /mnt/tracefs in the chaos pod so the trace pipe is reachable from the container rootfs. Replace helpers.TracePipeListen with a custom reader that routes output through zap and logs failures visibly. Add diagnostic log of bpf-disk-failure args (cgroupPath, pid, exitCode) to make filter setup observable without bpftool.
libbpfgo/helpers is no longer imported after replacing helpers.TracePipeListen() with a custom listenTracePipe function. Run go mod tidy && go mod vendor to clean up.
The helpers sub-package was removed from vendor in the previous commit; remove the corresponding entry from the license CSV to keep it in sync with the vendor directory.
aymericDD
force-pushed
the
aymeric.daurelle/fix/disk-failure
branch
from
77753da to
61f05b9
Compare
On x86_64, kprobes on __x64_sys_openat are ftrace-based ([FTRACE]
flag in /sys/kernel/debug/kprobes/list). bpf_override_return
silently does nothing through the ftrace path even when all
ALLOW_ERROR_INJECTION conditions are met. Switch to fmod_ret which
fires via the ftrace trampoline and overrides the return directly.
On ARM64, kprobes on __arm64_sys_openat are traditional int3-based
(no [FTRACE] flag). bpf_override_return works correctly there.
fmod_ret links are accepted but callbacks never fire on the tested
ARM64 kernels (6.8 Ubuntu), so keep the kprobe approach on ARM64.
The BPF C file now uses SEC("fmod_ret/__x64_sys_openat") on x86
and SEC("kprobe/__arm64_sys_openat") on ARM64 with the matching
argument-reading and override semantics for each. The Go side
branches on ebpf.UseKprobe to call AttachKprobe vs AttachGeneric.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Strip path comparison entirely so any openat call from the target process/cgroup triggers bpf_override_return. This validates whether the ARM64 kprobe override mechanism works on Datadog clusters before re-adding argument parsing.
Replace dual fmod_ret/kprobe paths with a single kprobe + bpf_override_return approach. Confirmed working on ARM64; the architecture distinction was a false lead. Remove const-arm.go and const-x64.go in favour of a single ebpf.SysOpenat() helper that returns the right kernel symbol at runtime. Drop the UseKprobe branch from main.go.
kprobe + bpf_override_return silently fails on nodes where
kprobes are FTRACE-based. fmod_ret fires via the ftrace
trampoline and its return value overrides the syscall directly,
no bpf_override_return needed.
Replace SEC("kprobe/...") with SEC("fmod_ret/...") and change
the override to a plain return -(int)exit_code. Switch
AttachKprobe to AttachGeneric in main.go. Remove the now-unused
SysOpenat helper and ebpf import.
With fmod_ret, __arm64_sys_openat and __x64_sys_openat both wrap syscall args in a single (const struct pt_regs *) argument, so PARM1(ctx) is the inner regs pointer on both architectures. Use a single code path: inner_regs dereference + bpf_probe_read_user for the path string (user-space pointer). Restore check_basename_prefix and check_relative_path helpers. The only remaining arch guard is the minimal SEC annotation.
fmod_ret programs (BPF_PROG_TYPE_TRACING) reject direct field reads from the ctx pointer (LDX instruction at ctx+offset). The BPF verifier only accepts reads via bpf_probe_read_kernel. PT_REGS_PARM1(ctx) compiles to a raw LDX — rejected. PT_REGS_PARM1_CORE(ctx) uses BPF_CORE_READ → bpf_probe_read_kernel — accepted.
Previously check_relative_path returned 0 immediately for any dirfd other than AT_FDCWD, so openat(fd, "file", ...) calls where fd is an open directory descriptor were never disrupted. On internal nodes this caused ~1-10% disruption rate instead of the expected 100%. Fix: when dirfd >= 0, walk task->files->fdt->fd[dirfd] to retrieve the directory inode and compare against filter_dir_inode / filter_dir_inode2, matching the same logic already used for the AT_FDCWD CWD case. Jira: CHAOSPLT-TBD
Add bpf_trace_printk calls to identify why relative-path openat calls are not being disrupted at 100% on internal clusters: - relpath no-inode-filter: filter_dir_inode not set by injector - relpath fd/found_ino/filter_ino: inode lookup result vs expected - relpath miss: both inode checks failed (mismatch details) - abs miss: absolute path did not match filter_path These traces appear in the chaos pod logs via trace_pipe and will show whether the issue is a missing inode filter, an inode mismatch, or a device number discrepancy on x86_64. Jira: CHAOSPLT-764 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
tracefs is frequently blocked by node security policy on internal clusters, making bpf_trace_printk output invisible. Switch to a debug_counters BPF_MAP_TYPE_ARRAY instead: - abs_hit/abs_miss: absolute path filter match/reject counts - rel_no_filter: relative path with no inode filter set - rel_hit/rel_miss: relative path inode match/reject counts The Go loader reads and logs non-zero counters every 10 s so the path filter behaviour is always visible in kubectl logs without requiring tracefs access. Jira: CHAOSPLT-764 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Two additional debug counters to distinguish root causes of rel_miss: - rel_ino_match: the dirfd's inode equals filter_dir_inode (the app opens the right directory) but the full check still fails. When this counter is high and rel_hit is low, the basename comparison is the culprit. When rel_ino_match stays 0, the dirfd never points to the expected directory. - rel_null_fd: the fdtable lookup returned a null struct file pointer. Previously these calls were silently dropped with no counter, masking the actual fdtable miss rate. Together with the existing rel_miss counter, these allow isolating whether the 1% disruption rate is caused by: (a) the app using a dirfd to a different directory, (b) a basename mismatch, or (c) the fdtable read failing silently. Jira: CHAOSPLT-764 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
bpf_current_task_under_cgroup returns != 1 for ~99% of container process invocations on the internal cluster despite those processes being correctly in the target cgroup. This causes the cgroup filter to exclude legitimate target processes, resulting in only 1% of openat calls being disrupted instead of 100%. Add PID-based fallback: when the cgroup filter returns 0 or an error, also accept processes whose ppid or tgid matches target_pid (the container's PID 1). This covers the common case where the container's init process directly spawns the test workload (dd's ppid == 110340). Add three cgroup-specific debug counters (cgroup_hit, cgroup_miss, cgroup_err) to measure how often bpf_current_task_under_cgroup returns each result, confirming the hypothesis. Jira: CHAOSPLT-764 Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Replace the one-level ppid check with a 10-level ancestor walk using is_in_target_tree(). The previous fallback only matched direct children of the container init, missing processes spawned via kubectl exec where the hierarchy is: container_init (target_pid) → exec_agent → shell → dd bpf_current_task_under_cgroup() returns 0 for ~99.8% of calls on the internal x86_64 cluster, so the fallback must cover the full process ancestry chain to reliably disrupt all openat calls. Jira: CHAOSPLT-1397
Add network namespace inode filtering as fallback when bpf_current_task_under_cgroup() fails (~99.8% miss rate on the internal x86_64 cluster). kubectl exec processes are placed in a separate process tree by containerd (containerd-shim → exec-agent → shell → dd), so they are NOT descendants of the container init. The PID ancestry walk cannot reach them. However, all processes in a container — whether from the original workload or from kubectl exec — share the same network namespace. The BPF program now falls back to comparing the current task's netns inode (task->nsproxy->net_ns->ns.inum via CO-RE) against the target container's netns inode. The injector stats /proc/<pid>/ns/net to obtain the inode and passes it via -netns-ino. Fallback priority: 1. bpf_current_task_under_cgroup (fast, O(1)) 2. netns inode match (catches kubectl exec) 3. process ancestry walk (last resort) Jira: CHAOSPLT-1397
The original kprobe implementation used bpf_probe_read to read the user-space path argument and path filter worked correctly. When we switched to fmod_ret for the syscall override (to replace the bpf_override_return that silently fails on the cluster), we also switched to bpf_probe_read_user. This caused 99% path match failures: bpf_probe_read_user intermittently fails in fmod_ret context on the x86_64 cluster, leaving the path buffer zeroed so the abs/rel branch never matches the filter. Reverting to bpf_probe_read restores the behaviour that worked in the original kprobe version. All kernel struct reads remain on the correct _kernel variant; only the user-supplied path argument bytes are affected. Jira: CHAOSPLT-1764
Contributor
Author
|
Closed in favor of #1095 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The eBPF disk-failure injector had two gaps:
cgroupv2 sub-cgroup miss — When targeting a pod-level disruption, the old PID filter only matched the container's main PID and its direct children.
kubectl execcreates acontainerd exec-<id>sub-cgroup whose processes share a different cgroup than the target, so they escaped disruption entirely.Relative-path opens not disrupted — If the target process does
cd /mnt/data && cat file, the kernel receivesopenat(AT_FDCWD, "file")with a relative path. The eBPF program only compared absolute path arguments againstfilter_path, so this open was never disrupted even though/mnt/data/filematches the configured prefix.Changes
cgroupv2 process filtering (
cgroup/,injector/disk_failure.go,ebpf/disk-failure/)CgroupV2Path()tocgroup.Managerinterface and implementation — returns the absolute cgroupv2 unified hierarchy path on v2, empty string on v1.-cgroup-pathtobpf-disk-failurewhen a cgroupv2 path is available; the BPF program populates aBPF_MAP_TYPE_CGROUP_ARRAYand usesbpf_current_task_under_cgroup()to match the container and all its sub-cgroups.HaveCgroupArrayMapType(alongside the existingHavePerfEventArrayMapTypecheck) so unsupported kernels get a clear error before BPF load.Relative-path disruption (
injector/disk_failure.go,ebpf/disk-failure/)<ProcRoot>/<pid>/root<path>and passes two inode/device pairs:filter_dir_inode/filter_dir_dev: parent directory inode for basename-prefix matching (cwd=/parent && cat dir/file)filter_dir_inode2/filter_dir_dev2: the path's own inode when it is a directory, for exact-CWD matching (cwd=/dir && cat file)check_relative_pathruns both checks and only fires whendirfd == AT_FDCWD, preventing false matches on non-CWD dirfd opens.encode_devformat to kernel MKDEV encoding (major<<20|minor) to matchsuper_block.s_dev— without this the device guard silently rejected every relative open.ProcRootis injected via config (default/proc) so unit tests set it to a nonexistent path and remain deterministic on both macOS and Linux CI.QA Instructions
cgroupv2 filtering
diskFailureon a cgroupv2 cluster.kubectl execinto the target pod and runcat /target/file— it should return the configured error code.use_cgroup_filter=0), exec'd processes were not disrupted.Relative-path disruption
paths: [/mnt/data/file]and have the target process docd /mnt/data && cat file— should now return the configured error.paths: [/mnt/data](directory) and docd /mnt/data && cat anyfile— should be disrupted;cat ../othershould not.Blast Radius
HaveCgroupArrayMapTypegate ensures unsupported kernels fail fast with a clear error rather than crashing at BPF load.AT_FDCWDare matched; multi-component relative opens from ancestor directories are not yet supported).Documentation
🤖 Generated with Claude Code