Skip to content

Fix process and path targeting in disk failure injector#1095

Open
aymericDD wants to merge 2 commits into
mainfrom
aymeric.daurelle/fix/disk-failure-v2
Open

Fix process and path targeting in disk failure injector#1095
aymericDD wants to merge 2 commits into
mainfrom
aymeric.daurelle/fix/disk-failure-v2

Conversation

@aymericDD

@aymericDD aymericDD commented Jun 18, 2026

Copy link
Copy Markdown
Contributor

Motivation

The disk failure injector mis-targeted both processes and paths, causing it to either fail to disrupt the intended opens or disrupt far more than the targeted path:

  • Process targeting only matched the container root process and its direct children (ppid == target_pid), missing grandchildren deeper in the process tree.
  • Path targeting only matched the absolute filter_path prefix. openat() also accepts relative paths, which skipped the filter entirely and were disrupted unconditionally — failing nearly every open of the targeted process regardless of -path.
  • The fmod_ret program read its argument via PT_REGS_PARM1 (offset 112), which the verifier rejects for fmod_ret, causing a permission-denied panic on load.
  • The root cgroup filter incorrectly rejected dirfd-relative opens and disrupted all relative paths regardless of target.

Changes

Process targeting — PID namespace inode filtering

  • BPF program: remove target_pid + parent traversal; add target_pid_ns_inum volatile read via BPF_CORE_READ chain task→nsproxy→pid_ns_for_children→ns.inum. Every process in a container shares the same PID namespace inode, so matching on it catches the whole process tree regardless of depth.
  • eBPF binary: replace the -process flag with -pid-ns-inum.
  • Injector: resolve the PID namespace inode via syscall.Stat at injection time; node-level disruptions pass inum=0 (match all).
  • Normalize cgroup scope to avoid false shared-namespace rejections.
  • Skip root cgroup paths in container identity comparison.
  • Reject pod-level disk failure when target container shares the host PID namespace.

Path targeting — relative path resolution

  • Resolve AT_FDCWD-relative paths to an absolute path by walking the task's pwd dentry chain across mount points (like d_path) into a per-CPU scratch buffer, then apply the same prefix filter used for absolute paths.
  • Check root filter before rejecting dirfd-relative opens.
  • Fail closed on truncated cwd depth, long relative paths, and when the mount-crossing loop exhausts its budget.
  • Reject dotdot paths to prevent traversal bypass.

Verifier fixes

  • Use the BPF_PROG macro for the fmod_ret program so the openat argument is read from BTF offset 0 instead of PT_REGS_PARM1.
  • Require CONFIG_FUNCTION_ERROR_INJECTION for fmod_ret; remove stale CONFIG_BPF_KPROBE_OVERRIDE validation.
  • Keep scratch buffers in a per-CPU array, read the filter into a runtime buffer, copy component names with bpf_probe_read_kernel_str at a masked offset, and cap cwd depth to keep verifier state from exploding (E2BIG).
  • Separate chain size from depth to correctly compute the cwd index mask.
  • Cap path writes and reject shared pod namespaces to prevent cross-container interference.

QA Instructions

  • Build and load the injector; confirm the fmod_ret program loads without the invalid bpf_context access off=112 verifier error.
  • Inject a disk failure targeting a container and verify that processes deep in the tree (grandchildren) are disrupted, not just direct children.
  • Verify a disruption scoped to -path /some/dir disrupts absolute opens under that prefix and relative opens that resolve under it, while leaving unrelated paths untouched.
  • Confirm that a pod sharing the host PID namespace is rejected at injection time.

Blast Radius

Limited to the eBPF disk failure injector (ebpf/disk-failure/, injector/disk_failure.go). No changes to other disruption kinds or the controller reconciliation path.

Documentation

@datadog-official

datadog-official Bot commented Jun 18, 2026

Copy link
Copy Markdown

Tests

🎉 All green!

🧪 All tests passed
❄️ No new flaky tests detected

🎯 Code Coverage (details)
Patch Coverage: 21.43%
Overall Coverage: 45.62% (-0.02%)

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 0112ff4 | Docs | Datadog PR Page | Give us feedback!

@aymericDD aymericDD force-pushed the aymeric.daurelle/fix/disk-failure-v2 branch from 2ef0bd5 to 1cc3b78 Compare June 18, 2026 15:00
@aymericDD aymericDD changed the title test fix(ebpf): use BPF_PROG macro for fmod_ret disk failure Jun 18, 2026
@aymericDD aymericDD force-pushed the aymeric.daurelle/fix/disk-failure-v2 branch from 39459fe to 59a7ed6 Compare June 19, 2026 07:56
@aymericDD aymericDD changed the title fix(ebpf): use BPF_PROG macro for fmod_ret disk failure fix(ebpf): correct process and path targeting in disk failure Jun 19, 2026
Fixes several bugs in eBPF-based disk failure injection:

- fmod_ret chaining and cwd index mask were broken, causing incorrect
  path resolution for relative paths
- Separate chain size from depth to correctly compute the cwd index mask
- Reject dotdot paths and use active PID namespace in disk failure eBPF
- Reject pod-level disk failure on shared host PID namespace
- Cap path writes and reject shared pod namespaces to prevent cross-
  container interference
- Fail closed on truncated cwd depth and long relative paths
- Fail closed when mount-crossing loop exhausts its budget
- Remove stale CONFIG_BPF_KPROBE_OVERRIDE validation
- Require CONFIG_FUNCTION_ERROR_INJECTION for fmod_ret
- Root filter was disrupting all relative paths; fix cgroup comparison
- Normalize cgroup scope to avoid false shared-namespace rejections
- Check root filter before rejecting dirfd-relative opens
- Skip root cgroup paths in container identity comparison

Jira: CHAOSPLT-DISK
@aymericDD aymericDD force-pushed the aymeric.daurelle/fix/disk-failure-v2 branch from 81a8614 to 5bfc170 Compare June 19, 2026 12:59
@aymericDD aymericDD changed the title fix(ebpf): correct process and path targeting in disk failure [FIX][CHAOS] Fix process and path targeting in disk failure injector Jun 19, 2026
@aymericDD

Copy link
Copy Markdown
Contributor Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 5bfc170ec2

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread ebpf/validator.go Outdated
The CONFIG_FUNCTION_ERROR_INJECTION check was placed in
GetRequiredSystemConfig(), which is shared by all eBPF users including
network disruptions (TC classifier). Kernels supporting TC-based eBPF
but lacking fmod_ret support would be incorrectly rejected for network
disruptions.

Move the check into a new ValidateDiskFailureRequiredConfig() method on
ConfigInformer, called only from the disk failure injector after the
common ValidateRequiredSystemConfig() check. Update tests and mock
accordingly, and refresh docs/disk_failure.md to reflect the current
fmod_ret hook and PID namespace targeting.
@aymericDD

Copy link
Copy Markdown
Contributor Author

@codex review

@aymericDD aymericDD changed the title [FIX][CHAOS] Fix process and path targeting in disk failure injector Fix process and path targeting in disk failure injector Jun 19, 2026
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review: Didn't find any major issues. 👍

Reviewed commit: 0112ff4af7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

@aymericDD aymericDD marked this pull request as ready for review June 19, 2026 14:53
@aymericDD aymericDD requested a review from a team as a code owner June 19, 2026 14:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant