feat(disruption): disk full injection.#1058
Conversation
Zenithar
force-pushed
the
zenithar/chaos-controller/disk_full_disruption
branch
from
d238abd to
6109573
Compare
🎉 All green!🧪 All tests passed 🎯 Code Coverage (details) 🔗 Commit SHA: 9bc6a04 | Docs | Datadog PR Page | Give us feedback! |
|
The diskFull disruption creates a ballast file on the host filesystem via the injector pod. The injector pod mounts the host root at /mnt/host, but that mount has ReadOnly: true — which was correct for all existing injectors (network, CPU, etc.) that only read the host. diskFull must write to the host, so it gets read-only file system ENOSPC before even starting. Root cause
{
Name: "host",
MountPath: "/mnt/host",
ReadOnly: true, // ← must be false for diskFull
},Fix
File: Change signature: func (m *chaosPodService) generateChaosPodSpec(..., hostWritable bool) corev1.PodSpec {Inside the function, use the parameter:
File: Spec: m.generateChaosPodSpec(
targetNodeName,
terminationGracePeriod,
activeDeadlineSeconds,
args,
hostPathDirectory,
hostPathFile,
kind == chaostypes.DisruptionKindDiskFull, // hostWritable
), |
|
Many thanks for the deep investigation. I will fix that ASAP. I still have concerns about allowing write to a complete FS for writing a ballast in a dedicated directory. It will allow someone with access to the pod to alter the disrupted pod/node for purposes other than the expected disruption. I will propose a security gate. |
|
Could you also create an example file to test locally the disruption:
# Unless explicitly stated otherwise all files in this repository are licensed
# under the Apache License Version 2.0.
# This product includes software developed at Datadog (https://www.datadoghq.com/).
# Copyright 2026 Datadog, Inc.
apiVersion: chaos.datadoghq.com/v1beta1
kind: Disruption
metadata:
name: disk-full
namespace: chaos-demo
spec:
level: pod
selector:
service: demo-curl
count: 1
duration: 10m
diskFull:
path: "/mnt/data"
capacity: "95%" |
|
Could you also update the |
|
Could you also update the |
Zenithar
force-pushed
the
zenithar/chaos-controller/disk_full_disruption
branch
from
fb46e35 to
6a5b614
Compare
|
Full review comments:
|
@aymericDD, it would be interesting to have configuration validation tests. Because most of the configuration files acting as examples are not validated in CI. That would help to spot such discrepancies, especially when we introduce new feature or schema breaking changes. |
Zenithar
force-pushed
the
zenithar/chaos-controller/disk_full_disruption
branch
from
1ad02eb to
02d1b48
Compare
@Zenithar Good point, we can create a ticket for that with a design proposal and for now you can rely on e2e-test. |
|
@codex review |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 02d1b48f20
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: Thibault NORMAND <thibault.normand@datadoghq.com> Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com> Signed-off-by: Thibault NORMAND <me@zenithar.org>
… address PR comments. Add diskFull to 5 missing registration points in validateGlobalDisruptionScope (at-least-one-kind check, ContainerFailure/NodeFailure/PodReplacement compatibility, OnInit compatibility), DisruptionCount(), and Explain(). Add writable shadow mount for the target path in chaos pod spec so the injector can write ballast files while keeping /mnt/host read-only. Add capacity mode test coverage, disk_full example, complete.yaml entry, and docs/README.md link. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove the secondary `disk-full-target` HostPath volume and its container mount - Drop unused `path/filepath` import that was only needed for the removed mount - Simplify the comment to reflect that a single kubelet-pods mount suffices Rationale: The `disk-full-target` volume was a secondary mount added as a fallback for hostPath-backed volumes where the host path equals the spec path. Since the injector already resolves the actual backing path via Runtime().HostPath() at runtime (covering emptyDir, PVC, ConfigMap, and Secret volumes through the kubelet-pods mount), the secondary mount is redundant and adds unnecessary complexity to the chaos pod spec. This commit made by [/dd:git:commit:atomic](https://github.com/DataDog/claude-marketplace/tree/main/dd/commands/git/commit/atomic.md)
Zenithar
force-pushed
the
zenithar/chaos-controller/disk_full_disruption
branch
from
4d1c593 to
d21716b
Compare
HostPath signature changed in main to require context.Context as first argument. Mirrors the pattern used in disk_pressure.go. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 13817d703a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
Bug: When targeting a PVC-backed volume on a k3s cluster, the injector fails with: Root cause: The chaos pod adds a writable override mount for Proposed fix: Rather than hardcoding the kubelet path, resolve the writable mount path dynamically at chaos pod creation time using the Kubernetes API:
This works for any Kubernetes distribution without configuration or runtime detection heuristics, since the PV host path is always available from the API. |
@aymericDD I don't consider this an error. Trying to create a file on a read-only volume is a configuration error, not a code error. For simplicity, I recommend using |
Disk-Full Disruption Behaviour AssessmentDate: 2026-06-11 1. Expectations (established before cluster interaction)
2. Pod-level disruption — PVC-backed volume (k3s local-path)2.1 Test:
|
| Phase | Used | Avail | Use% |
|---|---|---|---|
| Baseline | 4.2G | 92G | 5% |
| After injection | 48G | 48G | 50% |
| After cleanup | 4.3G | 92G | 5% |
Chaos pod volume spec (key section):
{
"name": "disk-full-writable",
"hostPath": {
"path": "/var/lib/rancher/k3s/storage/pvc-8bd0e3c3-d279-4ade-8552-a418be70a2da_chaos-demo_demo",
"type": "DirectoryOrCreate"
}
}Mounted at: /mnt/host/var/lib/rancher/k3s/storage/pvc-8bd0e3c3-d279-4ade-8552-a418be70a2da_chaos-demo_demo
Key injector log entries:
{"level":"info","message":"injecting disk full disruption",
"path":"/mnt/host/var/lib/rancher/k3s/storage/pvc-8bd0e3c3-d279-4ade-8552-a418be70a2da_chaos-demo_demo",
"ballastPath":"/mnt/host/.../ .chaos-diskfull-chaos-demo-disk-full-pvc-demo-curl-547d5bc489-r25pc",
"bytesToFill":46872190976,"totalBytes":102888095744,"availableBytes":98316238848}
{"level":"info","message":"disk full disruption injected successfully"}Observations:
- Dynamic PV resolution via
resolveDiskFullWritablePathcorrectly identified k3s storage path. - Three containers without
/mnt/datamount receivedwarn+ skip, not errors. - Second injection pulse logged
"volume already at or past target fill level, skipping injection"— 1 MiB floor working correctly. - Cleanup instant: chaos pod terminated, ballast file removed, disk back to 5%.
2.2 Test: remaining: "80Gi" on demo-storage-0 (StatefulSet with shared PVC)
Target: demo-storage-0 (PVC shared-storage-demo-storage-0 backed by k3s HostPath PV)
PV host path: /var/lib/rancher/k3s/storage/pvc-721998ce-0747-4683-a47c-f024a4a31371_chaos-demo_shared-storage-demo-storage-0
Manifest:
apiVersion: chaos.datadoghq.com/v1beta1
kind: Disruption
metadata:
name: disk-full-remaining
namespace: chaos-demo
spec:
level: pod
selector:
service: demo-storage
count: 1
duration: 5m
diskFull:
path: /mnt/shared
remaining: "80Gi"Disk state:
| Phase | Used | Avail | Use% |
|---|---|---|---|
| Baseline | 4.1G | 91.7G | 4% |
| After injection | 15.8G | 80.0G | 16% |
| After cleanup | 3.8G | 92.0G | 4% |
Observations:
remaining: "80Gi"filled exactly 80Gi target (12.0 GiB ballast file,bytesToFill=12879540224).- Ballast filename:
.chaos-diskfull-chaos-demo-disk-full-remaining-demo-storage-0— per-target suffix prevents collision between StatefulSet replicas. - Two containers in
demo-storage-0share the same/mnt/sharedvolume — both containers would see the same fill effect (correct by design, documented in webhook rejection ofcontainers:field).
3. Webhook validation
| Guard | Input | Expected rejection | Actual rejection | Status |
|---|---|---|---|---|
| Node level | level: node |
"disk full disruptions can only be applied at the pod level" | * Spec: disk full disruptions can only be applied at the pod level |
PASS |
| Containers specified | containers: [read-file] |
container isolation error | * Spec: disk full disruptions apply to the entire volume, specifying certain containers does not isolate the disruption |
PASS |
Path / |
path: / |
path disallowed | * Spec: path '/' is not allowed for disk full disruptions; it would fill the node's shared container storage |
PASS |
| Mutually exclusive | capacity: "50%" + remaining: "1Gi" |
exactly one required | * Spec: capacity and remaining are mutually exclusive, only one can be set |
PASS |
capacity: "100%" |
No unsafeMode |
rejected by 1 MiB floor check | ACCEPTED (safemode disabled on this cluster) | FINDING-01 |
4. Code flow summary
Disruption CR created
└── ValidateCreate webhook
├── Spec.Validate() — path, capacity%, remaining quantity, mutual exclusion
├── level != node check — api/v1beta1/disruption_types.go:816
└── containers == [] check — api/v1beta1/disruption_types.go:762
(safemode guards for floor — only if enableSafemode=true)
Reconcile loop
└── GenerateChaosPodsOfDisruption
└── GenerateChaosPodOfDisruption
└── resolveDiskFullWritablePath(disruption, targetName)
├── level == node → return spec.Path directly
└── level == pod → resolvePodVolumeHostPath (Reader, 10s timeout)
├── Get Pod from API (uncached Reader)
├── diskFullVolumeAtPath — deepest prefix match
├── PVC volume → resolvePVCHostPath
│ ├── Get PVC (Reader)
│ ├── Get PV (Reader)
│ └── return HostPath.Path or Local.Path
└── emptyDir/ConfigMap/Secret → /var/lib/kubelet/pods
└── generateChaosPodSpec — mounts resolved path writable as disk-full-writable
Chaos pod (injector) runs
└── NewDiskFullInjector
└── Runtime().HostPath(ctx, containerID, spec.Path) — 30s timeout
→ resolves container path to host path via container runtime
└── injectVolumeFill
├── syscall.Statfs — get totalBytes, availableBytes
├── computeBytesToFill (capacity% or remaining quantity)
├── 1 MiB floor check (skip/clamp if AllowNoFloor=false)
├── dry-run guard
└── fallocate.Fallocate — Linux: fallocate(2), Darwin: write zeros
└── Clean()
└── os.Remove(ballastPath) — idempotent on ErrNotExist
Relevant files
| File | Role |
|---|---|
api/v1beta1/disk_full.go |
DiskFullSpec, Validate(), GenerateArgs() |
api/v1beta1/disruption_types.go:762,816 |
Level and containers-field restrictions |
api/v1beta1/disruption_webhook.go:762-784 |
safetyNetDiskFullMinFreeSpace (safemode-gated) |
injector/disk_full.go |
NewDiskFullInjector, Inject, Clean |
services/chaospod.go:707-808 |
Dynamic PV host path resolution |
controllers/disruption_controller.go:14-15 |
RBAC markers for PVC/PV read access |
fallocate/ |
Linux fallocate(2) / Darwin zero-write |
5. Findings
| ID | Description | Severity | Status |
|---|---|---|---|
| FINDING-01 | capacity: "100%" is accepted when enableSafemode=false (default in dev clusters). The 1 MiB floor check in safetyNetDiskFullMinFreeSpace is gated behind if enableSafemode. A production operator with safemode disabled can accidentally schedule a disk-filling disruption with no floor, leaving 0 bytes. |
Warn | Known design — operators should enable safemode in prod |
| FINDING-02 | If the k8s node goes into DiskPressure during an active disk-full disruption (e.g., 50% capacity test on a small disk), new chaos pods for subsequent disruptions are evicted by the kubelet before they can inject. The node needs ~10s after cleanup to clear DiskPressure. | Info | Expected kubelet behaviour; use conservative capacity values |
| FINDING-03 | Containers without the targeted path receive warn: could not apply injector on container — correct skip behaviour, but callers observing injectedTargetsCount may see PartiallyInjected until the matching container injects. |
Info | Expected; documented in skip condition EX-008 |
| FINDING-04 | resolveDiskFullWritablePath uses context.Background() (not the reconcile ctx). If the reconcile is cancelled mid-flight, the PV resolution continues for up to 10s. |
Info | Low-impact; background context with 10s ceiling |
6. Infrastructure changes made during assessment
| Change | File | Reason |
|---|---|---|
Added persistentvolumeclaims and persistentvolumes RBAC |
controllers/disruption_controller.go:14-15 |
Controller lacked read access to PVCs/PVs needed for dynamic path resolution |
Switched resolvePodVolumeHostPath and resolvePVCHostPath to use m.config.Reader |
services/chaospod.go:739,769,778 |
The cache-backed Client has no informers for PVC/PV types; Reader bypasses the cache and calls the API directly |
7. Test artifacts
| File | Description |
|---|---|
examples/disk_full.yaml |
Existing example (updated selector to service: demo-curl for this cluster) |
/tmp/disk_full_assess_pod_pvc.yaml |
Pod-level, capacity 50%, service: demo-curl — main happy-path test |
/tmp/disk_full_assess_remaining.yaml |
Pod-level, remaining 80Gi, service: demo-storage — remaining-mode and StatefulSet test |
Zenithar
force-pushed
the
zenithar/chaos-controller/disk_full_disruption
branch
from
2d6508d to
9bc6a04
Compare
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 9bc6a042f6
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| for _, container := range pod.Spec.Containers { | ||
| for _, mount := range container.VolumeMounts { | ||
| if strings.HasPrefix(containerPath, mount.MountPath) && len(mount.MountPath) > bestLen { |
There was a problem hiding this comment.
When the requested disk-full path only shares a string prefix with another mount (for example a pod has /data mounted and the disruption uses /database or /data2), this selects the /data volume as the writable host path even though it is not an ancestor of the target path. That makes the chaos pod mount the wrong backing volume writable and can cause the injector to skip or fail against the actual target path; require an exact match or a / boundary after the mount path.
Useful? React with 👍 / 👎.
aymericDD
left a comment
aymericDD
left a comment
There was a problem hiding this comment.
I stamp it to unblock you
| case types.DisruptionKindDiskFull: | ||
| if b.Spec.DiskFull == nil { | ||
| b.Spec.DiskFull = &v1beta1.DiskFullSpec{ | ||
| Path: "/data", |
There was a problem hiding this comment.
Why are you defining default value for the disruption?
2 participants
What does this PR do?
Adds a new
diskFulldisruption kind that genuinely fills a target pod volume using thefallocate(2)syscall, causing real ENOSPC errors on all subsequent write operations. This fills a gap where existing disruptions (DiskPressure = I/O throttling, DiskFailure = eBPF onopenatonly) don't simulate actual disk exhaustion visible to monitoring and all syscalls.Features
fallocate(2)syscall (instant, O(1) on ext4/xfs) to genuinely consume disk space. Falls back to writing zeros on unsupported filesystems.unsafeMode.allowDiskFullNoFloor). Pod-level only. Webhook warning for ephemeral-storage eviction risk.fallocate/package (adapted from detailyang/go-fallocate, MIT) — no dependency onfallocateorddbinaries in the injector image.How it differs from existing disruptions
df/monitoring?openatonlyExample
Code Quality Checklist
Testing
unittests.Test coverage
Files changed (24 files, ~1350 lines)
api/v1beta1/disk_full.go,disruption_types.go,disruption_webhook.go,safemode.goinjector/disk_full.go(ballast file via fallocate)cli/injector/disk_full.go,cli/injector/main.gofallocate/(4 platform-specific files, adapted from go-fallocate MIT)safemode/safemode_disk_full.go,safemode/safemode.gotypes/types.go(DisruptionKindDiskFull)docs/disk_full.md,docs/disruption_catalogue.mdapi/v1beta1/disk_full_test.go,injector/disk_full_test.go