resctrl-mon: add NRI plugin for per-pod resctrl monitoring groups

[cmcantalupo]

Description

Add nri-resctrl-mon, a standalone NRI plugin that creates per-pod resctrl monitoring groups (mon_groups) to support passive monitoring of Application Energy Telemetry (AET) via consumers such as Kepler.

Motivation

Userspace daemon approaches to resctrl mon_group management suffer from a fork race: a container's first threads can execute before the daemon writes their PIDs into the mon_group's tasks file, causing energy attribution gaps. By using the NRI StartContainer hook, this plugin assigns the container's init PID to a mon_group before exec runs and threads fork, eliminating the race entirely.

What's included

• Plugin source (main.go, plugin.go, resctrl.go, state.go)
• Unit tests (plugin_test.go, resctrl_test.go)
• Dockerfile following the nri-memory-qos pattern
• Helm chart (Chart.yaml, values.yaml, templates, JSON schema)
• Documentation (new "monitoring" category, plugin docs, Helm docs)
• Sample configuration

Design decisions

• Hook selection: PostCreateContainer creates the mon_group directory (assigns the RMID), and StartContainer writes the init PID while the process is paused — before any user threads fork. PostStartContainer is a fallback in case the PID is not available at StartContainer (should not happen on containerd ≥ 2.x).
• Kernel RMID management: mkdir/rmdir on resctrl delegates RMID lifecycle to the kernel, avoiding userspace exhaustion bugs.
• Pod-level granularity: All containers in a pod share a single mon_group. The mon_group is created when the first container is created and removed when the last container stops.
• Crash recovery: Synchronize re-creates mon_groups for running pods on plugin restart and removes orphaned mon_groups left by a previous instance.
• Minimal privileges: SYS_ADMIN + DAC_OVERRIDE only (no privileged: true). hostPID: true is required so the plugin can write host-namespace PIDs to the resctrl tasks file.

Testing

• Unit tests cover plugin lifecycle, resctrl operations, namespace/label filtering, multi-container pods, orphan cleanup, path traversal rejection, and invalid UID rejection.
• Manually validated full pod lifecycle (create → run → delete) on a Clearwater Forest system with k3s, confirming mon_group creation, PID assignment, and cleanup.

Stats

22 files changed, 1958 insertions(+), 1 deletion(-)

[Copilot]

Pull request overview

Adds a new standalone NRI plugin (nri-resctrl-mon) to create per-pod resctrl monitoring groups (mon_groups) and optionally persist .begin/.end counter snapshots under a host directory for passive AET/Kepler-style consumers.

Changes:

• Introduces the nri-resctrl-mon plugin implementation (resctrl ops, lifecycle hooks, in-memory pod/container tracking, snapshot store) plus unit tests.
• Adds a Helm chart, sample configuration, and documentation under a new “monitoring” docs category.
• Wires the plugin into the repository build via the top-level Makefile.

Reviewed changes

Copilot reviewed 24 out of 24 changed files in this pull request and generated 7 comments.

Show a summary per file

File	Description
sample-configs/nri-resctrl-mon.yaml	Adds example YAML configuration for the new plugin.
Makefile	Registers nri-resctrl-mon in the build plugin list.
docs/monitoring/resctrl-mon.md	New end-user/developer documentation for resctrl-mon behavior and snapshots.
docs/monitoring/index.md	Adds a new “monitoring” docs section index.
docs/index.md	Links the new monitoring docs section from the main docs index.
docs/deployment/helm/resctrl-mon.md	Includes the Helm chart README into the docs site.
docs/deployment/helm/index.md	Adds resctrl-mon to the Helm deployment docs index.
deployment/helm/resctrl-mon/values.yaml	Default Helm values for the resctrl-mon DaemonSet/chart.
deployment/helm/resctrl-mon/values.schema.json	Helm values schema for chart parameter validation.
deployment/helm/resctrl-mon/templates/daemonset.yaml	DaemonSet template for running the plugin with required mounts/capabilities.
deployment/helm/resctrl-mon/templates/configmap.yaml	ConfigMap template for the plugin configuration file.
deployment/helm/resctrl-mon/templates/_helpers.tpl	Shared Helm template helpers (labels/selectors).
deployment/helm/resctrl-mon/README.md	Helm chart usage and configuration documentation.
deployment/helm/resctrl-mon/Chart.yaml	Helm chart metadata.
deployment/helm/resctrl-mon/.helmignore	Helm packaging ignore rules.
cmd/plugins/resctrl-mon/state.go	In-memory tracking for per-pod mon_group and container membership.
cmd/plugins/resctrl-mon/snapshot.go	Snapshot store implementation (.begin/.end, symlinks, pruning).
cmd/plugins/resctrl-mon/snapshot_test.go	Unit tests for snapshot store behavior and pruning.
cmd/plugins/resctrl-mon/resctrl.go	Resctrl filesystem operations (create/remove mon_groups, write tasks, read mon_data, cleanup).
cmd/plugins/resctrl-mon/resctrl_test.go	Unit tests for resctrl operations and safety validation.
cmd/plugins/resctrl-mon/plugin.go	Core NRI hook implementation and configuration parsing/validation.
cmd/plugins/resctrl-mon/plugin_test.go	Unit tests for lifecycle behavior, filtering, and snapshot integration.
cmd/plugins/resctrl-mon/main.go	Plugin entrypoint, flags, and NRI stub wiring.
cmd/plugins/resctrl-mon/Dockerfile	Container image build for the new plugin.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 6 comments.

[Copilot]

Pull request overview

Copilot reviewed 22 out of 22 changed files in this pull request and generated 4 comments.

[kad]

LGTM

@marquiz ?

[askervin]

LGTM.

@cmcantalupo , in case you wish to test how this plugin will feel like in helm users' point of view once nri-plugins gets released, I pushed your feature branch in my test/build/pr666 branch. That triggers github actions to publish helm charts and container images in ghcr.io. Now it should be possible to install nri-resctrl-mon from there:

helm install \
    --devel \
    -n kube-system \
    nri-resctrl-mon \
    oci://ghcr.io/askervin/nri-plugins/helm-charts/nri-resctrl-mon \
    --version v0.12-pr666-unstable \
    --set image.name=ghcr.io/askervin/nri-plugins/nri-resctrl-mon \
    --set image.tag=v0.12-pr666-unstable \
    --set image.pullPolicy=Always \
   ...

[marquiz]

Thanks @cmcantalupo for the PR. And apologies for sitting on it for so long...

A few questions/notes below but I think we could merge this.

The biggest question maybe is why no use the goresctrl library for managing the resctrl fs? I think it's API should be able to cover the stuff that's now re-implemented in resctrl.go. If possible, I'd like to see us use one implementation for that, to reduce duplicate maintenance work (and have wider usage for the one impl)

[cmcantalupo]

The biggest question maybe is why no use the goresctrl library for managing the resctrl fs? I think it's API should be able to cover the stuff that's now re-implemented in resctrl.go. If possible, I'd like to see us use one implementation for that, to reduce duplicate maintenance work (and have wider usage for the one impl)

I've actually been working on moving most of the implementation for this plugin into goresctrl to enable use cases outside of NRI. This is on my fork
https://github.com/cmcantalupo/goresctrl/tree/resctrl-mon

The modified nri-plugin that uses the new package:
https://github.com/cmcantalupo/nri-plugins/tree/resctrl-mon-goresctrl

I hope this resolution can be made in a future PR.

[marquiz]

Thanks @cmcantalupo

LGTM

I hope we can ditch the resctrl.go in a future PR