feat(evidence): mode-aware CNCF dra-support and secure-access collection

[yuanchen8911]

Summary

Makes CNCF submission evidence collection (aicr validate --cncf-submission) mode-aware: the dra-support and secure-access sections now follow the recipe-configured GPU allocation policy (threaded via AICR_GPU_ALLOCATION_POLICY, fail-closed on unknown values) or, on standalone runs, a fail-closed capability detection mirroring allocmode.Detect — instead of hardcoding full-GPU DRA at resource.k8s.io/v1.

Motivation / Context

The Go validators became policy/mode-aware in the #1327 train, but collect-evidence.sh still hardcoded a full-GPU gpu.nvidia.com ResourceClaim in both paths. After the device-plugin default flip (#1671), those sections fail structurally on every stock cluster while the CLI reports success. This PR brings the submission script to parity with the validator semantics (dra_support_check.go, secure_access_check.go).

Fixes: #1629
Related: #1327, #1671, #1692 (failure propagation — deliberately out of scope here), kubernetes-sigs/ai-conformance#75 (device-plugin secure-access shape)

Type of Change

• Bug fix (non-breaking change that fixes an issue)
• New feature (non-breaking change that adds functionality)

Component(s) Affected

• CLI (cmd/aicr, pkg/cli)
• Docs/examples (docs/, examples/)
• Other: CNCF evidence collector (pkg/evidence/cncf)

Implementation Notes

• Policy threading: --cncf-submission resolves the recipe's policy via ResolveGPUAllocationPolicy and passes it to the collector (WithAllocationPolicy → AICR_GPU_ALLOCATION_POLICY). Recipe load/resolution errors fail closed before any evidence is collected. No recipe → standalone detection.
• Fail-closed policy consumption: the script maps device-plugin-extended-resource / dra-resource-claim directly; unknown or reserved values (e.g. dra-extended-resource) FAIL rather than silently falling back — mirrors allocmode.Verify.
• Standalone detection mirrors allocmode.Detect as far as bash reasonably can (DeviceClass + node-local gpu.nvidia.com slice devices on Ready, schedulable nodes; allocatable nvidia.com/gpu for device plugin), with the validator's preference order (DRA when usable, else device plugin, else FAIL). Every query failure is fail-closed and recorded; the evidence notes that pool-generation/taint validation remains Go-side.
• dra-support: always records the served resource.k8s.io version and per-driver ResourceSlice inventory (compute-domain slices count as valid NVIDIA publication — the feat(validator): GPU allocation capability inspection and hardening #1620 forward constraint). Behavioral claim pod only under DRA mode, with a version-correct claim (v1/v1beta2 exactly wrapper; v1beta1 bare deviceClassName) from served-version discovery. Under device-plugin mode the behavioral subtest is recorded Not applicable, verdict from API + slice evidence — matching dra_support_check.go.
• secure-access: new device-plugin isolation path (authorized container with nvidia.com/gpu: 1 must see exactly one GPU via nvidia-smi; unauthorized sibling must have no /dev/nvidia* and no working nvidia-smi), ported from the ai-conformance PR fix: open webhook container ports in NetworkPolicy workaround #75 shape; DRA path kept, version-corrected.
• Claim-shape test: a hidden emit-claim script subcommand lets a Go test exercise the embedded emitter for all three API versions (asserted against vendored types) plus the fail-closed unknown-version branch.
• Failure propagation (Result: FAIL → nonzero exit) is intentionally out of scope — it needs first-class SKIP semantics first; tracked in evidence: propagate section FAIL as collection error; add SKIP for absent prerequisites #1692.

Testing

make qualify   # full gate — PASS (isolated golangci-lint cache)

• Coverage: pkg/evidence/cncf 77.9% → 86.3% (+8.4%); pkg/cli 69.0% → 69.1%.
• New tests: claim shapes ×3 versions + fail-closed unknown version; WithAllocationPolicy env threading (subprocess); resolveCNCFAllocationPolicy table test. bash -n clean; shellcheck delta vs baseline: one style note.

Live matrix (EKS GB200 yljtrxpmzu, stock post-flip cluster, chart 0.4.1, 2026-07-09):

Cell	Setup	Result
Policy-threaded (device-plugin)	binary from this branch, stock training recipe	gpuAllocationPolicy=device-plugin-extended-resource logged; dra-support PASS (behavioral N/A note, compute-domain slice evidence); secure-access PASS via the new device-plugin isolation test (authorized saw exactly 1 GPU, sibling saw none)
Standalone detection (device-plugin)	direct script run, no recipe	detection resolved device-plugin with fail-closed provenance ("DeviceClass gpu.nvidia.com not found"); both sections PASS
Standalone detection (DRA)	full-GPU DRA advertisement temporarily re-enabled (dual state)	detection preferred dra (validator ordering); behavioral claim pod PASS via version-discovered resource.k8s.io/v1 claim; DRA isolation PASS

| Full policy-threaded collection at final head 3175208b (2026-07-10) | binary from the final head, stock recipe, deployment values verified == main's stock hydrated tuple | 4/4 PASS (dra-support N/A-path, secure-access device-plugin isolation, gang-scheduling, accelerator-metrics) — exercises the fresh-namespace and status-checked-apply fixes live |

| 8-feature run, generated inference recipe (eks/gb200/inference/dynamo/ubuntu, 2026-07-10) | recipe generated by the head binary; stock vllm-agg DGD deployed for the Dynamo-dependent sections | 8/8 PASS — the 4 core cells plus ai-service-metrics (Dynamo PodMonitor discovery), inference-gateway, robust-operator, pod-autoscaling (HPA on gpu_utilization); cluster-autoscaling excluded (scales the shared cluster's nodegroups). First pass surfaced two prerequisite-shaped issues, both resolved by deploying the DGD: robust-operator needs a live DynamoGraphDeployment, and ai-service-metrics exceeded the per-section timeout while self-deploying one (its evidence file read PASS while the section timed out — exactly the FAIL/exit-status inconsistency #1692 will formalize) |

Cluster restored to the stock post-flip state afterward (compute-domain-only slices, slinky workload 2/2 with IMEX claims re-allocated).

Risk Assessment

• Low — Isolated change, well-tested, easy to revert

Rollout notes: No behavior change for aicr validate outside --cncf-submission. DRA-mode clusters keep the previous evidence behavior (version-corrected); device-plugin clusters go from structurally-failing evidence to correct mode-aware evidence. Section FAIL still exits 0 (pre-existing; #1692).

Checklist

• Tests pass locally (make test with -race)
• Linter passes (make lint)
• I did not skip/disable tests to make CI green
• I added/updated tests for new functionality
• I updated docs if user-facing behavior changed
• Changes follow existing patterns in the codebase
• Commits are cryptographically signed (git commit -S)

[github-actions]

🌿 Preview your docs: https://nvidia-preview-issue-1629-mode-aware-evidence.docs.buildwithfern.com/aicr

[github-actions]

Recipe evidence check

No leaf overlays affected by this PR.

This gate is warning-only and never blocks merge.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

This PR updates CNCF conformance evidence collection to resolve a GPU allocation policy from recipe context, pass it into the collector, and propagate it into the evidence script. The script now detects the served resource.k8s.io version, emits version-correct ResourceClaim manifests, resolves allocation mode from policy or cluster capabilities, and routes dra-support and secure-access through device-plugin or DRA-specific evidence paths. Docs, feature descriptions, and tests were updated to match the new mode-aware behavior.

Estimated code review effort: 4 (Complex) | ~60 minutes

Possibly related PRs

• NVIDIA/aicr#1664: The main PR wires the recipe-resolved GPUAllocationPolicy into CNCF evidence collection via validate.go, Collector.WithAllocationPolicy, and AICR_GPU_ALLOCATION_POLICY.
• NVIDIA/aicr#1671: The main PR depends on the policy-resolution behavior that drives the CNCF evidence path.

Suggested labels: area/ci, area/tests

Suggested reviewers: mchmarny

🚥 Pre-merge checks | ✅ 4

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The PR implements #1629 with mode detection, version-correct ResourceClaim generation, and a device-plugin secure-access path.
Out of Scope Changes check	✅ Passed	The added tests, docs, and collector plumbing all support the requested evidence-mode changes, with no clear unrelated scope.
Title check	✅ Passed	The title clearly summarizes the main change: mode-aware CNCF evidence collection for dra-support and secure-access.
Description check	✅ Passed	The description is detailed and directly related to the changeset, covering mode-aware evidence collection, policy threading, and tests.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

pkg/evidence/cncf/scripts/collect-evidence.sh (1)

835-904: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Minor: kubectl apply here is not routed through capture (unlike collect_dra).

The DRA-support path records the apply via capture "Apply test manifest" ... (Line 611), but this isolation path applies bare (Line 903), so the generated manifest command isn't reflected in the evidence file. Consider wrapping it for parity/auditability of the evidence artifact.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/evidence/cncf/scripts/collect-evidence.sh` around lines 835 - 904, The
isolation test path applies the manifest directly, so the `kubectl apply`
command is not captured in the evidence output like the `collect_dra` flow.
Update the `collect-evidence.sh` isolation test section that builds
`test_manifest` and applies it so the apply goes through `capture` with a clear
label, preserving parity and auditability in the evidence file.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/cli/validate.go`:
- Around line 785-810: The recipe-client setup and teardown logic in
resolveCNCFAllocationPolicy is duplicated from the main validateCmd
recipe-loading flow, which risks the two paths drifting. Extract the shared
recipeClientFromCmd → PropagateOrWrap → defer Close → slog.Info → LoadRecipe
sequence into a small helper that both validateCmd and
resolveCNCFAllocationPolicy call, and keep the helper responsible for client
lifecycle and load error handling.

---

Outside diff comments:
In `@pkg/evidence/cncf/scripts/collect-evidence.sh`:
- Around line 835-904: The isolation test path applies the manifest directly, so
the `kubectl apply` command is not captured in the evidence output like the
`collect_dra` flow. Update the `collect-evidence.sh` isolation test section that
builds `test_manifest` and applies it so the apply goes through `capture` with a
clear label, preserving parity and auditability in the evidence file.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: 58ab79e6-d63e-4353-8ffe-d25bc8690706

📥 Commits

Reviewing files that changed from the base of the PR and between 30413d9 and 119415a.

📒 Files selected for processing (8)

• demos/cuj2-demo.md
• docs/conformance/cncf/index.md
• pkg/cli/validate.go
• pkg/cli/validate_test.go
• pkg/evidence/cncf/claim_shape_test.go
• pkg/evidence/cncf/collector.go
• pkg/evidence/cncf/collector_test.go
• pkg/evidence/cncf/scripts/collect-evidence.sh

[coderabbitai]

Actionable comments posted: 2

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)

pkg/evidence/cncf/scripts/collect-evidence.sh (1)

881-929: 🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Make the DRA isolation test assert exactly one visible GPU.

The container only prints GPU information. It exits successfully—and reports isolation PASS—even if a misconfigured runtime exposes multiple GPUs.

Proposed fix

-          nvidia-smi -L
+          gpu_list=$(nvidia-smi -L) || exit 1
+          echo "${gpu_list}"
+          gpu_count=$(printf '%s\n' "${gpu_list}" | grep -c '^GPU ')
+          if [ "${gpu_count}" -ne 1 ]; then
+            echo "FAIL: expected exactly one visible GPU, saw ${gpu_count}"
+            exit 1
+          fi

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/evidence/cncf/scripts/collect-evidence.sh` around lines 881 - 929, The
DRA isolation test currently only logs GPU visibility from the gpu-test
container and can pass even if more than one GPU is exposed. Update the test
manifest and verification logic in collect-evidence.sh around the isolation-test
/ gpu-test flow to assert that exactly one GPU is visible, using the existing
nvidia-smi output from the container. Fail the test if the visible GPU count is
not 1, and include the count in the evidence so the final PASS verdict in the
isolation test is only emitted when a single allocated GPU is confirmed.

♻️ Duplicate comments (1)

pkg/cli/validate.go (1)

792-810: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Duplicate recipe-client lifecycle boilerplate (repeat of a prior finding).

resolveCNCFAllocationPolicy repeats the same recipeClientFromCmd → PropagateOrWrap → defer client.Close() → slog.Info → client.LoadRecipe sequence already present in the main validateCmd body (lines 699-710). This duplication was flagged in a previous review round on this file, and the same pattern persists here.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@pkg/cli/validate.go` around lines 792 - 810, `resolveCNCFAllocationPolicy`
repeats the same recipe-client setup and teardown flow already used in
`validateCmd`, so factor that shared `recipeClientFromCmd`/error wrapping/`defer
client.Close()`/recipe loading logic into a reusable helper. Keep the
policy-specific part in `resolveCNCFAllocationPolicy` focused on resolving the
GPU allocation policy after the recipe is loaded, and reuse the helper from both
places to remove the duplicated boilerplate.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/evidence/cncf/scripts/collect-evidence.sh`:
- Around line 589-612: The DRA GPU evidence flow can report a stale PASS because
`cleanup_ns` errors are ignored, `capture` hides the `kubectl apply` status, and
the later apply steps are not checked. Update the `collect-evidence.sh` test
path around `cleanup_ns`, `capture`, and the `kubectl apply` calls to fail fast
on cleanup/apply errors, verify the old `dra-test` namespace/pod is gone before
proceeding, and emit FAIL instead of continuing when any deployment step does
not succeed. Also apply the same guard pattern to the other affected apply paths
in the script.
- Around line 556-571: The DRA inventory path in collect-evidence.sh can still
return PASS when ResourceSlice counting fails because gpu_slices and cd_slices
are forced through with || true. Update the count_resourceslices_for_driver
calls and the subsequent inventory/reporting flow so failures are detected and
propagated in both device-plugin and DRA modes, and ensure
collect_dra_device_plugin_verdict or the DRA equivalent cannot treat an
inventory error as a success. Use the existing symbols
count_resourceslices_for_driver, collect_dra_device_plugin_verdict, and
ALLOC_MODE to locate the affected logic.

---

Outside diff comments:
In `@pkg/evidence/cncf/scripts/collect-evidence.sh`:
- Around line 881-929: The DRA isolation test currently only logs GPU visibility
from the gpu-test container and can pass even if more than one GPU is exposed.
Update the test manifest and verification logic in collect-evidence.sh around
the isolation-test / gpu-test flow to assert that exactly one GPU is visible,
using the existing nvidia-smi output from the container. Fail the test if the
visible GPU count is not 1, and include the count in the evidence so the final
PASS verdict in the isolation test is only emitted when a single allocated GPU
is confirmed.

---

Duplicate comments:
In `@pkg/cli/validate.go`:
- Around line 792-810: `resolveCNCFAllocationPolicy` repeats the same
recipe-client setup and teardown flow already used in `validateCmd`, so factor
that shared `recipeClientFromCmd`/error wrapping/`defer client.Close()`/recipe
loading logic into a reusable helper. Keep the policy-specific part in
`resolveCNCFAllocationPolicy` focused on resolving the GPU allocation policy
after the recipe is loaded, and reuse the helper from both places to remove the
duplicated boilerplate.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: e5bee5be-0508-4bb7-ae3c-0b860541ddd1

📥 Commits

Reviewing files that changed from the base of the PR and between 119415a and 145d9d9.

📒 Files selected for processing (8)

• demos/cuj2-demo.md
• docs/conformance/cncf/index.md
• pkg/cli/validate.go
• pkg/cli/validate_test.go
• pkg/evidence/cncf/claim_shape_test.go
• pkg/evidence/cncf/collector.go
• pkg/evidence/cncf/collector_test.go
• pkg/evidence/cncf/scripts/collect-evidence.sh

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@pkg/evidence/cncf/scripts/collect-evidence.sh`:
- Around line 883-918: The DRA isolation check in collect-evidence.sh is only
confirming that nvidia-smi runs, so a pod with multiple visible GPUs can still
pass; update the probe logic in the ResourceClaim test flow after
emit_resourceclaim_yaml to verify exactly one GPU is visible before writing a
PASS result. Use the existing test manifest/namespace setup and the
container-side nvidia-smi output to count visible devices, and fail the test
unless the count is one.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Enterprise

Run ID: e03a121e-fbc8-469c-b349-9fbc18501b9e

📥 Commits

Reviewing files that changed from the base of the PR and between 145d9d9 and 9db8b99.

📒 Files selected for processing (8)

• demos/cuj2-demo.md
• docs/conformance/cncf/index.md
• pkg/cli/validate.go
• pkg/cli/validate_test.go
• pkg/evidence/cncf/claim_shape_test.go
• pkg/evidence/cncf/collector.go
• pkg/evidence/cncf/collector_test.go
• pkg/evidence/cncf/scripts/collect-evidence.sh

[mchmarny]

Mode-aware evidence collection done right — the script mirrors the validator semantics faithfully: the policy vocabulary matches allocation_policy.go exactly (including the reserved dra-extended-resource failing closed and unspecified falling to detection), ambient AICR_GPU_ALLOCATION_POLICY is stripped before threading, and ensure_fresh_namespace closes a real false-PASS hole (stale Succeeded pod satisfying wait_for_pod). I verified capture's new status propagation is safe: the script doesn't run set -e and run_check grades from the evidence file, so diagnostics-only callers are unaffected. The claim-shape test exercising the shipped emitter via the hidden emit-claim subcommand (asserted against vendored types) is a nice pattern.

Three low nits inline (error message swallowed by a stdout redirect, a 60s vs POD_TIMEOUT inconsistency between the two secure-access modes, and missing securityContext hardening on the new isolation pod) — none blocking. Several CI lanes (tests/Test, E2E, GPU, Tier-1 UAT) were still running at review time; good to merge from my side once they're green.