Skip to content

fix(installer): retain qualified Station DKMS revision#7214

Merged
jyaunches merged 7 commits into
mainfrom
fix/station-qualified-dkms-drift-7211
Jul 20, 2026
Merged

fix(installer): retain qualified Station DKMS revision#7214
jyaunches merged 7 commits into
mainfrom
fix/station-qualified-dkms-drift-7211

Conversation

@sandl99

@sandl99 sandl99 commented Jul 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

Generic-Ubuntu DGX Station preparation now warns and retains the qualified installed DKMS revision 1:3.4.1-1ubuntu1 instead of rejecting it against the clean-host 1:3.4.0-1ubuntu1 pin. Unknown prerequisite drift still fails closed, and retained DKMS remains outside every APT transaction.

Related Issue

Fixes #7211

Changes

  • Add the one reviewed retained-compatible DKMS version to generic-Ubuntu package classification and report the retained decision during check, apply, and verify.
  • Treat exact and retained-compatible prerequisites as ready while keeping missing packages and the existing factory DKMS transition as the only APT mutation candidates.
  • Protect the current DGX Station consumer with focused tests for classification, warning output, no-op apply, APT exclusion, unknown mismatch rejection, and container/reboot safety. A direct pin replacement would change clean-host installation behavior, so the retained state is distinct from the existing target pin.
  • Document the qualified DKMS forward revision and the unchanged fail-closed boundary for every other mismatch.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification:
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: Pending maintainer review; no waiver requested.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Verification

  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — npx vitest run --project installer-integration test/install-station-host-preparation.test.ts test/install-station-package-transaction.test.ts --testTimeout=15000 (65 passed); npx vitest run --project integration test/install-station-container-coexistence.test.ts --testTimeout=15000 (17 passed); npm run test:projects:check; npm run test-size:check.
  • Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — not applicable to this focused installer package-state change.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only) — completed with 0 errors and 2 existing warnings.
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: San Dang sdang@nvidia.com

Summary by CodeRabbit

  • New Features
    • Generic Ubuntu station preparation now supports a retained-compatible DKMS forward revision, treating it as acceptable while still excluding it from APT transactions.
  • Bug Fixes
    • Improved “package readiness” handling and status/warning messages so retained-compatible DKMS is accepted/ready, with runtime validation still enforced.
    • Refined reboot/runtime gating to proceed only when prerequisites are fully ready (and the driver isn’t loaded).
  • Documentation
    • Updated the preparation guide with retained-compatible version details and the retention lifecycle/invalidations.
  • Tests
    • Expanded and updated coverage to reflect the new readiness flow, including retained DKMS acceptance/rejection cases.

Signed-off-by: San Dang <sdang@nvidia.com>
@sandl99 sandl99 added area: install Install, setup, prerequisites, or uninstall flow platform: dgx-station Affects DGX Station hardware or workflows labels Jul 20, 2026
@sandl99 sandl99 self-assigned this Jul 20, 2026
@coderabbitai

coderabbitai Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
📝 Walkthrough

Walkthrough

The Station preparation script accepts the qualified DKMS forward revision as retained-compatible, excludes it from APT transactions, and applies readiness-based checks across preparation, verification, reboot, and runtime validation. Tests and documentation cover the updated behavior.

Changes

Retained-compatible DKMS support

Layer / File(s) Summary
Package state and transaction handling
scripts/prepare-dgx-station-host.sh
Adds qualified DKMS retention classification and warnings, and excludes retained-compatible packages from APT transactions.
Readiness-based preparation workflow
scripts/prepare-dgx-station-host.sh
Updates apply, verify, reboot, driver-failure, and post-install gates to accept exact or retained-compatible prerequisites.
Behavior coverage and preparation guidance
docs/get-started/dgx-station-preparation.mdx, test/install-station-*.test.ts
Documents the qualified DKMS revision and tests retained-package handling, readiness predicates, output, rejection cases, and reboot gating.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Sequence Diagram(s)

sequenceDiagram
  participant run_apply
  participant package_state
  participant APT
  participant verify_host
  run_apply->>package_state: classify prerequisite versions
  package_state-->>run_apply: exact or retained-compatible readiness
  run_apply->>APT: request only missing package transactions
  APT-->>run_apply: apply package changes
  run_apply->>verify_host: validate accepted packages and runtime state
  verify_host-->>run_apply: verification result
Loading

Possibly related PRs

Suggested labels: feature, area: docs

Suggested reviewers: senthilr-nv, apurvvkumaria, ericksoa

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name Status Explanation Resolution
Docstring Coverage ⚠️ Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (4 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed It clearly describes the main change: retaining a qualified Station DKMS revision.
Linked Issues check ✅ Passed The changes implement the qualified DKMS retention policy, warnings, APT exclusion, validation, and rejection of unqualified drift.
Out of Scope Changes check ✅ Passed The diff stays focused on the DKMS retention feature, related tests, and documentation.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/station-qualified-dkms-drift-7211

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

github-code-quality Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in commit e130e40 in the fix/station-qualifie... branch remains at 96%, unchanged from commit 36e646d in the main branch.

TypeScript / code-coverage/cli

The overall coverage in commit e130e40 in the fix/station-qualifie... branch remains at 80%, unchanged from commit 36e646d in the main branch.

Show a code coverage summary of the most impacted files.
File main 36e646d fix/station-qualifie... e130e40 +/-
src/lib/domain/.../connect-env.ts 97% 89% -8%
src/lib/messagi...nnels/policy.ts 100% 100% 0%
src/lib/sandbox...rce-identity.ts 91% 91% 0%
src/lib/securit...ntial-filter.ts 93% 93% 0%
src/lib/state/m...ock-identity.ts 95% 95% 0%
src/lib/state/m...lock-storage.ts 97% 97% 0%
src/lib/state/m...-acquisition.ts 84% 89% +5%
src/lib/platform.ts 84% 89% +5%
src/lib/inferen...er-lifecycle.ts 65% 71% +6%
src/lib/inferen...lama/process.ts 50% 100% +50%

Updated July 20, 2026 17:28 UTC

@github-actions

Copy link
Copy Markdown
Contributor

@github-actions

github-actions Bot commented Jul 20, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — Informational

Advisor assessment: Informational / high confidence
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions
Status: No actionable findings remain in the canonical review ledger.

Model lanes

  • GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 1 warning · 0 suggestions
  • Model comparison: normalized findings differ; normalized E2E selections differ; Nemotron reported the same number of blockers, 1 more warning, the same number of suggestions.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: None

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

Signed-off-by: San Dang <sdang@nvidia.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@docs/get-started/dgx-station-preparation.mdx`:
- Around line 58-60: Update the retained-version allowlist documentation around
the “complete generic-Ubuntu package tuple” to explicitly state that a retained
DKMS revision is accepted only when every package in the tuple matches its
qualified value; DKMS-version-only matches must not qualify.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 67ce1079-3865-4c1b-baf5-56afc0deaf5f

📥 Commits

Reviewing files that changed from the base of the PR and between ba4283c and 8eac70c.

📒 Files selected for processing (3)
  • docs/get-started/dgx-station-preparation.mdx
  • scripts/prepare-dgx-station-host.sh
  • test/install-station-package-transaction.test.ts
🚧 Files skipped from review as they are similar to previous changes (1)
  • test/install-station-package-transaction.test.ts

Comment thread docs/get-started/dgx-station-preparation.mdx Outdated
@sandl99 sandl99 added v0.0.89 bug-fix PR fixes a bug or regression labels Jul 20, 2026
@sandl99
sandl99 requested a review from senthilr-nv July 20, 2026 08:07
Signed-off-by: San Dang <sdang@nvidia.com>

@senthilr-nv senthilr-nv left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed final head 26cf2c9. No blocking findings. The retained DKMS revision is bound to the complete qualified generic-Ubuntu package tuple, excluded from APT mutation, and remains fail-closed for unlisted versions or companion-package drift. Verified 103 focused tests locally; required CI, ShellCheck, CodeQL, and E2E gate are green.

@cjagwani cjagwani left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking the product-scope gate on exact head 26cf2c9d626694d035f3af6512f371e0b4049ed1; the code/security review itself passed.

Issue #7211 explicitly says it requests a product and compatibility decision and does not establish support, requires ownership and lifecycle for the compatibility policy, has no Project item or release-decision comment, and is not itself targeted to v0.0.89. This PR nonetheless makes dkms=1:3.4.1-1ubuntu1 a canonical retained-compatible Station tuple and documents that behavior as supported.

Please record an explicit maintainer product/compatibility decision covering qualification evidence, tuple ownership, lifecycle/removal criteria, supported Station profiles, security expectations, and hardware validation before this can be approved for merge. Green CI and the existing technical approval do not substitute for that decision.

Signed-off-by: Senthil Ravichandran <senthilr@nvidia.com>
@senthilr-nv

Copy link
Copy Markdown
Collaborator

v0.0.89 verification status

Acknowledging @cjagwani's valid product-scope review: this update supplies the requested technical and physical evidence, but does not treat that review as resolved without an explicit maintainer decision.

Current head e130e402a6cfc365d636043820d387c32d2398f9 is signed and GitHub-verified. It merges current main without broadening scope: the PR remains limited to its original five retained-DKMS files, with PackageKit restoration from #7241 preserved.

Completed verification:

  • 91 focused installer-integration tests passed.
  • 17 focused container-coexistence tests passed.
  • Shell syntax, diff checks, normal commit hooks, and pre-push TypeScript checks passed.
  • The candidate Station script remains byte-identical after the merge (sha256:4fa1a1d974baed5cbc23e58dfef89c5f2cf0da43151c71b6c32bc5ec7fe61cb2).
  • That exact script passed cold generic-Ubuntu Station --check and --apply on pmgb300ws-0042, including the complete pinned package transaction, PackageKit restoration, clean dpkg --audit, and expected reboot boundary. The host lost network reachability after reboot, so the staged DKMS retention passes could not finish there.

pmgb300ws-0016 is being freshly reprovisioned for the complete cold plus dkms=1:3.4.1-1ubuntu1 retained-version qualification. Per the current release schedule, that physical run is planned immediately after the v0.0.89 tag.

The required product decision and explicit validation-deferral choice are recorded for maintainer action in #7211. This PR should remain subject to that explicit decision; this status does not claim that the remaining physical pass is complete.

@jyaunches
jyaunches merged commit 70d7d28 into main Jul 20, 2026
53 checks passed
@jyaunches
jyaunches deleted the fix/station-qualified-dkms-drift-7211 branch July 20, 2026 17:31
@senthilr-nv

Copy link
Copy Markdown
Collaborator

Post-merge physical qualification is complete: PASS on a freshly reprovisioned generic Ubuntu 24.04 ARM64 DGX Station GB300.

Immutable source:

  • PR head: e130e402a6cfc365d636043820d387c32d2398f9
  • Merge commit: 70d7d287f03f82355eafccfc8084de3741069281
  • prepare-dgx-station-host.sh SHA-256: 4fa1a1d974baed5cbc23e58dfef89c5f2cf0da43151c71b6c32bc5ec7fe61cb2

Evidence:

  • Cold generic Ubuntu profile had no /etc/dgx-release, NVIDIA driver, Docker, or containers. --check returned READY_TO_APPLY; --apply completed the qualified package transaction with clean dpkg --audit and returned exit 10 / APPLY_RESULT=REBOOT_REQUIRED.
  • The host rebooted and returned with a changed boot ID. Post-reboot validation found exactly one GB300 on driver 610.43.02, Docker 29.6.1, Toolkit 1.19.1, working CDI/Docker GPU contracts, and ECC 0/0.
  • A controlled fixture advanced only DKMS from validated 1:3.4.0-1ubuntu1 to 1:3.4.1-1ubuntu1. Two consecutive --check, --apply, and --verify passes retained the forward-compatible DKMS version, returned APPLY_RESULT=COMPLETE and STATION_HOST_READY, and left APT history and the dpkg log byte-for-byte unchanged.
  • Final dpkg --audit was clean, all non-DKMS package versions remained exact, PackageKit had no residual runtime mask, and zero containers remained.

No NemoClaw onboarding, OpenShell setup, model download, or inference was run; this qualification was intentionally scoped to host preparation and retained-package behavior. v0.0.89 was not present when the test started, so the eventual tag still needs the normal immutable-commit inclusion check.

@ericksoa ericksoa mentioned this pull request Jul 20, 2026
21 tasks
ericksoa added a commit that referenced this pull request Jul 20, 2026
<!-- markdownlint-disable MD041 -->
## Summary

Add the canonical dated changelog entry for NemoClaw `v0.0.89` before
the release plan captures the tag commit.
The entry summarizes the user-visible Station preparation, inference,
recovery, policy-disclosure, and CLI-containment changes merged since
`v0.0.88`.

## Changes

- Add `docs/changelog/2026-07-20.mdx` with the exact `## v0.0.89`
release heading, parser-safe SPDX comment, summary, and detailed
bullets.
- Link each shipped theme to the most specific published OpenClaw
documentation routes.
- Keep internal E2E, advisory-registry, and review-workflow refactors
out of the user-facing release summary.

Source summary:

- #7214, #7241, #7237, #7223, #7204, #7202, #7183, and #7090 ->
`docs/changelog/2026-07-20.mdx`: Summarize qualified DGX Station
identity, package-state, PackageKit, DKMS, and reboot-handoff fixes.
- #7242, #7221, #7186, #7164, and #6874 ->
`docs/changelog/2026-07-20.mdx`: Summarize inference endpoint
provenance, provider attachment, managed vLLM cleanup and progress, and
Ollama selection guidance.
- #7225, #7216, #7192, #7136, #7096, and #6910 ->
`docs/changelog/2026-07-20.mdx`: Summarize sandbox readiness, recovery
guidance, rebuilt model routing, durable Hermes state, and gateway
restart behavior.
- #7187 -> `docs/changelog/2026-07-20.mdx`: Summarize complete effective
messaging-preset egress disclosure before policy mutation.
- #7218, #7165, and #7184 -> `docs/changelog/2026-07-20.mdx`: Summarize
structured output containment, terminal-safe route display, and Hermes
forward cleanup.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [ ] Code change with doc updates
- [x] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [ ] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [x] Tests not applicable — justification: This PR adds release-history
prose only; the focused changelog contract test validates its required
structure and routes.
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [ ] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [ ] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification:
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification

- [x] PR description includes a `Signed-off-by:` line and every commit
appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — `mise exec node@22.23.1 -- npx vitest
run test/changelog-docs.test.ts` (6 passed).
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — not applicable to this doc-only release
entry.
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only) — `mise
exec node@22.23.1 -- npm run docs` completed with 0 errors and 2
existing site-wide warnings.
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)
— the native changelog entry uses the required parser-safe MDX SPDX
comment and does not use frontmatter.

---
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Expanded DGX Station installation support for qualified GB300 and
OTA-upgraded environments.
* Preserved selected inference providers, endpoints, model pins, and
tuning settings during sandbox creation and rebuilds.
* Improved sandbox recovery by validating availability and stability
before restarting services.
* Added clearer policy output showing the complete effective messaging
egress configuration.

* **Bug Fixes**
* Hardened status and inference route displays by sanitizing terminal
control characters.
* Improved Hermes uninstall behavior by stopping detached dashboard
forwarding.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->

Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: install Install, setup, prerequisites, or uninstall flow bug-fix PR fixes a bug or regression platform: dgx-station Affects DGX Station hardware or workflows

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[DGX Station][Installer] Warn and retain qualified prerequisite version drift

5 participants