Skip to content

fix(e2e): align gateway upgrade evidence matrix#7055

Merged
apurvvkumaria merged 5 commits into
mainfrom
codex/fix-gateway-upgrade-evidence-matrix
Jul 17, 2026
Merged

fix(e2e): align gateway upgrade evidence matrix#7055
apurvvkumaria merged 5 commits into
mainfrom
codex/fix-gateway-upgrade-evidence-matrix

Conversation

@apurvvkumaria

@apurvvkumaria apurvvkumaria commented Jul 16, 2026

Copy link
Copy Markdown
Collaborator

Summary

The trusted PR E2E controller currently cannot select the existing OpenShell gateway-upgrade job because that job uses an unsupported named matrix. This change expresses the same three upgrade fixtures with the controller-supported matrix.include contract and gives each fixture a unique evidence shard, allowing the exact-head gate for #6726 to proceed after this PR is merged.

Changes

  • Convert the existing OpenShell gateway-upgrade matrix from matrix.legacy to matrix.include without changing its three release fixtures.
  • Publish a unique, safe NEMOCLAW_E2E_SHARD for each fixture and retain fixture-specific artifact names.
  • Update the gateway-upgrade boundary and PR controller tests to enforce the supported evidence contract.
  • Preserve and credit the original gateway-upgrade contributors, Aaron Erickson and Prekshi Vyas, as co-authors.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification: this only changes the internal trusted E2E evidence matrix and controller contract; user-facing behavior is unchanged.
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: pending human review of the trusted workflow change.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Verification

  • PR description includes a Signed-off-by: line and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — test/pr-e2e-gate.test.ts (27 passed) and gateway/artifact E2E workflow boundary tests (11 passed).
  • Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — not applicable because the change is confined to one workflow job and its focused contracts.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only)
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Apurv Kumaria akumaria@nvidia.com

Summary by CodeRabbit

  • Refactor

    • Streamlined the OpenShell gateway upgrade workflow configuration by flattening the job matrix fields.
    • Updated how job runner/shard values and environment variables are derived, and aligned the upgrade workflow artifact naming with the new matrix identifiers.
  • Tests

    • Updated e2e workflow boundary validation to match the revised matrix structure, including runner/shard expectations and the expected signal shard publication.
    • Adjusted artifact and live validation assertions to reflect updated workflow behavior.

Co-authored-by: Aaron Erickson <aerickson@nvidia.com>
Co-authored-by: Prekshi Vyas <prekshiv@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
@apurvvkumaria apurvvkumaria self-assigned this Jul 16, 2026
@coderabbitai

coderabbitai Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 8d8d222c-db83-437c-afd2-07fb6a3bc26f

📥 Commits

Reviewing files that changed from the base of the PR and between 8a5dc15 and 942727f.

📒 Files selected for processing (1)
  • test/pr-e2e-gate.test.ts
🚧 Files skipped from review as they are similar to previous changes (1)
  • test/pr-e2e-gate.test.ts

📝 Walkthrough

Walkthrough

The OpenShell gateway upgrade E2E job now uses top-level matrix fields for identifiers, runners, shards, environment values, and artifact names. Boundary validation and PR gate tests verify the updated matrix and shard contracts.

Changes

OpenShell gateway upgrade workflow

Layer / File(s) Summary
Top-level matrix wiring
.github/workflows/e2e.yaml
The upgrade job replaces nested legacy matrix fields with direct id, runner, and shard fields and rewires its environment values.
Shard-aware boundary validation
tools/e2e/openshell-gateway-upgrade-workflow-boundary.mts, test/e2e/support/...
Fixture extraction and workflow boundary tests validate included matrix entries, runner wiring, shard wiring, and v0.0.55 fixture data.
Artifact and gate contracts
.github/workflows/e2e.yaml, tools/e2e/upload-e2e-artifacts-workflow-boundary.mts, test/pr-e2e-gate.test.ts
Artifact naming uses matrix.id, and the PR gate expects the three upgrade signal shards.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Possibly related PRs

Suggested reviewers: cv, ericksoa, jyaunches

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title accurately reflects the main change: aligning the gateway upgrade E2E evidence matrix contract.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch codex/fix-gateway-upgrade-evidence-matrix

Comment @coderabbitai help to get the list of available commands.

@apurvvkumaria apurvvkumaria added v0.0.85 Release target area: ci CI workflows, checks, release automation, or GitHub Actions area: e2e End-to-end tests, nightly failures, or validation infrastructure bug-fix PR fixes a bug or regression labels Jul 16, 2026
@github-actions

github-actions Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

PR Review Advisor — Informational

Advisor assessment: Informational / high confidence
Next action: No advisor follow-up needed.
Findings: 0 blockers · 0 warnings · 0 suggestions
Status: No actionable findings remain in the canonical review ledger.

Model lanes

  • GPT-5.6 Terra (primary): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Nemotron 3 Ultra (second opinion): Completed · high confidence · 0 blockers · 0 warnings · 0 suggestions
  • Model comparison: normalized findings match; normalized E2E selections differ; severity counts match.

Nemotron output stays in workflow artifacts and does not change the assessment above.

E2E guidance

Advisory only. E2E / PR Gate selects and runs jobs independently.

Recommended E2E: cloud-onboard, credential-sanitization, security-posture

1 optional E2E recommendation
  • openshell-gateway-upgrade

Workflow run details

This automated review informs maintainers. Warnings and suggestions do not require a response. A maintainer decides whether to merge.

@github-code-quality

github-code-quality Bot commented Jul 16, 2026

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage remains at 96%, unchanged from the branch.

TypeScript / code-coverage/cli

The overall coverage in the branch remains at 80%, unchanged from the branch.

Show a code coverage summary of the most impacted files.
File a9fac78 9c87e4a +/-
src/lib/inference/local.ts 80% 82% +2%
src/lib/inferen...er-lifecycle.ts 65% 71% +6%
src/lib/inferen...lama/process.ts 50% 100% +50%

Updated July 17, 2026 03:02 UTC
Code Coverage is in Public Preview. Learn more and provide us with your feedback.

@apurvvkumaria
apurvvkumaria requested a review from cv July 17, 2026 00:56
@apurvvkumaria

Copy link
Copy Markdown
Collaborator Author

@cv Human review requested for exact head d402fadd97004d27e0e82acae56d8ebac9d71ed9. All ordinary checks and the authorized exact-diff E2E gate are green; cloud-onboard, credential-sanitization, and both OpenClaw/Hermes security-posture shards passed with verified evidence. The PR is mergeable with no unresolved threads, and this matrix fix unblocks #6726. Please review/approve when available; no merge was performed.

@apurvvkumaria
apurvvkumaria enabled auto-merge (squash) July 17, 2026 01:03
@cv
cv requested review from ericksoa and jyaunches July 17, 2026 01:06
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
@cv

cv commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

Release sequencing hold: please land #7054 before this PR. #7054 fixes the native retry observer used by this workflow path, and merging #7055 first would invalidate #7054 exact-head/base evidence and force another full protected E2E cycle. Current #7055 head 5054745d7a7cad69ce9a037fab93df576f149c19 is clean against dddfc51daa9f9e69dcb85ec009f86ded27ca0793, with ordinary CI and Advisor green and exact E2E running, but its final merge proof must be refreshed once onto the main that contains #7054. No approval or merge is requested yet.

@cv

cv commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

Sequencing update after the refresh to head 942727fabbe3ce6f2dd6b2f7c75de65db5b8d5e8 on base 94b5e46e14d8d425948ba75693bb86c466996869:

  • Exact ordinary CI 29548793259 passed, and the exact Advisor/CodeRabbit review is clean.
  • Trusted controller 29549179465 then failed before authorization because current main sends an invalid null conclusion in an in-progress check PATCH.
  • The minimal verified prerequisite is fix(ci): omit null conclusion from running checks #7065. Do not dispatch control-plane E2E against the current failed check.

After #7065 lands, refresh this branch on the repaired main and regenerate exact CI, automated review, and E2E evidence before merge.

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
@cv

cv commented Jul 17, 2026

Copy link
Copy Markdown
Collaborator

Maintainer refresh update for the exact landing candidate:

  • fix(ci): omit null conclusion from running checks #7065 is merged on main as a9fac78020e89cc4f617fd77fe6d7301618b50bc.
  • This PR is refreshed non-force at GitHub-Verified head 9c87e4a77925880b00a9786e20a7cae4ce52033e, with parents 942727fabbe3ce6f2dd6b2f7c75de65db5b8d5e8 and a9fac78020e89cc4f617fd77fe6d7301618b50bc.
  • The conflict-free integrated tree is 5a009bfdb69dc1a2826ed3e9a236e68ff74a44da; the original five-file PR delta remains +34/-20.
  • Local exact-tree validation passed: 160 focused gate/artifact-boundary tests, CLI build, sequential CLI type-check, source-shape check, test-size check, Biome, commit hooks, and pre-push hooks.
  • Documentation-writer review found no user-facing docs change is required: this is internal matrix/evidence plumbing, while fixtures, runners, versions, hashes, image pins, commands, and user-visible gateway-upgrade behavior remain unchanged.
  • The repaired trusted controller has created the exact coordination check and is currently waiting for ordinary CI. No credentialed E2E has been dispatched yet.

Once ordinary CI and automated review are clean and the controller records the expected explicit maintainer-authorization requirement, the next step is an exact-SHA run-control-plane dispatch followed by native-observer verification. Human approval and merge remain separate.

@cjagwani cjagwani left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@apurvvkumaria
apurvvkumaria merged commit c87ba63 into main Jul 17, 2026
52 checks passed
@apurvvkumaria
apurvvkumaria deleted the codex/fix-gateway-upgrade-evidence-matrix branch July 17, 2026 03:14
apurvvkumaria added a commit that referenced this pull request Jul 17, 2026
<!-- markdownlint-disable MD041 -->
## Summary

This PR upgrades every stable NemoClaw OpenShell selector and consumed
artifact from `0.0.72` to stable `0.0.85`, including the #6379
credential-rewrite fix and OpenShell's multiline exec contract. Stable
`v0.0.85` is verified at
[`3dee5570a46076a57a3b056f35f35ebc0861ac85`](NVIDIA/OpenShell@3dee557).
Base-owned structured trust landed in #6778 and #6779, the exact
`v0.0.85` manifest identities landed separately in #7069, and the
controller-compatible legacy gateway evidence matrix landed in #7055.

The branch is refreshed on current `main`. Exact-head GitHub CI,
automated review, protected runtime proofs, and platform-specific
acceptance are running or pending below.

## Related Issue

Fixes #6379.

## Changes

- [x] Audit all 13 adjacent OpenShell ranges from `v0.0.72` through
`v0.0.85`: 67 commits and 283 changed paths, including the unpublished
`v0.0.84` source boundary. Release notes were treated as leads rather
than proof.
- [x] Verify stable tag `v0.0.85` at `3dee5570`, successful producer run
[`29507522595`](https://github.com/NVIDIA/OpenShell/actions/runs/29507522595),
and inclusion of required credential fix `40194f93`.
- [x] Advance the blueprint floor/ceiling, installer floor/ceiling/dev
floor, Brev defaults, stable E2E gateway-auth pin, supervisor index,
sandbox-build fallback, credential manifest, current docs, and
user-visible messages to `0.0.85`.
- [x] Pin all eight CLI/gateway/sandbox release archives, three
component manifests, extracted release-binary identities, and the
multi-arch supervisor image to immutable `v0.0.85` identities.
Historical `0.0.72` and `0.0.82` references remain only where cleanup,
fallback, trust, or compatibility evidence requires them.
- [x] Retain both `v0.0.82` standalone sandbox digest fallbacks when
host-side `--version` cannot run, and exercise the true `0.0.86`
above-maximum boundary.
- [x] Replace the migration ledger with
`docs/security/openshell-0.0.85-migration-review.md`, covering the
complete range, dependency delta, consumed contracts, evidence-backed
exclusions, mitigations, and blocking runtime/release gates.
- [x] Reject OpenShell's revisioned credential-name namespace before
provider mutation while preserving cleanup of persisted legacy entries
across host and Hermes adapters.
- [x] Report OpenShell's fail-closed `CONNECT 503` as unavailable TLS
credential rewriting instead of a generic curl transport error.
- [x] Preserve LF, CR, CRLF, quotes, and heredocs byte-exactly across
multiline exec while retaining NUL, multiline-workdir, and
request-environment boundaries.
- [x] Reject supervisor-only `OPENSHELL_TLS_CA`, `OPENSHELL_TLS_CERT`,
and `OPENSHELL_TLS_KEY` if an entrypoint, exec, or connect child
receives them.
- [x] Validate that each selected OpenShell archive contains exactly one
expected regular binary before extraction; reject absolute paths,
traversal, duplicates, extras, links, and devices.
- [ ] Prove the exact head on the supported Linux x86, macOS Docker
Desktop/Colima, WSL, Colossus, and DGX Spark paths selected by
CI/advisor.
- [ ] On a physical Docker 27 DGX Spark arm64 host, prove honest `mcp
status`, real credential substitution, an authenticated MCP tool call,
rotation/rebuild/removal behavior, and absence of literal
`openshell:resolve:env:` leakage.
- [ ] Record the platform decision for Docker 27 versus upstream
OpenShell's Docker 28 requirement.

## Release provenance

- All eight consumed release archives have valid GitHub SLSA
attestations bound to source commit `3dee5570`; the three published
checksum manifests and all ten installer/Brev references were
independently reverified after the update.
- The selected supervisor image is the immutable multi-arch index
`sha256:f4226253a3525c3832adac5b38b419a0f27d1e915effe565b5885e20f93cd5e9`.
- GitHub currently exposes no OCI referrer/attestation/SBOM for that
supervisor image, and its configs contain no source/revision/version
labels. The review records that upstream provenance limitation rather
than inferring missing evidence.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [ ] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: pending exact-head
maintainer review
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification

- [x] PR description includes DCO sign-off declarations and all 78
commits currently appear as `Verified` in GitHub
- [x] Normal commit/push hooks and `npm run check:diff` passed; the
generated-platform citation repair is signed, and current exact head
`610f7cd9ff9cc8d3fdc47f40271d0b56b452a0cc` is GitHub-verified
- [x] The checker is byte-identical to current `main`; the base-owned
`scripts/check-installer-hash.sh` passed all three manifests and all ten
consumed release references against published `v0.0.85` assets
- [x] Focused current-head repairs passed: feature gate 15/15, installer
integration 47 passed with 1 intentional skip, installer-hash and
migration ledger 78/78, gateway workflow boundary 4/4, PR E2E controller
27/27, and generated platform docs 19/19
- [x] Broader compatibility, MCP boundary, Hermes, and E2E-support tests
passed earlier in the migration; the opt-in version-pin live suite
passed all three `gh` and curl paths
- [x] The #7046 conflict refresh preserves all four gateway-upgrade
shards and the typed DCode target; combined integration passed 172/172,
serial E2E-support passed 1,156 with 2 intentional skips, and
file-size/project-membership checks passed
- [x] `npm run docs` completed successfully, including generated
variants, route validation, and Fern checks (0 errors; 2 existing
warnings)
- [ ] Exact-head GitHub CI, automated review, and protected E2E passed —
ordinary CI, both advisor lanes, and protected E2E are running or
pending at head `610f7cd9ff9cc8d3fdc47f40271d0b56b452a0cc`
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)
— the security review has the required SPDX header and is not a Fern
page.

---
Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>

---------

Signed-off-by: Aaron Erickson <aerickson@nvidia.com>
Signed-off-by: Prekshi Vyas <prekshiv@nvidia.com>
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Charan Jagwani <cjagwani@nvidia.com>
Co-authored-by: Prekshi Vyas <prekshiv@nvidia.com>
Co-authored-by: Carlos Villela <cvillela@nvidia.com>
Co-authored-by: Apurv Kumaria <akumaria@nvidia.com>
Co-authored-by: Charan Jagwani <cjagwani@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area: ci CI workflows, checks, release automation, or GitHub Actions area: e2e End-to-end tests, nightly failures, or validation infrastructure bug-fix PR fixes a bug or regression v0.0.85 Release target

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants