Skip to content

fix(hermes): avoid Anthropic SSE for compatible endpoints#6295

Closed
chengjiew wants to merge 3 commits into
mainfrom
fix/6289_anthropic_sse_message_start
Closed

fix(hermes): avoid Anthropic SSE for compatible endpoints#6295
chengjiew wants to merge 3 commits into
mainfrom
fix/6289_anthropic_sse_message_start

Conversation

@chengjiew

@chengjiew chengjiew commented Jul 6, 2026 •
edited by coderabbitai Bot
Loading

Copy link
Copy Markdown
Contributor

Summary

NemoClaw now keeps Hermes compatible-Anthropic endpoint switches on the managed OpenAI-compatible inference route so hermes -z avoids the broken Anthropic SSE message stream. True Anthropic provider routing remains on Anthropic Messages, and the runtime switching docs now describe that distinction.

Related Issue

Fixes #6289

Changes

  • Force Hermes compatible-anthropic-endpoint runtime API resolution to openai-completions in src/lib/actions/inference-route-api.ts.
  • Add regression coverage in src/lib/actions/inference-route-api.test.ts for Hermes compatible-Anthropic endpoint routing.
  • Update src/lib/actions/inference-set-hermes-run.test.ts to assert Hermes writes the managed /v1 OpenAI-compatible proxy config without anthropic_messages mode.
  • Clarify the Hermes API-family sync docs in docs/inference/switch-inference-providers.mdx.

Type of Change

  • Code change (feature, bug fix, or refactor)
  • Code change with doc updates
  • Doc only (prose changes, no code sample modifications)
  • Doc only (includes code sample changes)

Quality Gates

  • Tests added or updated for changed behavior
  • Existing tests cover changed behavior — justification:
  • Tests not applicable — justification:
  • Docs updated for user-facing behavior changes
  • Docs not applicable — justification:
  • Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
  • Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: pending maintainer review in this PR; no waiver requested.
  • Non-success, skipped, or missing CI check accepted by maintainer — check name, approval link, and follow-up issue:

Verification

  • PR description includes the DCO sign-off declaration and every commit appears as Verified in GitHub
  • Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
  • Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result or justification: git diff --check && npm test -- src/lib/actions/inference-route-api.test.ts src/lib/actions/inference-set-hermes-run.test.ts passed; 2 files and 15 tests.
  • Applicable broad gate passed — npm test for broad runtime/test-harness changes; npm run check for repo-wide validation/coverage changes — command/result: full npm test was attempted after focused verification but exceeded the interactive delivery window and was terminated as stale; the changed inference-route tests passed.
  • Quality Gates section completed with required justifications or waivers
  • No secrets, API keys, or credentials committed
  • npm run docs builds without warnings (doc changes only): npm run docs exited 0 with 0 errors; Fern reported 2 warnings.
  • Doc pages follow the style guide (doc changes only)
  • New doc pages include SPDX header and frontmatter (new pages only)

Signed-off-by: Chengjie Wang chengjiew@nvidia.com

Summary by CodeRabbit

  • Bug Fixes
    • Fixed Hermes routing for sessions using a compatible Anthropic endpoint to preserve OpenAI-compatible behavior instead of switching to Anthropic-style streaming.
    • Improved inference API selection when session preferences and model settings differ, ensuring the correct route API is chosen.
    • Updated Hermes-related expectations and configuration assertions to match the refined routing behavior and resulting settings.

@chengjiew
fix(hermes): avoid Anthropic SSE for compatible endpoints
a971562 
Signed-off-by: Chengjie Wang <chengjiew@nvidia.com>
@coderabbitai

coderabbitai Bot commented Jul 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Review Change Stack

📝 Walkthrough

Walkthrough

Hermes routing now returns openai-completions for compatible-anthropic-endpoint, while related tests and docs were updated to reflect the preserved OpenAI-compatible /v1 proxy behavior.

Changes

Hermes routing fix

Layer / File(s) Summary
Routing logic change
src/lib/actions/inference-route-api.ts, src/lib/actions/inference-route-api.test.ts		 Adds an early return for Hermes on compatible-anthropic-endpoint, and updates the route-resolution test suite with a new case for #6289.
Test and docs updates
src/lib/actions/inference-set-hermes-run.test.ts, docs/inference/switch-inference-providers.mdx		 Adjusts Hermes run expectations for the managed OpenAI-compatible route and clarifies the documented API-family behavior for Hermes routing.

Estimated code review effort: 2 (Simple) | ~10 minutes

Suggested labels: bug-fix

Suggested reviewers: ericksoa, jyaunches

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Linked Issues check ✅ Passed The code and tests align with #6289 by routing Hermes-compatible Anthropic endpoints to OpenAI-compatible completions instead of Anthropic Messages.
Out of Scope Changes check ✅ Passed The changes stay focused on the Hermes routing fix, regression tests, and supporting docs.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely matches the main change: Hermes avoids Anthropic SSE for compatible endpoints.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/6289_anthropic_sse_message_start

Comment @coderabbitai help to get the list of available commands.

@github-code-quality

github-code-quality Bot commented Jul 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Code Coverage Overview

Languages: TypeScript

TypeScript / code-coverage/plugin

The overall coverage in the branch is 96%. Coverage data for the branch is not yet available.

Show a code coverage summary of the most covered files.
File 2ab2389 +/-
nemoclaw/src/se...cret-scanner.ts — 100% —
nemoclaw/src/commands/slash.ts — 100% —
nemoclaw/src/li...bprocess-env.ts — 100% —
nemoclaw/src/bl...eprint/state.ts — 98% —
nemoclaw/src/onboard/config.ts — 98% —
nemoclaw/src/bl...int/snapshot.ts — 97% —
nemoclaw/src/bl...print/runner.ts — 95% —
nemoclaw/src/co...ration-state.ts — 94% —
nemoclaw/src/bl...ate-networks.ts — 94% —
nemoclaw/src/index.ts — 94% —

TypeScript / code-coverage/cli

The overall coverage in the branch is 73%. Coverage data for the branch is not yet available.

Show a code coverage summary of the most covered files.
File 2ab2389 +/-
src/lib/shields...nsition-lock.ts — 87% —
src/lib/onboard/preflight.ts — 83% —
src/lib/actions...all/run-plan.ts — 81% —
src/lib/state/o...oard-session.ts — 81% —
src/lib/state/sandbox.ts — 71% —
src/lib/onboard...er-gpu-patch.ts — 69% —
src/lib/shields/index.ts — 68% —
src/lib/policy/index.ts — 66% —
src/lib/actions...licy-channel.ts — 63% —
src/lib/onboard.ts — 24% —

Updated July 06, 2026 18:31 UTC
Code Coverage is in Public Preview. Learn more and provide us with your feedback.

@github-actions

github-actions Bot commented Jul 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

PR Review Advisor (Nemotron Ultra) — No blocking findings

Merge posture: No blocking advisor findings
Primary next action: Fix PRA-1: Early return forces openai-completions for Hermes + compatible-anthropic-endpoint, overriding session preference; then add or justify PRA-T1.
Open items: 1 required · 0 warnings · 3 suggestions · 1 test follow-up
Since last review: 4 prior items resolved · 0 still apply · 0 new items found

Action checklist

  • PRA-1 Fix: Early return forces openai-completions for Hermes + compatible-anthropic-endpoint, overriding session preference in src/lib/actions/inference-route-api.ts:112
  • PRA-T1 Add or justify test follow-up: Mocked behavioral coverage for routing change is sufficient
  • PRA-2 In-scope improvement: No security impact — pure routing logic change in src/lib/actions/inference-route-api.ts:112
  • PRA-3 In-scope improvement: Mocked behavioral coverage for routing change is sufficient in src/lib/actions/inference-route-api.ts:112
  • PRA-4 In-scope improvement: Early return simplifies resolver logic (shrink) in src/lib/actions/inference-route-api.ts:112

Findings index

ID Severity Category Location Required action
PRA-1 Required correctness src/lib/actions/inference-route-api.ts:112 Current behavior is correct per issue #6289. No code change needed. Verify operator expectations match: compatible-anthropic-endpoint providers must use openai-completions format regardless of session preference.
PRA-2 Improvement security src/lib/actions/inference-route-api.ts:112 No action needed. Documenting for audit trail.
PRA-3 Improvement tests src/lib/actions/inference-route-api.ts:112 Current test coverage is sufficient. No additional tests needed.
PRA-4 Improvement architecture src/lib/actions/inference-route-api.ts:112 Accept — simplification improves clarity

🚨 Required before merge

Address these before merging unless a maintainer explicitly overrides the advisor with rationale.

PRA-1 Required — Early return forces openai-completions for Hermes + compatible-anthropic-endpoint, overriding session preference

Review findings by urgency: 1 required fix, 0 items to resolve/justify, 3 in-scope improvements

⚠️ Resolve or justify before merge

Investigate these in the current review; either fix them, explain why they are not applicable, or document the accepted risk.

  • None.

💡 In-scope improvements

These are lower-risk, not throwaway. Prefer fixing them in this PR when they are local to changed code; defer only with rationale or a linked follow-up.

PRA-2 Improvement — No security impact — pure routing logic change

  • Location: src/lib/actions/inference-route-api.ts:112
  • Category: security
  • Problem: All 9 security checklist categories PASS. The change is a 3-line early return in inference API routing logic with no credential handling, auth changes, crypto, input validation, dependencies, or config modifications.
  • Impact: No security regression. The fix actually improves reliability by avoiding a broken streaming path that could cause inference failures.
  • Suggested action: No action needed. Documenting for audit trail.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Review diff: no credentials, auth, crypto, input validation, dependencies, or config changes. Security review applied all 9 categories: Secrets/Credentials PASS, Input Validation PASS, Auth/Authorization PASS, Dependencies PASS, Error Handling PASS, Cryptography PASS, Config/Headers PASS, Security Testing PASS, Holistic Posture PASS
  • Missing regression test: N/A — no security regression test needed
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: Security review applied all 9 categories with PASS verdicts. Diff shows only routing logic change.

PRA-3 Improvement — Mocked behavioral coverage for routing change is sufficient

  • Location: src/lib/actions/inference-route-api.ts:112
  • Category: tests
  • Problem: Test coverage is sufficient for this pure routing logic fix. inference-route-api.test.ts:180-192 tests the resolver directly with mocked config/session. inference-set-hermes-run.test.ts:47-107 tests the full runInferenceSet flow with mocked deps. Both verify the early return behavior.
  • Impact: Behavioral change is covered by existing updated tests.
  • Suggested action: Current test coverage is sufficient. No additional tests needed.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Run test suite: vitest run src/lib/actions/inference-route-api.test.ts src/lib/actions/inference-set-hermes-run.test.ts
  • Missing regression test: Existing tests cover the behavioral change: unit test at inference-route-api.test.ts:180-192 and integration test at inference-set-hermes-run.test.ts:47-107
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: Both test files updated in this PR with explicit coverage for the early return behavior

PRA-4 Improvement — Early return simplifies resolver logic (shrink)

  • Location: src/lib/actions/inference-route-api.ts:112
  • Category: architecture
  • Problem: Early return replaces implicit fallthrough to final fallback returning anthropic-messages for compatible-anthropic-endpoint (previously line 128). Makes Hermes behavior explicit and consistent with getSandboxInferenceConfig's openai-completions branch for compatible-anthropic-endpoint. Reduces cognitive load; no security boundary removed.
  • Impact: Improves code clarity and maintainability by making Hermes-specific behavior explicit rather than relying on implicit fallback.
  • Suggested action: Accept — simplification improves clarity
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Compare resolver logic before/after: previously fell through to line 128 return anthropic-messages; now explicit early return at lines 112-114
  • Missing regression test: N/A — simplification with existing test coverage
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: inference-route-api.ts:112-114 early return; getSandboxInferenceConfig in config.ts already uses openai-completions branch for compatible-anthropic-endpoint when inferenceApi===openai-completions
Simplification opportunities: 1 possible cut

These are safe simplification checks only. Do not remove validation, security controls, data-loss prevention, or required tests.

Test follow-ups to resolve or justify

If these cover changed behavior, prefer adding them in this PR; otherwise state why existing coverage is enough or link the follow-up.

  • PRA-T1 Mocked behavioral coverage for routing change is sufficient — Current test coverage is sufficient. No additional tests needed.
Since last review details

Current findings, using the urgency labels above:

PRA-1 Required — Early return forces openai-completions for Hermes + compatible-anthropic-endpoint, overriding session preference

PRA-2 Improvement — No security impact — pure routing logic change

  • Location: src/lib/actions/inference-route-api.ts:112
  • Category: security
  • Problem: All 9 security checklist categories PASS. The change is a 3-line early return in inference API routing logic with no credential handling, auth changes, crypto, input validation, dependencies, or config modifications.
  • Impact: No security regression. The fix actually improves reliability by avoiding a broken streaming path that could cause inference failures.
  • Suggested action: No action needed. Documenting for audit trail.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Review diff: no credentials, auth, crypto, input validation, dependencies, or config changes. Security review applied all 9 categories: Secrets/Credentials PASS, Input Validation PASS, Auth/Authorization PASS, Dependencies PASS, Error Handling PASS, Cryptography PASS, Config/Headers PASS, Security Testing PASS, Holistic Posture PASS
  • Missing regression test: N/A — no security regression test needed
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: Security review applied all 9 categories with PASS verdicts. Diff shows only routing logic change.

PRA-3 Improvement — Mocked behavioral coverage for routing change is sufficient

  • Location: src/lib/actions/inference-route-api.ts:112
  • Category: tests
  • Problem: Test coverage is sufficient for this pure routing logic fix. inference-route-api.test.ts:180-192 tests the resolver directly with mocked config/session. inference-set-hermes-run.test.ts:47-107 tests the full runInferenceSet flow with mocked deps. Both verify the early return behavior.
  • Impact: Behavioral change is covered by existing updated tests.
  • Suggested action: Current test coverage is sufficient. No additional tests needed.
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Run test suite: vitest run src/lib/actions/inference-route-api.test.ts src/lib/actions/inference-set-hermes-run.test.ts
  • Missing regression test: Existing tests cover the behavioral change: unit test at inference-route-api.test.ts:180-192 and integration test at inference-set-hermes-run.test.ts:47-107
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: Both test files updated in this PR with explicit coverage for the early return behavior

PRA-4 Improvement — Early return simplifies resolver logic (shrink)

  • Location: src/lib/actions/inference-route-api.ts:112
  • Category: architecture
  • Problem: Early return replaces implicit fallthrough to final fallback returning anthropic-messages for compatible-anthropic-endpoint (previously line 128). Makes Hermes behavior explicit and consistent with getSandboxInferenceConfig's openai-completions branch for compatible-anthropic-endpoint. Reduces cognitive load; no security boundary removed.
  • Impact: Improves code clarity and maintainability by making Hermes-specific behavior explicit rather than relying on implicit fallback.
  • Suggested action: Accept — simplification improves clarity
  • Expected follow-up: Prefer a current-PR fix when local to changed code; defer only with rationale or linked follow-up.
  • Verification: Compare resolver logic before/after: previously fell through to line 128 return anthropic-messages; now explicit early return at lines 112-114
  • Missing regression test: N/A — simplification with existing test coverage
  • Done when: The local improvement is applied, or the PR notes why it should be deferred.
  • Evidence: inference-route-api.ts:112-114 early return; getSandboxInferenceConfig in config.ts already uses openai-completions branch for compatible-anthropic-endpoint when inferenceApi===openai-completions

Workflow run details

This is an automated, non-binding review; it still expects maintainers and agents to respond to each required or warning item. Treat suggestions as current-PR improvements when they touch changed code; defer only with maintainer rationale or a linked follow-up. A human maintainer must make the final merge decision.

@github-actions

github-actions Bot commented Jul 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

E2E Advisor Recommendation

Required E2E: hermes-inference-switch, bedrock-runtime-compatible-anthropic
Optional E2E: inference-routing, openclaw-inference-switch

Dispatch hint: hermes-inference-switch,bedrock-runtime-compatible-anthropic

Workflow run

Full advisor summary

E2E Recommendation Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E

  • hermes-inference-switch (high): Directly exercises a real Hermes sandbox through install/onboard, nemoclaw inference set, Hermes config patching, route state, and live in-sandbox chat after switching providers. This is the primary E2E boundary for the changed Hermes compatible Anthropic route-resolution behavior.
  • bedrock-runtime-compatible-anthropic (high): Covers compatible-anthropic-endpoint behavior for OpenAI-compatible Bedrock Runtime adapter routes, including the Hermes matrix lane. The PR changes the exact boundary this test protects: compatible Anthropic endpoints should stay OpenAI-compatible for Hermes rather than forcing Anthropic Messages mode.

Optional E2E

  • inference-routing (medium): Useful adjacent coverage for inference.local routing and provider API-family behavior, but the PR is focused on inference set/Hermes config sync rather than initial provider onboarding or the generic routing negative-path suite.
  • openclaw-inference-switch (high): Optional comparison coverage for OpenClaw inference switching with compatible Anthropic routes. The runtime code change is Hermes-specific, but OpenClaw shares nearby route API resolution logic and documentation was updated for provider switching semantics.

New E2E recommendations

  • Hermes compatible Anthropic inference switching (high): Existing Hermes inference-switch coverage appears oriented around compatible-anthropic-endpoint using Anthropic Messages mode. Add or update a live E2E assertion for the new [DGX Station][Inference] NemoHermes -z fails "no final response" on Anthropic-compatible route — managed proxy emits duplicate SSE message_start #6289 contract: switching Hermes to compatible-anthropic-endpoint keeps base_url at https://inference.local/v1, does not write model.api_mode: anthropic_messages, records preferredInferenceApi: openai-completions, and verifies chat through the managed OpenAI-compatible /v1 proxy.
    • Suggested test: Update test/e2e/live/hermes-inference-switch.test.ts or add a dedicated Hermes compatible-anthropic OpenAI-compatible switch lane.

Dispatch hint

  • Workflow: .github/workflows/e2e.yaml
  • jobs input: hermes-inference-switch,bedrock-runtime-compatible-anthropic

@github-actions

github-actions Bot commented Jul 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

E2E Target Recommendation

Required E2E targets: hermes-inference-switch
Optional E2E targets: bedrock-runtime-compatible-anthropic

Dispatch required E2E targets:

  • gh workflow run e2e.yaml --ref <pr-head-ref> --field jobs=hermes-inference-switch

Workflow run

Full E2E target advisor summary

E2E Target Advisor

Base: origin/main
Head: HEAD
Confidence: high

Required E2E targets

  • hermes-inference-switch: The PR changes Hermes inference route API resolution for compatible-anthropic-endpoint switches, including persisted-session/config behavior. The hermes-inference-switch free-standing E2E job exercises Hermes live onboarding, inference set provider switching, config sync, registry/session state, and in-sandbox inference.local/Hermes API probes for this surface.
    • Dispatch: gh workflow run e2e.yaml --ref <pr-head-ref> --field jobs=hermes-inference-switch

Optional E2E targets

  • bedrock-runtime-compatible-anthropic: Adjacent coverage for compatible Anthropic/Bedrock-style routes on both OpenClaw and Hermes, useful if reviewers want an additional live check that these providers remain OpenAI-compatible through the managed proxy.
    • Dispatch: gh workflow run e2e.yaml --ref <pr-head-ref> --field jobs=bedrock-runtime-compatible-anthropic

Relevant changed files

  • src/lib/actions/inference-route-api.ts

@chengjiew
docs(inference): clarify Hermes compatible Anthropic routing
f7555b3 
Signed-off-by: Chengjie Wang <chengjiew@nvidia.com>
@github-actions

github-actions Bot commented Jul 6, 2026

Copy link
Copy Markdown
Contributor

🌿 Preview your docs: https://nvidia-preview-pr-6295.docs.buildwithfern.com/nemoclaw

coderabbitai[bot]
coderabbitai Bot reviewed Jul 6, 2026
View reviewed changes

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (2)
src/lib/actions/inference-route-api.test.ts (1)

195-218: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low value

Good regression coverage; consider one more case.

Both the updated anthropic-prod test and the new #6289 test correctly exercise the routing function directly and confirm the transition away from the previously-preferred anthropic-messages. One gap: the new test only covers currentProvider === provider (both "compatible-anthropic-endpoint"); since the override in inference-route-api.ts intentionally ignores sameProvider, a case where currentProvider differs (e.g., switching into compatible-anthropic-endpoint from another provider) would confirm the override applies unconditionally, not just when staying on the same provider.

🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/actions/inference-route-api.test.ts` around lines 195 - 218, Add one
more regression test in inference-route-api.test.ts around resolve to cover
switching into compatible-anthropic-endpoint from a different currentProvider,
not just when currentProvider equals provider. Reuse the existing
resolve/session setup and assert that the Hermes-compatible endpoint override
still returns openai-completions even when the provider transition is from
another provider, so the sameProvider-independent behavior in
inference-route-api.ts is exercised.
src/lib/actions/inference-route-api.ts (1)

112-114: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

LGTM — correct fix, but add a comment explaining the override.

The unconditional early return matches the PR objective (avoid the broken Anthropic SSE double message_start on managed compatible endpoints) and is well covered by the new tests. Since this silently overrides any persisted session/config preference (including a user-set anthropic-messages), a short comment referencing the underlying issue would help future maintainers avoid accidentally "fixing" this back to config-driven resolution.

📝 Suggested comment 
   if (provider === "anthropic-prod") return "anthropic-messages";
+  // Hermes on compatible-anthropic-endpoint must stay on the managed
+  // OpenAI-compatible route: the Anthropic SSE path emits a duplicate
+  // message_start, breaking Hermes' "no final response" detection (`#6289`).
   if (agentName === "hermes" && provider === "compatible-anthropic-endpoint") {
     return "openai-completions";
   }
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/actions/inference-route-api.ts` around lines 112 - 114, Add a brief
inline comment in the inference route resolution logic near the `agentName ===
"hermes" && provider === "compatible-anthropic-endpoint"` early return to
explain that this override is intentional. Mention that `resolveInferenceRoute`
(or the corresponding route-selection path in `inference-route-api.ts`) must
force `openai-completions` for Hermes on compatible Anthropic endpoints to avoid
the broken Anthropic SSE double `message_start`, even if a session or config
prefers `anthropic-messages`.
🤖 Prompt for all review comments with AI agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/lib/actions/inference-route-api.test.ts`:
- Around line 195-218: Add one more regression test in
inference-route-api.test.ts around resolve to cover switching into
compatible-anthropic-endpoint from a different currentProvider, not just when
currentProvider equals provider. Reuse the existing resolve/session setup and
assert that the Hermes-compatible endpoint override still returns
openai-completions even when the provider transition is from another provider,
so the sameProvider-independent behavior in inference-route-api.ts is exercised.

In `@src/lib/actions/inference-route-api.ts`:
- Around line 112-114: Add a brief inline comment in the inference route
resolution logic near the `agentName === "hermes" && provider ===
"compatible-anthropic-endpoint"` early return to explain that this override is
intentional. Mention that `resolveInferenceRoute` (or the corresponding
route-selection path in `inference-route-api.ts`) must force
`openai-completions` for Hermes on compatible Anthropic endpoints to avoid the
broken Anthropic SSE double `message_start`, even if a session or config prefers
`anthropic-messages`.
ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: fe095a1e-3953-4162-a371-e585807717a3

📥 Commits

Reviewing files that changed from the base of the PR and between f0d2549 and a971562.

📒 Files selected for processing (3)
  • src/lib/actions/inference-route-api.test.ts
  • src/lib/actions/inference-route-api.ts
  • src/lib/actions/inference-set-hermes-run.test.ts

@github-actions

github-actions Bot commented Jul 6, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

PR Review Advisor — Changes requested

Merge posture: Do not merge yet
Primary next action: Fix PRA-2: Tie the Hermes OpenAI-compatible override to gateway provider registration/validation; then add or justify PRA-T1.
Open items: 1 required · 1 warning · 0 suggestions · 8 test follow-ups
Since last review: 0 prior items resolved · 0 still apply · 1 new item found

Action checklist

  • PRA-2 Fix: Tie the Hermes OpenAI-compatible override to gateway provider registration/validation in src/lib/actions/inference-route-api.ts:112
  • PRA-1 Resolve or justify: Source-of-truth review needed: Hermes `compatible-anthropic-endpoint` runtime API override
  • PRA-T1 Add or justify test follow-up: Mocked behavioral coverage
  • PRA-T2 Add or justify test follow-up: Mocked behavioral coverage
  • PRA-T3 Add or justify test follow-up: Mocked behavioral coverage
  • PRA-T4 Add or justify test follow-up: Mocked behavioral coverage
  • PRA-T5 Add or justify test follow-up: Acceptance clause
  • PRA-T6 Add or justify test follow-up: Acceptance clause
  • PRA-T7 Add or justify test follow-up: Acceptance clause
  • PRA-T8 Add or justify test follow-up: Acceptance clause

Findings index

ID Severity Category Location Required action
PRA-1 Resolve/justify architecture — Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
PRA-2 Required correctness src/lib/actions/inference-route-api.ts:112 Before persisting `openai-completions` for Hermes + `compatible-anthropic-endpoint`, verify the gateway provider metadata is OpenAI-compatible or re-register/replace the provider using the existing OpenAI-surface probe/replace path; if the endpoint only supports `/v1/messages`, fail with an actionable error instead of writing OpenAI-compatible registry/config state. Alternatively, move this fix to the onboarding/provider-registration source so the runtime resolver and gateway provider type cannot diverge.

🚨 Required before merge

Address these before merging unless a maintainer explicitly overrides the advisor with rationale.

PRA-2 Required — Tie the Hermes OpenAI-compatible override to gateway provider registration/validation

  • Location: src/lib/actions/inference-route-api.ts:112
  • Category: correctness
  • Problem: The new Hermes-specific branch forces `compatible-anthropic-endpoint` to `openai-completions` before consulting session/config state, and the updated run test proves NemoClaw then writes `/v1` config and `preferredInferenceApi: "openai-completions"`. However, the linked issue's reproduction starts from onboarding a NemoHermes sandbox on the Custom Anthropic-compatible provider. Nearby onboarding code still validates that selection through the Anthropic path and `src/lib/onboard/inference-providers/remote.ts` registers `compatible-anthropic-endpoint` as `type=anthropic` unless the persisted `preferredInferenceApi` is already `openai-completions`. A gateway provider left in Anthropic-only mode accepts the Anthropic Messages protocol, not OpenAI chat completions, so this PR can replace the duplicate-SSE failure with a protocol mismatch after `inference set` updates NemoClaw registry/config.
  • Impact: A user following issue [DGX Station][Inference] NemoHermes -z fails "no final response" on Anthropic-compatible route — managed proxy emits duplicate SSE message_start #6289's onboard-then-`hermes -z` path may still fail at runtime: Hermes will call the managed `/v1` OpenAI-compatible route while OpenShell may still have `compatible-anthropic-endpoint` registered for the Anthropic Messages surface.
  • Required action: Before persisting `openai-completions` for Hermes + `compatible-anthropic-endpoint`, verify the gateway provider metadata is OpenAI-compatible or re-register/replace the provider using the existing OpenAI-surface probe/replace path; if the endpoint only supports `/v1/messages`, fail with an actionable error instead of writing OpenAI-compatible registry/config state. Alternatively, move this fix to the onboarding/provider-registration source so the runtime resolver and gateway provider type cannot diverge.
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: Read `src/lib/onboard/setup-nim-selection.ts` for the `anthropicCompatible` validation assignment, `src/lib/onboard/inference-providers/remote.ts` around the `useOpenAiSurface` branch, and `src/lib/actions/inference-set.ts` around `registryMetadataForProviderSwitch()`/`resolveRuntimeInferenceApi()` to confirm whether the provider is re-registered before the forced OpenAI route is persisted.
  • Missing regression test: Add a mocked `runInferenceSet`/provider-metadata test where a Hermes sandbox switches to `compatible-anthropic-endpoint` while the gateway provider metadata is still Anthropic-only; assert the code re-registers/verifies an OpenAI provider before writing `preferredInferenceApi: "openai-completions"`, or rejects without changing registry/config. Add a companion onboarding test proving Hermes Custom Anthropic-compatible onboarding either registers `type=openai` after probing `/v1/chat/completions` or rejects endpoints that only serve `/v1/messages`.
  • Done when: The required change is committed and verification passes: Read `src/lib/onboard/setup-nim-selection.ts` for the `anthropicCompatible` validation assignment, `src/lib/onboard/inference-providers/remote.ts` around the `useOpenAiSurface` branch, and `src/lib/actions/inference-set.ts` around `registryMetadataForProviderSwitch()`/`resolveRuntimeInferenceApi()` to confirm whether the provider is re-registered before the forced OpenAI route is persisted.
  • Evidence: Changed code at `src/lib/actions/inference-route-api.ts:112` returns `"openai-completions"` for Hermes + `compatible-anthropic-endpoint`. The changed Hermes run test asserts registry/session/config are updated to OpenAI-compatible values, but no changed code updates the OpenShell provider type. Nearby `remote.ts` only uses OpenAI registration when `preferredInferenceApi === "openai-completions"`, while the Custom Anthropic-compatible onboarding validator persists the Anthropic validation result.
Review findings by urgency: 1 required fix, 1 item to resolve/justify, 0 in-scope improvements

⚠️ Resolve or justify before merge

Investigate these in the current review; either fix them, explain why they are not applicable, or document the accepted risk.

PRA-1 Resolve/justify — Source-of-truth review needed: Hermes `compatible-anthropic-endpoint` runtime API override

  • Location: not file-specific
  • Category: architecture
  • Problem: The advisor marked localized patch analysis as needs_followup.
  • Impact: A localized workaround can preserve or hide an invalid state when the source boundary is unclear.
  • Recommended action: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Missing regression test: Current tests prove local resolver/config outputs, but they do not prove the gateway provider is registered as OpenAI-compatible before the forced `openai-completions` state is persisted.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Evidence: The new branch in `src/lib/actions/inference-route-api.ts` bypasses session/config fallbacks; `src/lib/onboard/inference-providers/remote.ts` only registers `compatible-anthropic-endpoint` as OpenAI when `preferredInferenceApi === "openai-completions"` before registration.

💡 In-scope improvements

These are lower-risk, not throwaway. Prefer fixing them in this PR when they are local to changed code; defer only with rationale or a linked follow-up.

  • None.
Test follow-ups to resolve or justify

If these cover changed behavior, prefer adding them in this PR; otherwise state why existing coverage is enough or link the follow-up.

  • PRA-T1 Mocked behavioral coverage — Hermes compatible-Anthropic switch re-registers or rejects when gateway provider metadata is still Anthropic-only before writing OpenAI-compatible registry/config. The changed unit tests cover the local resolver and Hermes config mutation, but this is provider/protocol routing behavior across registry, session, config, and OpenShell provider metadata boundaries. Behavioral mocks should verify that the forced OpenAI-compatible Hermes state matches the gateway provider registration before relying on it.
  • PRA-T2 Mocked behavioral coverage — Hermes compatible-Anthropic onboarding validates `/v1/chat/completions` and registers provider as `type=openai` when runtime switching will force OpenAI completions. The changed unit tests cover the local resolver and Hermes config mutation, but this is provider/protocol routing behavior across registry, session, config, and OpenShell provider metadata boundaries. Behavioral mocks should verify that the forced OpenAI-compatible Hermes state matches the gateway provider registration before relying on it.
  • PRA-T3 Mocked behavioral coverage — Hermes compatible-Anthropic endpoint that only serves `/v1/messages` fails with an actionable onboarding or inference-set error instead of persisting `preferredInferenceApi: openai-completions`. The changed unit tests cover the local resolver and Hermes config mutation, but this is provider/protocol routing behavior across registry, session, config, and OpenShell provider metadata boundaries. Behavioral mocks should verify that the forced OpenAI-compatible Hermes state matches the gateway provider registration before relying on it.
  • PRA-T4 Mocked behavioral coverage — After switching Hermes compatible-Anthropic provider, a mocked gateway route accepts OpenAI chat-completions protocol rather than only asserting Hermes config file shape. The changed unit tests cover the local resolver and Hermes config mutation, but this is provider/protocol routing behavior across registry, session, config, and OpenShell provider metadata boundaries. Behavioral mocks should verify that the forced OpenAI-compatible Hermes state matches the gateway provider registration before relying on it.
  • PRA-T5 Acceptance clause — NemoHermes run via `hermes -z` on an Anthropic-compatible inference route fails with **"no final response was produced"** and exits 1 in ~9s. — add test evidence or identify existing coverage. The diff changes resolver/config behavior to avoid Hermes Anthropic Messages mode for `compatible-anthropic-endpoint`, and tests assert `/v1` OpenAI-compatible config. It does not prove the onboarded gateway provider accepts OpenAI chat completions or that `hermes -z` succeeds.
  • PRA-T6 Acceptance clause — The root cause is in the managed inference proxy's streaming layer: the SSE stream emits the `message_start` event **twice with an identical message id**, which breaks the streaming client's state machine so it never sees a completed message. — add test evidence or identify existing coverage. The PR sidesteps the Anthropic SSE path by forcing `openai-completions` for Hermes compatible-Anthropic switches. There is no protocol-boundary test showing the resulting OpenAI-compatible route is valid for the existing gateway provider.
  • PRA-T7 Acceptance clause — The same model on the same route returns correct content when called **non-streaming**, so the model and network path are fine — only the streamed event sequence is wrong. — add test evidence or identify existing coverage. No changed test or code verifies non-streaming behavior or validates that the same registered route serves OpenAI chat completions after the forced resolver decision.
  • PRA-T8 Acceptance clause — 1. Onboard a NemoHermes sandbox on the Custom Anthropic-compatible provider (the Inference Hub endpoint, model `nvidia/nvidia/nemotron-3-super-v3`). — add test evidence or identify existing coverage. The changed tests construct mocked Hermes state after onboarding, but the PR does not change or test the onboarding/provider-registration source that creates the Custom Anthropic-compatible route.
Since last review details

Current findings, using the urgency labels above:

PRA-1 Resolve/justify — Source-of-truth review needed: Hermes `compatible-anthropic-endpoint` runtime API override

  • Location: not file-specific
  • Category: architecture
  • Problem: The advisor marked localized patch analysis as needs_followup.
  • Impact: A localized workaround can preserve or hide an invalid state when the source boundary is unclear.
  • Recommended action: Identify the invalid state, source boundary, source-fix constraint, regression test, and removal condition before merging the localized behavior.
  • Expected follow-up: Resolve in this PR or explain why the risk is acceptable.
  • Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Missing regression test: Current tests prove local resolver/config outputs, but they do not prove the gateway provider is registered as OpenAI-compatible before the forced `openai-completions` state is persisted.
  • Done when: The risk is fixed or explicitly justified in the PR. Verification: Inspect the localized patch and source-of-truth review fields for a concrete invalid state, source boundary, source-fix constraint, regression test, and removal condition.
  • Evidence: The new branch in `src/lib/actions/inference-route-api.ts` bypasses session/config fallbacks; `src/lib/onboard/inference-providers/remote.ts` only registers `compatible-anthropic-endpoint` as OpenAI when `preferredInferenceApi === "openai-completions"` before registration.

PRA-2 Required — Tie the Hermes OpenAI-compatible override to gateway provider registration/validation

  • Location: src/lib/actions/inference-route-api.ts:112
  • Category: correctness
  • Problem: The new Hermes-specific branch forces `compatible-anthropic-endpoint` to `openai-completions` before consulting session/config state, and the updated run test proves NemoClaw then writes `/v1` config and `preferredInferenceApi: "openai-completions"`. However, the linked issue's reproduction starts from onboarding a NemoHermes sandbox on the Custom Anthropic-compatible provider. Nearby onboarding code still validates that selection through the Anthropic path and `src/lib/onboard/inference-providers/remote.ts` registers `compatible-anthropic-endpoint` as `type=anthropic` unless the persisted `preferredInferenceApi` is already `openai-completions`. A gateway provider left in Anthropic-only mode accepts the Anthropic Messages protocol, not OpenAI chat completions, so this PR can replace the duplicate-SSE failure with a protocol mismatch after `inference set` updates NemoClaw registry/config.
  • Impact: A user following issue [DGX Station][Inference] NemoHermes -z fails "no final response" on Anthropic-compatible route — managed proxy emits duplicate SSE message_start #6289's onboard-then-`hermes -z` path may still fail at runtime: Hermes will call the managed `/v1` OpenAI-compatible route while OpenShell may still have `compatible-anthropic-endpoint` registered for the Anthropic Messages surface.
  • Required action: Before persisting `openai-completions` for Hermes + `compatible-anthropic-endpoint`, verify the gateway provider metadata is OpenAI-compatible or re-register/replace the provider using the existing OpenAI-surface probe/replace path; if the endpoint only supports `/v1/messages`, fail with an actionable error instead of writing OpenAI-compatible registry/config state. Alternatively, move this fix to the onboarding/provider-registration source so the runtime resolver and gateway provider type cannot diverge.
  • Expected follow-up: Fix before merge or get explicit maintainer override.
  • Verification: Read `src/lib/onboard/setup-nim-selection.ts` for the `anthropicCompatible` validation assignment, `src/lib/onboard/inference-providers/remote.ts` around the `useOpenAiSurface` branch, and `src/lib/actions/inference-set.ts` around `registryMetadataForProviderSwitch()`/`resolveRuntimeInferenceApi()` to confirm whether the provider is re-registered before the forced OpenAI route is persisted.
  • Missing regression test: Add a mocked `runInferenceSet`/provider-metadata test where a Hermes sandbox switches to `compatible-anthropic-endpoint` while the gateway provider metadata is still Anthropic-only; assert the code re-registers/verifies an OpenAI provider before writing `preferredInferenceApi: "openai-completions"`, or rejects without changing registry/config. Add a companion onboarding test proving Hermes Custom Anthropic-compatible onboarding either registers `type=openai` after probing `/v1/chat/completions` or rejects endpoints that only serve `/v1/messages`.
  • Done when: The required change is committed and verification passes: Read `src/lib/onboard/setup-nim-selection.ts` for the `anthropicCompatible` validation assignment, `src/lib/onboard/inference-providers/remote.ts` around the `useOpenAiSurface` branch, and `src/lib/actions/inference-set.ts` around `registryMetadataForProviderSwitch()`/`resolveRuntimeInferenceApi()` to confirm whether the provider is re-registered before the forced OpenAI route is persisted.
  • Evidence: Changed code at `src/lib/actions/inference-route-api.ts:112` returns `"openai-completions"` for Hermes + `compatible-anthropic-endpoint`. The changed Hermes run test asserts registry/session/config are updated to OpenAI-compatible values, but no changed code updates the OpenShell provider type. Nearby `remote.ts` only uses OpenAI registration when `preferredInferenceApi === "openai-completions"`, while the Custom Anthropic-compatible onboarding validator persists the Anthropic validation result.

Workflow run details

This is an automated, non-binding review; it still expects maintainers and agents to respond to each required or warning item. Treat suggestions as current-PR improvements when they touch changed code; defer only with maintainer rationale or a linked follow-up. A human maintainer must make the final merge decision.

@wscurran wscurran added the v0.0.75 Release target label Jul 6, 2026
@apurvvkumaria apurvvkumaria mentioned this pull request Jul 6, 2026
Merged
21 tasks
@cv cv mentioned this pull request Jul 6, 2026
Merged
11 tasks
@cv

cv commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

v0.0.75 release-gate note: CI is green after refreshing this branch, but the current implementation still misses the fresh-onboarding path.

Fresh Hermes custom-Anthropic onboarding validates /v1/messages, persists anthropic-messages, and registers the OpenShell provider as type=anthropic. runInferenceSet later rewrites Hermes-local config/session/registry state to openai-completions, but it does not re-register or replace that provider. Because the provider type controls the protocol, fresh installs can still send OpenAI-shaped traffic through an Anthropic-typed provider.

Before approval, this needs one explicit contract implemented end to end: either (1) compatible-Anthropic for Hermes exposes the OpenAI surface and provider registration is updated safely across onboarding/resume/migration/inference-set, or (2) the stack keeps native Anthropic semantics and fixes/filters the duplicate SSE behavior at the appropriate upstream boundary. The existing green checks do not exercise this mismatch.

@github-actions github-actions Bot mentioned this pull request Jul 6, 2026
Merged
21 tasks
cv pushed a commit that referenced this pull request Jul 6, 2026
@apurvvkumaria @chengjiew
fix(hermes): use OpenAI frontend for custom Anthropic (#6335)
5443538 
## Summary

Fix Hermes custom Anthropic routing by using the endpoint's verified
OpenAI Chat Completions surface end to end. NemoClaw selects
`https://inference.local/v1`, verifies `/v1/chat/completions`, and
aligns the OpenShell provider to `type=openai` with `OPENAI_BASE_URL`,
avoiding the duplicate Anthropic SSE `message_start` sequence that
caused `hermes -z` to finish with `no final response`.

The core managed-frontend direction was first proposed by @chengjiew in
#6295, and Chengjie Wang is included as a commit co-author.
@TonyLuo-NV's #6297 contributed streaming-failure investigation and
regression analysis. Thanks to @hulynn for the reproducible
managed-proxy report.

## Related Issue

Fixes #6289

## Changes

- Resolve Hermes `compatible-anthropic-endpoint` routes to the managed
`openai-completions` frontend and reuse #6298's verified OpenAI-surface
provider registration while retaining `COMPATIBLE_ANTHROPIC_API_KEY` as
the credential binding.
- Persist the normalized frontend during fresh onboarding; repair stale
provider identity, registry metadata, and sandbox configuration during
rebuild or resume.
- Reject conflicting explicit API choices and legacy `type=anthropic`
runtime switches before mutating OpenShell, registry, or in-sandbox
state.
- Preserve native Anthropic Messages routing for OpenClaw custom
endpoints and first-party Anthropic routes; preserve the existing AWS
Bedrock adapter behavior.
- Extend unit, integration, command-shape, and live E2E coverage,
including the reported `hermes -z` path; document the verified-surface
requirement and rebuild migration.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: independent final
review found no blockers. Endpoint probing and provider replacement
reuse the fail-closed #6298 boundary; this PR adds exact non-secret
provider identity checks and introduces no credential values. Human
maintainer approval remains required.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification

- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — 289 CLI tests, 7 OpenAI-surface
onboarding integration tests, 1 focused Hermes config integration test,
and 18 E2E support tests passed
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — required GitHub Actions checks pending
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only) —
completed with 0 errors and 2 existing Fern warnings
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

Additional local verification:

- `npm run build:cli`
- `npm run typecheck:cli`
- `npm run checks`
- `npm run test-size:check`
- `npm run test:titles:check`
- `npm run source-shape:check`
- `npm run test:projects:check`
- `npm run test:imports:check`
- `npm run docs` (0 errors; 2 existing warnings)

---
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>

---------

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Chengjie Wang <chengjiew@nvidia.com>
Co-authored-by: Chengjie Wang <chengjiew@nvidia.com>
@cv

cv commented Jul 6, 2026

Copy link
Copy Markdown
Collaborator

Closing as superseded by #6335, which merged the same Hermes compatible-Anthropic routing fix end to end, explicitly credits #6295, and also adds the required OpenShell provider registration, validation, migration, and broader regression coverage. Rebasing #6295 onto current main resolves all four conflicts to the #6335 implementation and leaves no remaining diff, so a conflict-only merge commit would not add code to v0.0.75.

@cv cv closed this Jul 6, 2026
cv added a commit that referenced this pull request Jul 7, 2026
@TonyLuo-NV @cv
fix(inference): validate custom Anthropic endpoint streaming during o…
6027d7b 
…nboarding (#6289) (#6297)

## Summary

Custom Anthropic-compatible endpoints were validated with a
non-streaming `/v1/messages` probe only, so endpoints whose SSE
streaming layer is malformed (e.g. duplicate `message_start` events with
the same message id, observed on the Inference Hub route) passed
onboarding and then failed at runtime inside the sandbox with Hermes'
cryptic `no final response was produced` (exit 1). This PR adds
streaming-event validation to the custom Anthropic onboarding path —
mirroring the existing `/v1/responses` streaming validation from #1833 —
so the defect is diagnosed at onboarding time with an actionable
message.

## Related Issue

Refs #6289

This PR is the onboarding guardrail for the malformed Anthropic SSE
signature. It intentionally rejects a broken compatible endpoint before
sandbox creation; the functional runtime routing fix remains in #6295.

## Changes

- Add `runAnthropicStreamingEventProbe`
(`src/lib/adapters/http/probe.ts`): sends a `stream: true` request to
`/v1/messages` and validates the SSE event sequence — exactly one
`message_start`, at least one `content_block_delta`, one `message_stop`.
Refactors the shared curl/SSE capture into `captureSseEventCounts`,
reused by the existing `/v1/responses` streaming probe with identical
behavior (trace events, temp cleanup, curl exit-28 tolerance).
- Wire the streaming probe into `probeAnthropicEndpoint` behind a new `{
probeStreaming }` option (`src/lib/inference/probe-anthropic.ts`);
tighter `--max-time 15` timing so validation cannot hang the wizard.
- Enable it in `validateCustomAnthropicSelection`
(`src/lib/onboard/inference-selection-validation.ts`). Skipped when
`NEMOCLAW_REASONING=true`, matching the custom OpenAI-compatible path.
The official Anthropic provider path is unchanged; Bedrock-classified
endpoints are unaffected (streaming probe only runs after the
non-streaming `/v1/messages` probe succeeds).
- Move the two Anthropic credential-retry integration tests into a
focused `test/onboard-selection-anthropic-retry.test.ts` (fake curl now
serves a well-formed SSE stream for `"stream":true` probe bodies),
shrinking `test/onboard-selection.test.ts` below its legacy size budget;
budget ratcheted down 6146 → 5935.
- Docs: validation-table row and Anthropic-compatible server section in
`docs/inference/inference-options.mdx`; new troubleshooting entry for
the onboarding-time duplicate `message_start` failure in
`docs/reference/troubleshooting.mdx`.

## Type of Change

- [x] Code change with doc updates

## Quality Gates

- [x] Tests added or updated for changed behavior
- [x] Docs updated for user-facing behavior changes
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [ ] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: requesting maintainer
review; probes reuse the existing trusted `--config` credential routing
(no credential appears in argv), and the new probe adds no new
credential surface.

## Verification

- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — command/result: `npx vitest run
src/lib/adapters/http/probe.test.ts
src/lib/inference/probe-anthropic.test.ts
src/lib/onboard/inference-selection-validation.test.ts --project cli`
(68 passed); `npx vitest run test/onboard-selection.test.ts --project
integration` (67 passed); `npx vitest run
test/onboard-selection-anthropic-retry.test.ts --project integration` (2
passed); `npx vitest run src/lib/inference/onboard-probes.test.ts
src/lib/inference/onboard-probes-responses-fallback.test.ts --project
cli` (25 passed, 1 skipped); `npm run test:projects:check` (disjoint);
`npm run typecheck:cli` (clean)
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [x] `npm run docs` builds without warnings (doc changes only) — 0
errors; 2 pre-existing warnings on main (login-gated redirects check,
theme contrast), unrelated to this change

## DCO

Signed-off-by: Tony Luo <xialuo@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added Anthropic-compatible streaming SSE validation during onboarding
to catch malformed `/v1/messages` event sequences.
* Streaming validation can be skipped for reasoning-only models via
`NEMOCLAW_REASONING`.
* **Bug Fixes**
* Improved reporting and recovery behavior for missing, duplicate, and
out-of-order streaming events, including clearer diagnostic messaging.
* **Documentation**
* Updated inference validation docs and added troubleshooting guidance
for onboarding failures caused by malformed SSE events.
* **Tests**
* Expanded streaming probe and onboarding validation tests; added
Anthropic onboarding retry UX coverage; removed obsolete Anthropic retry
tests.
* **Chores**
  * Adjusted CI test file size budget.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Tony Luo <xialuo@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Co-authored-by: Carlos Villela <cvillela@nvidia.com>
@wscurran wscurran added area: inference Inference routing, serving, model selection, or outputs area: providers Inference provider integrations and provider behavior bug-fix PR fixes a bug or regression integration: hermes Hermes integration behavior platform: dgx-station Affects DGX Station hardware or workflows labels Jul 7, 2026
Hadar301 pushed a commit to Hadar301/NemoClaw-OpenShift that referenced this pull request Jul 12, 2026
@apurvvkumaria @chengjiew
fix(hermes): use OpenAI frontend for custom Anthropic (NVIDIA#6335)
37b7e96 
## Summary

Fix Hermes custom Anthropic routing by using the endpoint's verified
OpenAI Chat Completions surface end to end. NemoClaw selects
`https://inference.local/v1`, verifies `/v1/chat/completions`, and
aligns the OpenShell provider to `type=openai` with `OPENAI_BASE_URL`,
avoiding the duplicate Anthropic SSE `message_start` sequence that
caused `hermes -z` to finish with `no final response`.

The core managed-frontend direction was first proposed by @chengjiew in
NVIDIA#6295, and Chengjie Wang is included as a commit co-author.
@TonyLuo-NV's NVIDIA#6297 contributed streaming-failure investigation and
regression analysis. Thanks to @hulynn for the reproducible
managed-proxy report.

## Related Issue

Fixes NVIDIA#6289

## Changes

- Resolve Hermes `compatible-anthropic-endpoint` routes to the managed
`openai-completions` frontend and reuse NVIDIA#6298's verified OpenAI-surface
provider registration while retaining `COMPATIBLE_ANTHROPIC_API_KEY` as
the credential binding.
- Persist the normalized frontend during fresh onboarding; repair stale
provider identity, registry metadata, and sandbox configuration during
rebuild or resume.
- Reject conflicting explicit API choices and legacy `type=anthropic`
runtime switches before mutating OpenShell, registry, or in-sandbox
state.
- Preserve native Anthropic Messages routing for OpenClaw custom
endpoints and first-party Anthropic routes; preserve the existing AWS
Bedrock adapter behavior.
- Extend unit, integration, command-shape, and live E2E coverage,
including the reported `hermes -z` path; document the verified-surface
requirement and rebuild migration.

## Type of Change

- [ ] Code change (feature, bug fix, or refactor)
- [x] Code change with doc updates
- [ ] Doc only (prose changes, no code sample modifications)
- [ ] Doc only (includes code sample changes)

## Quality Gates

- [x] Tests added or updated for changed behavior
- [ ] Existing tests cover changed behavior — justification:
- [ ] Tests not applicable — justification:
- [x] Docs updated for user-facing behavior changes
- [ ] Docs not applicable — justification:
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [x] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: independent final
review found no blockers. Endpoint probing and provider replacement
reuse the fail-closed NVIDIA#6298 boundary; this PR adds exact non-secret
provider identity checks and introduces no credential values. Human
maintainer approval remains required.
- [ ] Non-success, skipped, or missing CI check accepted by maintainer —
check name, approval link, and follow-up issue:

## Verification

- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — 289 CLI tests, 7 OpenAI-surface
onboarding integration tests, 1 focused Hermes config integration test,
and 18 E2E support tests passed
- [ ] Applicable broad gate passed — `npm test` for broad
runtime/test-harness changes; `npm run check` for repo-wide
validation/coverage changes — required GitHub Actions checks pending
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [ ] `npm run docs` builds without warnings (doc changes only) —
completed with 0 errors and 2 existing Fern warnings
- [x] Doc pages follow the [style
guide](https://github.com/NVIDIA/NemoClaw/blob/main/docs/CONTRIBUTING.md)
(doc changes only)
- [ ] New doc pages include SPDX header and frontmatter (new pages only)

Additional local verification:

- `npm run build:cli`
- `npm run typecheck:cli`
- `npm run checks`
- `npm run test-size:check`
- `npm run test:titles:check`
- `npm run source-shape:check`
- `npm run test:projects:check`
- `npm run test:imports:check`
- `npm run docs` (0 errors; 2 existing warnings)

---
Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>

---------

Signed-off-by: Apurv Kumaria <akumaria@nvidia.com>
Signed-off-by: Chengjie Wang <chengjiew@nvidia.com>
Co-authored-by: Chengjie Wang <chengjiew@nvidia.com>
Hadar301 pushed a commit to Hadar301/NemoClaw-OpenShift that referenced this pull request Jul 12, 2026
@TonyLuo-NV @cv
fix(inference): validate custom Anthropic endpoint streaming during o…
6aebc37 
…nboarding (NVIDIA#6289) (NVIDIA#6297)

## Summary

Custom Anthropic-compatible endpoints were validated with a
non-streaming `/v1/messages` probe only, so endpoints whose SSE
streaming layer is malformed (e.g. duplicate `message_start` events with
the same message id, observed on the Inference Hub route) passed
onboarding and then failed at runtime inside the sandbox with Hermes'
cryptic `no final response was produced` (exit 1). This PR adds
streaming-event validation to the custom Anthropic onboarding path —
mirroring the existing `/v1/responses` streaming validation from NVIDIA#1833 —
so the defect is diagnosed at onboarding time with an actionable
message.

## Related Issue

Refs NVIDIA#6289

This PR is the onboarding guardrail for the malformed Anthropic SSE
signature. It intentionally rejects a broken compatible endpoint before
sandbox creation; the functional runtime routing fix remains in NVIDIA#6295.

## Changes

- Add `runAnthropicStreamingEventProbe`
(`src/lib/adapters/http/probe.ts`): sends a `stream: true` request to
`/v1/messages` and validates the SSE event sequence — exactly one
`message_start`, at least one `content_block_delta`, one `message_stop`.
Refactors the shared curl/SSE capture into `captureSseEventCounts`,
reused by the existing `/v1/responses` streaming probe with identical
behavior (trace events, temp cleanup, curl exit-28 tolerance).
- Wire the streaming probe into `probeAnthropicEndpoint` behind a new `{
probeStreaming }` option (`src/lib/inference/probe-anthropic.ts`);
tighter `--max-time 15` timing so validation cannot hang the wizard.
- Enable it in `validateCustomAnthropicSelection`
(`src/lib/onboard/inference-selection-validation.ts`). Skipped when
`NEMOCLAW_REASONING=true`, matching the custom OpenAI-compatible path.
The official Anthropic provider path is unchanged; Bedrock-classified
endpoints are unaffected (streaming probe only runs after the
non-streaming `/v1/messages` probe succeeds).
- Move the two Anthropic credential-retry integration tests into a
focused `test/onboard-selection-anthropic-retry.test.ts` (fake curl now
serves a well-formed SSE stream for `"stream":true` probe bodies),
shrinking `test/onboard-selection.test.ts` below its legacy size budget;
budget ratcheted down 6146 → 5935.
- Docs: validation-table row and Anthropic-compatible server section in
`docs/inference/inference-options.mdx`; new troubleshooting entry for
the onboarding-time duplicate `message_start` failure in
`docs/reference/troubleshooting.mdx`.

## Type of Change

- [x] Code change with doc updates

## Quality Gates

- [x] Tests added or updated for changed behavior
- [x] Docs updated for user-facing behavior changes
- [x] Sensitive paths changed (security, policy, credentials, preflight,
onboarding, inference, runner, sandbox, or messaging)
- [ ] Sensitive-path review completed or maintainer-approved waiver
recorded — reviewer/approval link/justification: requesting maintainer
review; probes reuse the existing trusted `--config` credential routing
(no credential appears in argv), and the new probe adds no new
credential surface.

## Verification

- [x] PR description includes the DCO sign-off declaration and every
commit appears as `Verified` in GitHub
- [x] Normal `pre-commit`, `commit-msg`, and `pre-push` hooks passed, or
`npm run check:diff` passed when hooks were skipped or unavailable
- [x] Targeted behavior tests pass for the current change set, or tests
are marked not applicable above — command/result: `npx vitest run
src/lib/adapters/http/probe.test.ts
src/lib/inference/probe-anthropic.test.ts
src/lib/onboard/inference-selection-validation.test.ts --project cli`
(68 passed); `npx vitest run test/onboard-selection.test.ts --project
integration` (67 passed); `npx vitest run
test/onboard-selection-anthropic-retry.test.ts --project integration` (2
passed); `npx vitest run src/lib/inference/onboard-probes.test.ts
src/lib/inference/onboard-probes-responses-fallback.test.ts --project
cli` (25 passed, 1 skipped); `npm run test:projects:check` (disjoint);
`npm run typecheck:cli` (clean)
- [x] Quality Gates section completed with required justifications or
waivers
- [x] No secrets, API keys, or credentials committed
- [x] `npm run docs` builds without warnings (doc changes only) — 0
errors; 2 pre-existing warnings on main (login-gated redirects check,
theme contrast), unrelated to this change

## DCO

Signed-off-by: Tony Luo <xialuo@nvidia.com>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added Anthropic-compatible streaming SSE validation during onboarding
to catch malformed `/v1/messages` event sequences.
* Streaming validation can be skipped for reasoning-only models via
`NEMOCLAW_REASONING`.
* **Bug Fixes**
* Improved reporting and recovery behavior for missing, duplicate, and
out-of-order streaming events, including clearer diagnostic messaging.
* **Documentation**
* Updated inference validation docs and added troubleshooting guidance
for onboarding failures caused by malformed SSE events.
* **Tests**
* Expanded streaming probe and onboarding validation tests; added
Anthropic onboarding retry UX coverage; removed obsolete Anthropic retry
tests.
* **Chores**
  * Adjusted CI test file size budget.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Tony Luo <xialuo@nvidia.com>
Signed-off-by: Carlos Villela <cvillela@nvidia.com>
Co-authored-by: Carlos Villela <cvillela@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@coderabbitai coderabbitai[bot] coderabbitai[bot] left review comments

Assignees

@cv cv

Labels

area: inference Inference routing, serving, model selection, or outputs area: providers Inference provider integrations and provider behavior bug-fix PR fixes a bug or regression integration: hermes Hermes integration behavior platform: dgx-station Affects DGX Station hardware or workflows v0.0.75 Release target

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

[DGX Station][Inference] NemoHermes -z fails "no final response" on Anthropic-compatible route — managed proxy emits duplicate SSE message_start

3 participants

@chengjiew @cv @wscurran