fix(inference): validate custom Anthropic endpoint streaming during onboarding (#6289)

[TonyLuo-NV]

Summary

Custom Anthropic-compatible endpoints were validated with a non-streaming /v1/messages probe only, so endpoints whose SSE streaming layer is malformed (e.g. duplicate message_start events with the same message id, observed on the Inference Hub route) passed onboarding and then failed at runtime inside the sandbox with Hermes' cryptic no final response was produced (exit 1). This PR adds streaming-event validation to the custom Anthropic onboarding path — mirroring the existing /v1/responses streaming validation from #1833 — so the defect is diagnosed at onboarding time with an actionable message.

Related Issue

Refs #6289

This PR is the onboarding guardrail for the malformed Anthropic SSE signature. It intentionally rejects a broken compatible endpoint before sandbox creation; the functional runtime routing fix remains in #6295.

Changes

• Add runAnthropicStreamingEventProbe (src/lib/adapters/http/probe.ts): sends a stream: true request to /v1/messages and validates the SSE event sequence — exactly one message_start, at least one content_block_delta, one message_stop. Refactors the shared curl/SSE capture into captureSseEventCounts, reused by the existing /v1/responses streaming probe with identical behavior (trace events, temp cleanup, curl exit-28 tolerance).
• Wire the streaming probe into probeAnthropicEndpoint behind a new { probeStreaming } option (src/lib/inference/probe-anthropic.ts); tighter --max-time 15 timing so validation cannot hang the wizard.
• Enable it in validateCustomAnthropicSelection (src/lib/onboard/inference-selection-validation.ts). Skipped when NEMOCLAW_REASONING=true, matching the custom OpenAI-compatible path. The official Anthropic provider path is unchanged; Bedrock-classified endpoints are unaffected (streaming probe only runs after the non-streaming /v1/messages probe succeeds).
• Move the two Anthropic credential-retry integration tests into a focused test/onboard-selection-anthropic-retry.test.ts (fake curl now serves a well-formed SSE stream for "stream":true probe bodies), shrinking test/onboard-selection.test.ts below its legacy size budget; budget ratcheted down 6146 → 5935.
• Docs: validation-table row and Anthropic-compatible server section in docs/inference/inference-options.mdx; new troubleshooting entry for the onboarding-time duplicate message_start failure in docs/reference/troubleshooting.mdx.

Type of Change

• Code change with doc updates

Quality Gates

• Tests added or updated for changed behavior
• Docs updated for user-facing behavior changes
• Sensitive paths changed (security, policy, credentials, preflight, onboarding, inference, runner, sandbox, or messaging)
• Sensitive-path review completed or maintainer-approved waiver recorded — reviewer/approval link/justification: requesting maintainer review; probes reuse the existing trusted --config credential routing (no credential appears in argv), and the new probe adds no new credential surface.

Verification

• PR description includes the DCO sign-off declaration and every commit appears as Verified in GitHub
• Normal pre-commit, commit-msg, and pre-push hooks passed, or npm run check:diff passed when hooks were skipped or unavailable
• Targeted behavior tests pass for the current change set, or tests are marked not applicable above — command/result: npx vitest run src/lib/adapters/http/probe.test.ts src/lib/inference/probe-anthropic.test.ts src/lib/onboard/inference-selection-validation.test.ts --project cli (68 passed); npx vitest run test/onboard-selection.test.ts --project integration (67 passed); npx vitest run test/onboard-selection-anthropic-retry.test.ts --project integration (2 passed); npx vitest run src/lib/inference/onboard-probes.test.ts src/lib/inference/onboard-probes-responses-fallback.test.ts --project cli (25 passed, 1 skipped); npm run test:projects:check (disjoint); npm run typecheck:cli (clean)
• Quality Gates section completed with required justifications or waivers
• No secrets, API keys, or credentials committed
• npm run docs builds without warnings (doc changes only) — 0 errors; 2 pre-existing warnings on main (login-gated redirects check, theme contrast), unrelated to this change

DCO

Signed-off-by: Tony Luo xialuo@nvidia.com

Summary by CodeRabbit

• New Features
  • Added Anthropic-compatible streaming SSE validation during onboarding to catch malformed /v1/messages event sequences.
  • Streaming validation can be skipped for reasoning-only models via NEMOCLAW_REASONING.
• Bug Fixes
  • Improved reporting and recovery behavior for missing, duplicate, and out-of-order streaming events, including clearer diagnostic messaging.
• Documentation
  • Updated inference validation docs and added troubleshooting guidance for onboarding failures caused by malformed SSE events.
• Tests
  • Expanded streaming probe and onboarding validation tests; added Anthropic onboarding retry UX coverage; removed obsolete Anthropic retry tests.
• Chores
  • Adjusted CI test file size budget.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Adds Anthropic-compatible onboarding streaming validation for /v1/messages, including SSE event counting, sequence checks, probe wiring, diagnostic summaries, retry coverage, and updated documentation. NEMOCLAW_REASONING disables the streaming validation path.

Changes

Anthropic Streaming Validation

Layer / File(s)	Summary
Shared SSE capture and streaming probe implementation src/lib/adapters/http/probe.ts, src/lib/adapters/http/probe.test.ts	Adds shared SSE capture, occurrence counting, sequence validation, and the exported Anthropic streaming probe, with tests covering valid streams, malformed ordering, missing events, timeouts, and trace cleanup.
probeAnthropicEndpoint streaming integration src/lib/inference/probe-anthropic.ts, src/lib/inference/probe-anthropic.test.ts	Adds probe options, centralizes /v1/messages payload generation, and conditionally runs the streaming SSE probe with dedicated timing args and failure handling, with tests for invocation and error paths.
Onboarding validation wiring and streaming diagnostics src/lib/onboard/inference-selection-validation.ts, src/lib/onboard/probe-diagnostics.ts, src/lib/onboard/inference-selection-validation.test.ts, src/lib/onboard/probe-diagnostics.test.ts	Passes probeStreaming based on NEMOCLAW_REASONING, maps safe diagnostic codes to display text, and verifies streaming-enabled, streaming-skipped, malformed-stream, and summary-display behavior.
End-to-end onboarding retry tests test/onboard-selection-anthropic-retry.test.ts, test/onboard-selection.test.ts, ci/test-file-size-budget.json	Adds end-to-end Anthropic retry coverage in a new test file, removes the old duplicated retry cases and helper, and updates the file-size budget.
Documentation and troubleshooting updates docs/inference/inference-options.mdx, docs/reference/troubleshooting.mdx	Documents the two-step Anthropic-compatible validation flow, the SSE sequence requirements, the runtime-vs-onboarding distinction, and the new troubleshooting guidance.

Estimated code review effort: 4 (Complex) | ~75 minutes

Suggested labels: bug-fix, area: cli

Suggested reviewers: jyaunches, ericksoa, prekshivyas

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: adding streaming validation for custom Anthropic onboarding.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (2)

src/lib/inference/probe-anthropic.test.ts (1)

89-89: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Stub the negative-call streaming spies.

vi.spyOn() calls through by default, so if this regression ever occurs the test can invoke the real curl-backed streaming probe before failing.
Mock a harmless return value while still asserting not.toHaveBeenCalled().

Also applies to: 180-180

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/inference/probe-anthropic.test.ts` at line 89, The streaming spy
setup in probe-anthropic.test.ts currently uses vi.spyOn() without stubbing, so
it can call the real curl-backed probe before the assertion fails. Update the
negative-call tests around runAnthropicStreamingEventProbe (and the other
matching spy at the second occurrence) to return a harmless mocked value while
still verifying not.toHaveBeenCalled(), so the test never executes the real
streaming implementation.

test/onboard-selection-anthropic-retry.test.ts (1)

75-75: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Test titles missing required issue-ref suffix.

Neither test title ends with a local issue reference like (#6289).

As per coding guidelines, "Root-level integration tests under test/ should import source code, use ESM imports, and use behavior-oriented titles with local issue refs in a final (#1234) suffix."

Also applies to: 153-153

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@test/onboard-selection-anthropic-retry.test.ts` at line 75, The test titles
in this suite are missing the required local issue-reference suffix. Update the
behavior-oriented titles in the affected `it(...)` blocks, including the one in
`onboard-selection-anthropic-retry.test.ts` and the other referenced test, so
each ends with a final local issue tag like `(`#6289`)`. Keep the titles
descriptive and ensure the suffix is appended exactly once at the end of each
test name.

Source: Coding guidelines

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@src/lib/adapters/http/probe.ts`:
- Line 595: The Anthropic stream validation in probe.ts is currently only
checking event presence/counts, so malformed sequences can still pass. Update
the validation around the capture helper and the Anthropic stream checks to
preserve the ordered event list, add message_stop to the singleton-event
enforcement alongside SINGLETON_ANTHROPIC_STREAMING_EVENTS, and assert the
stream order is valid: message_start must come before any content deltas, and
message_stop must be the final terminal event after all content.

---

Nitpick comments:
In `@src/lib/inference/probe-anthropic.test.ts`:
- Line 89: The streaming spy setup in probe-anthropic.test.ts currently uses
vi.spyOn() without stubbing, so it can call the real curl-backed probe before
the assertion fails. Update the negative-call tests around
runAnthropicStreamingEventProbe (and the other matching spy at the second
occurrence) to return a harmless mocked value while still verifying
not.toHaveBeenCalled(), so the test never executes the real streaming
implementation.

In `@test/onboard-selection-anthropic-retry.test.ts`:
- Line 75: The test titles in this suite are missing the required local
issue-reference suffix. Update the behavior-oriented titles in the affected
`it(...)` blocks, including the one in
`onboard-selection-anthropic-retry.test.ts` and the other referenced test, so
each ends with a final local issue tag like `(`#6289`)`. Keep the titles
descriptive and ensure the suffix is appended exactly once at the end of each
test name.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 5ad66d1a-0eed-4771-aedb-7ce60ed88580

📥 Commits

Reviewing files that changed from the base of the PR and between f0d2549 and a25a109.

📒 Files selected for processing (11)

• ci/test-file-size-budget.json
• docs/inference/inference-options.mdx
• docs/reference/troubleshooting.mdx
• src/lib/adapters/http/probe.test.ts
• src/lib/adapters/http/probe.ts
• src/lib/inference/probe-anthropic.test.ts
• src/lib/inference/probe-anthropic.ts
• src/lib/onboard/inference-selection-validation.test.ts
• src/lib/onboard/inference-selection-validation.ts
• test/onboard-selection-anthropic-retry.test.ts
• test/onboard-selection.test.ts

💤 Files with no reviewable changes (1)

• test/onboard-selection.test.ts

[coderabbitai]

♻️ Duplicate comments (1)

src/lib/adapters/http/probe.ts (1)

617-629: 🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Validate message_stop against all known content events, not only deltas.

Line 626 only compares message_stop to the last content_block_delta, so streams like message_start → content_block_delta → message_stop → content_block_stop can still pass even though message_stop is not terminal for the known Anthropic sequence. Check the first/last index of all Anthropic content/message payload events while still ignoring unknown interleavings like ping.

Proposed fix

+const ANTHROPIC_CONTENT_STREAMING_EVENTS = [
+  "content_block_start",
+  "content_block_delta",
+  "content_block_stop",
+  "message_delta",
+];
+
 function anthropicSequenceErrors(eventSequence: string[]): string[] {
   const errors: string[] = [];
   const firstStart = eventSequence.indexOf("message_start");
-  const firstDelta = eventSequence.indexOf("content_block_delta");
-  const lastDelta = eventSequence.lastIndexOf("content_block_delta");
+  const contentIndexes = eventSequence
+    .map((event, index) => (ANTHROPIC_CONTENT_STREAMING_EVENTS.includes(event) ? index : -1))
+    .filter((index) => index >= 0);
+  const firstContent = contentIndexes[0] ?? -1;
+  const lastContent = contentIndexes[contentIndexes.length - 1] ?? -1;
   const lastStop = eventSequence.lastIndexOf("message_stop");
-  if (firstDelta < firstStart) {
-    errors.push("content_block_delta before message_start");
+  if (firstContent >= 0 && firstContent < firstStart) {
+    errors.push("content event before message_start");
   }
-  if (lastStop < lastDelta) {
-    errors.push("content_block_delta after message_stop");
+  if (lastContent >= 0 && lastStop < lastContent) {
+    errors.push("content event after message_stop");
   }
   return errors;
 }

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/adapters/http/probe.ts` around lines 617 - 629, The Anthropic
sequence validation in anthropicSequenceErrors only checks message_stop against
content_block_delta, so it can miss non-terminal stops when other known events
appear after it. Update the logic to compare message_stop against the full set
of known Anthropic content/message payload events in eventSequence, using the
relevant first/last index checks in anthropicSequenceErrors, while continuing to
ignore unknown interleavings like ping.

🧹 Nitpick comments (1)

src/lib/adapters/http/probe.test.ts (1)

871-946: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Minor duplication: repeated argv literal across test cases.

The same ["-sS", "--max-time", "15", "https://example.test/v1/messages"] array is repeated in each new test case. Extracting it to a shared const would reduce duplication without affecting behavioral coverage.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/adapters/http/probe.test.ts` around lines 871 - 946, The new probe
tests repeat the same Anthropic streaming argv literal in multiple cases, so
extract that shared argument array into a single const in probe.test.ts and
reuse it across the affected runAnthropicStreamingEventProbe calls. Keep the
existing test behavior the same, and use the shared symbol near mockStreaming
and runAnthropicStreamingEventProbe to make the repeated setup easier to
maintain.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Duplicate comments:
In `@src/lib/adapters/http/probe.ts`:
- Around line 617-629: The Anthropic sequence validation in
anthropicSequenceErrors only checks message_stop against content_block_delta, so
it can miss non-terminal stops when other known events appear after it. Update
the logic to compare message_stop against the full set of known Anthropic
content/message payload events in eventSequence, using the relevant first/last
index checks in anthropicSequenceErrors, while continuing to ignore unknown
interleavings like ping.

---

Nitpick comments:
In `@src/lib/adapters/http/probe.test.ts`:
- Around line 871-946: The new probe tests repeat the same Anthropic streaming
argv literal in multiple cases, so extract that shared argument array into a
single const in probe.test.ts and reuse it across the affected
runAnthropicStreamingEventProbe calls. Keep the existing test behavior the same,
and use the shared symbol near mockStreaming and runAnthropicStreamingEventProbe
to make the repeated setup easier to maintain.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 15ff264c-0ced-4524-8ff1-975383c94247

📥 Commits

Reviewing files that changed from the base of the PR and between a25a109 and bbe7400.

📒 Files selected for processing (4)

• src/lib/adapters/http/probe.test.ts
• src/lib/adapters/http/probe.ts
• src/lib/inference/probe-anthropic.test.ts
• test/onboard-selection-anthropic-retry.test.ts

🚧 Files skipped from review as they are similar to previous changes (2)

• src/lib/inference/probe-anthropic.test.ts
• test/onboard-selection-anthropic-retry.test.ts

[coderabbitai]

🧹 Nitpick comments (1)

src/lib/inference/probe-anthropic.test.ts (1)

90-97: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Repeated streaming-probe mock payload boilerplate across tests.

The { ok, curlStatus, missingEvents, duplicateEvents, sequenceErrors, message } shape is duplicated across five tests with only a couple of fields varying. A small factory (e.g., makeStreamingResult(overrides)) would reduce duplication and make future field additions (like curlStatus, added in this PR) a one-place change.

♻️ Suggested helper

function streamingResult(
  overrides: Partial<ReturnType<typeof probe.runAnthropicStreamingEventProbe>> = {},
) {
  return {
    ok: true,
    curlStatus: 0,
    missingEvents: [],
    duplicateEvents: [],
    sequenceErrors: [],
    message: "",
    ...overrides,
  };
}

Also applies to: 126-133, 166-176, 203-210, 240-247

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/inference/probe-anthropic.test.ts` around lines 90 - 97, The
streaming probe tests duplicate the same mock result shape in multiple places,
so replace the repeated inline objects in probe-anthropic.test.ts with a small
helper like a streamingResult/makeStreamingResult factory near the existing
runAnthropicStreamingEventProbe spies. Use that helper in each test and pass
only the per-test overrides, keeping the default fields centralized so future
additions like curlStatus only need one update.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/lib/inference/probe-anthropic.test.ts`:
- Around line 90-97: The streaming probe tests duplicate the same mock result
shape in multiple places, so replace the repeated inline objects in
probe-anthropic.test.ts with a small helper like a
streamingResult/makeStreamingResult factory near the existing
runAnthropicStreamingEventProbe spies. Use that helper in each test and pass
only the per-test overrides, keeping the default fields centralized so future
additions like curlStatus only need one update.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: f0066ab4-c793-499d-a53d-4ef7b9aca8e5

📥 Commits

Reviewing files that changed from the base of the PR and between e158191 and 31ca7a8.

📒 Files selected for processing (4)

• src/lib/adapters/http/probe.test.ts
• src/lib/adapters/http/probe.ts
• src/lib/inference/probe-anthropic.test.ts
• src/lib/inference/probe-anthropic.ts

🚧 Files skipped from review as they are similar to previous changes (3)

• src/lib/adapters/http/probe.test.ts
• src/lib/inference/probe-anthropic.ts
• src/lib/adapters/http/probe.ts

[coderabbitai]

🧹 Nitpick comments (1)

src/lib/onboard/probe-diagnostics.test.ts (1)

27-44: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Test doesn't exercise the added diagnosticCodes field.

diagnosticCodes was added to this failure object (Line 36), but since curlStatus: 28 is non-zero, summarizeProbeFailureForDisplay returns before the diagnostics branch runs. The assertions only check the pre-existing curl-exit summary, so this addition isn't actually verified by the test. Consider removing it here if it's not meant to test anything, or moving this coverage to the new streaming test below.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/onboard/probe-diagnostics.test.ts` around lines 27 - 44, The new
diagnosticCodes field in this probe test is not actually being exercised because
summarizeProbeFailureForDisplay exits early on the curlStatus path. Update the
probe diagnostics tests so this case either removes the unused diagnosticCodes
entry from the curl/timeout failure fixture, or move that assertion into the
streaming-related test that reaches the diagnostics branch in
summarizeProbeForDisplay/summarizeProbeFailureForDisplay.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/lib/onboard/probe-diagnostics.test.ts`:
- Around line 27-44: The new diagnosticCodes field in this probe test is not
actually being exercised because summarizeProbeFailureForDisplay exits early on
the curlStatus path. Update the probe diagnostics tests so this case either
removes the unused diagnosticCodes entry from the curl/timeout failure fixture,
or move that assertion into the streaming-related test that reaches the
diagnostics branch in summarizeProbeForDisplay/summarizeProbeFailureForDisplay.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: aa1f675e-1dd9-4ed5-b36f-987053369a5f

📥 Commits

Reviewing files that changed from the base of the PR and between 31ca7a8 and 939498a.

📒 Files selected for processing (5)

• src/lib/inference/probe-anthropic.test.ts
• src/lib/inference/probe-anthropic.ts
• src/lib/onboard/inference-selection-validation.test.ts
• src/lib/onboard/probe-diagnostics.test.ts
• src/lib/onboard/probe-diagnostics.ts

🚧 Files skipped from review as they are similar to previous changes (3)

• src/lib/onboard/inference-selection-validation.test.ts
• src/lib/inference/probe-anthropic.test.ts
• src/lib/inference/probe-anthropic.ts

[cv]

Maintainer review: the bounded SSE probe validates the expected event sequence, preserves credential-file handling, and has focused malformed-stream and timeout coverage. CI, DCO, commit verification, and review-thread checks are clean.