fix(status): probe inference.local route for cloud providers

[souvikDevloper]

Summary

nemoclaw <sbx> status and nemoclaw <sbx> doctor reported inference as
healthy / [ok] by probing the upstream provider endpoint directly
(e.g. the NVIDIA Cloud /v1/models endpoint), instead of the
inference.local route the agent actually calls. When inference.local
was broken inside the sandbox but the upstream stayed reachable, both
commands still reported "healthy" — directly contradicting
nemoclaw <sbx> connect, which probes the real route and correctly
reports it BROKEN.

This is the same class of bug as the previously-fixed #3265 ("status
reports Inference: healthy by probing Ollama directly, bypassing the auth
proxy / gateway"). That fix added an inference.local gateway-chain
subprobe, but gated it to ollama-local / vllm-local providers only,
so the cloud / inference.local path still went unchecked.

Fixes #6192.

What changed

The inference.local gateway-chain subprobe is now run for every known
provider, not just local ones, in both call sites:

• src/lib/actions/sandbox/doctor.ts — collectInferenceSubprobes no
  longer early-returns for non-local providers. Unknown providers still
  never reach it (collectInferenceChecks returns early on
  provider === "unknown").
• src/lib/actions/sandbox/status-snapshot.ts — the gateway-chain gate
  changes from ollama-local || vllm-local to currentProvider !== "unknown".

Cloud providers route through the same in-sandbox openclaw gateway / auth
proxy as local ones, so a reachable upstream endpoint does not prove the
agent's route works. A broken inference.local route now surfaces as a
failing Provider health (gateway) check (doctor -> attention needed /
exit 1) and an unhealthy Inference (gateway) line (status), so
status, doctor, and connect agree.

Upstream reachability is still reported on its own line, so no signal is lost.

Testing

• src/lib/actions/sandbox/doctor-flow.test.ts — new test: a cloud
  provider (nvidia-prod) with a reachable upstream but a broken
  inference.local route now produces a failing gateway check and a
  fail summary.
• src/lib/actions/sandbox/status-snapshot.test.ts — new file: the
  gateway subprobe runs for a cloud provider (broken route -> ok: false
  subprobe) and is skipped when the provider is unknown.

Both new tests were confirmed to fail on main (before the fix) and pass
with it. Ran the affected cli-project suites locally: doctor-flow,
status-flow, and status-snapshot.

DCO Sign-Off

Signed-off-by: souvikDevloper gshsouvik01@gmail.com

Summary by CodeRabbit

• Bug Fixes
  • Updated sandbox doctor and status snapshot inference checks to probe the in-app inference gateway chain for every known provider (not only local-style providers), surfacing gateway-route failures as a distinct “gateway chain” failure.
• Documentation
  • Refreshed inference health guidance to clarify the inference.local route is used for all known providers and should be treated as unhealthy when broken.
• Tests
  • Added Vitest coverage ensuring correct behavior for cloud providers vs unknown providers, including gateway-chain probe invocation and expected subprobe/report statuses.
• Chores
  • Added a shared helper to build the “Inference gateway chain” health subprobe.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

Sandbox inference checks now probe the inference.local gateway chain for any known provider, not just local providers. A shared helper builds the gateway subprobe, and tests cover cloud-provider failures plus the unknown provider skip path.

Changes

Inference gateway probe scope fix

Layer / File(s)	Summary
Shared gateway subprobe helper src/lib/inference/health.ts	Adds buildInferenceGatewaySubprobe for constructing the inference gateway chain subprobe from a gateway probe result.
Doctor probes gateway for any known provider src/lib/actions/sandbox/doctor.ts, src/lib/actions/sandbox/doctor-flow.test.ts	Removes local-provider gating from inference subprobe collection, uses the shared helper for gateway results, and adds a flow test where a cloud provider reports upstream health as ok while the in-sandbox gateway probe fails.
Status snapshot probes any known provider src/lib/actions/sandbox/status-snapshot.ts, src/lib/actions/sandbox/status-snapshot.test.ts	Broadens the status snapshot gateway condition to any non-unknown provider, uses the shared helper in the success path, and adds tests for cloud-provider failure and unknown provider skip behavior.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Suggested labels: bug-fix, v0.0.56

Suggested reviewers: jyaunches, prekshivyas, ericksoa

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title is concise and accurately summarizes the main inference.local routing fix for cloud providers.
Linked Issues check	✅ Passed	The PR makes status and doctor probe inference.local for known providers and skips unknown ones, matching the linked route bug fix.
Out of Scope Changes check	✅ Passed	The changes are focused on the inference route fix and matching tests/helpers, with no unrelated feature work.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands.}

[coderabbitai]

🧹 Nitpick comments (2)

src/lib/actions/sandbox/doctor.ts (1)

355-366: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Duplicate subprobe-construction logic with status-snapshot.ts.

This object literal (ok, probed: true, providerLabel: "Inference gateway chain", endpoint, detail, probeLabel: "gateway", conditional failureLabel) is duplicated almost verbatim in src/lib/actions/sandbox/status-snapshot.ts (lines 230-238). Since this PR touches both call sites to broaden the same gating condition, it's a good opportunity to extract a shared helper (e.g. in src/lib/inference/health.ts or a small adapter-facing utility) so future changes to the subprobe shape don't need to be kept in sync manually across two files.

♻️ Proposed shared helper

+// src/lib/inference/health.ts (or similar shared module)
+export function buildInferenceGatewaySubprobe(gateway: {
+  ok: boolean;
+  endpoint: string;
+  detail: string;
+}): ProviderHealthStatus {
+  return {
+    ok: gateway.ok,
+    probed: true,
+    providerLabel: "Inference gateway chain",
+    endpoint: gateway.endpoint,
+    detail: gateway.detail,
+    probeLabel: "gateway",
+    ...(gateway.ok ? {} : { failureLabel: "unreachable" as const }),
+  };
+}

Then both doctor.ts and status-snapshot.ts call buildInferenceGatewaySubprobe(gateway).

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/actions/sandbox/doctor.ts` around lines 355 - 366, The gateway
subprobe object construction in doctor.ts is duplicated in status-snapshot.ts,
so extract the shared shape into a helper and have both call sites use it.
Create a small reusable function such as buildInferenceGatewaySubprobe that
takes the gateway result and returns the `ok`, `probed`, `providerLabel`,
`endpoint`, `detail`, `probeLabel`, and conditional `failureLabel` fields, then
replace the inline object literal in doctor.ts with that helper and update
status-snapshot.ts to use the same helper.

src/lib/actions/sandbox/status-snapshot.test.ts (1)

26-32: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick win

Shared mutable fixture is mutated in place by the code under test.

reachableCloudHealth is a module-level const returned directly (not cloned) by probeProviderHealthImpl (line 77). collectSandboxStatusSnapshot then mutates that exact object's subprobes field in place (status-snapshot.ts line 239). Today only one test in this file exercises that branch, so it happens to pass, but the pattern is fragile: any future test reusing reachableCloudHealth (or a re-run within the same process) would see stale subprobes already appended from a prior invocation, causing spurious failures or false passes.

🛠️ Return a fresh clone per invocation

   const collectSandboxStatusSnapshot: CollectSandboxStatusSnapshot =
     requireDist(snapshotModulePath).collectSandboxStatusSnapshot;

   return {
     probeSandboxInferenceGatewayHealthSpy,
     run: () =>
       collectSandboxStatusSnapshot("alpha", {
         deps: {
           getSandbox: (() => sandboxEntry) as never,
           reconcile: async () => presentLookup,
           probeProviderHealthImpl: (provider: string) =>
-            provider === "unknown" ? null : reachableCloudHealth,
+            provider === "unknown" ? null : { ...reachableCloudHealth },
         },
       }),
   };

Also applies to: 76-79

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@src/lib/actions/sandbox/status-snapshot.test.ts` around lines 26 - 32, The
shared reachableCloudHealth fixture is being returned by probeProviderHealthImpl
and then mutated in collectSandboxStatusSnapshot via its subprobes field, which
makes later tests/state reuse fragile. Update the test setup so
probeProviderHealthImpl returns a fresh cloned ProviderHealthStatus object each
time instead of the module-level reachableCloudHealth instance, and keep the
mutable subprobes append behavior isolated to that per-invocation copy.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@src/lib/actions/sandbox/doctor.ts`:
- Around line 355-366: The gateway subprobe object construction in doctor.ts is
duplicated in status-snapshot.ts, so extract the shared shape into a helper and
have both call sites use it. Create a small reusable function such as
buildInferenceGatewaySubprobe that takes the gateway result and returns the
`ok`, `probed`, `providerLabel`, `endpoint`, `detail`, `probeLabel`, and
conditional `failureLabel` fields, then replace the inline object literal in
doctor.ts with that helper and update status-snapshot.ts to use the same helper.

In `@src/lib/actions/sandbox/status-snapshot.test.ts`:
- Around line 26-32: The shared reachableCloudHealth fixture is being returned
by probeProviderHealthImpl and then mutated in collectSandboxStatusSnapshot via
its subprobes field, which makes later tests/state reuse fragile. Update the
test setup so probeProviderHealthImpl returns a fresh cloned
ProviderHealthStatus object each time instead of the module-level
reachableCloudHealth instance, and keep the mutable subprobes append behavior
isolated to that per-invocation copy.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 6f8d2f9d-1ba9-4e72-9930-ce26d7488b14

📥 Commits

Reviewing files that changed from the base of the PR and between 3f89cd0 and 545f910.

📒 Files selected for processing (4)

• src/lib/actions/sandbox/doctor-flow.test.ts
• src/lib/actions/sandbox/doctor.ts
• src/lib/actions/sandbox/status-snapshot.test.ts
• src/lib/actions/sandbox/status-snapshot.ts

[souvikDevloper]

Addressed both CodeRabbit nitpicks in 790b718:

• Duplicate subprobe construction (doctor.ts / status-snapshot.ts) — extracted a shared buildInferenceGatewaySubprobe helper in src/lib/inference/health.ts; both call sites now use it, so the subprobe shape stays in sync.
• Shared mutable fixture (status-snapshot.test.ts) — probeProviderHealthImpl now returns a fresh { ...reachableCloudHealth } clone per invocation, so the in-place subprobes append no longer leaks across runs.

Behavior and assertions are unchanged; the cli suites (doctor-flow, status-flow, status-snapshot) pass locally and the growth guardrail stays green.

[souvikDevloper]

@cv hope u gonna take a look over these soon.

[cv]

The exact-head advisor found a correctness blocker beyond the formatting failure: when the inference.local gateway subprobe fails, status appends the failed subcheck but leaves the aggregate inferenceHealth.ok value true. JSON and text output can therefore still report healthy even though the route the agent actually uses is broken. Propagate that failure into the aggregate or render an explicit degraded/unhealthy result, and add regression coverage asserting the top-level JSON and text status. Current static-checks also need the createRequire-budget failure resolved.

[souvikDevloper]

Thanks @cv — addressed in ead75a2.

Correctness blocker (aggregate stayed healthy): collectSandboxStatusSnapshot now propagates a failed inference.local gateway subprobe into the aggregate — it sets inferenceHealth.ok = false, a failureLabel, and an explanatory detail ("Provider endpoint is reachable, but the inference.local route the agent uses is broken…"). Because report.inferenceHealth is surfaced verbatim in --json and drives the text summary, both now report unhealthy instead of contradicting connect.

Regression coverage (top-level JSON + text):

• status-snapshot.test.ts — asserts the aggregate degrades to ok: false + failureLabel: "unreachable" when the route is broken, stays ok: true when it is reachable, and skips the probe for unknown providers. That object is exactly what --json emits as inferenceHealth.
• status-flow.test.ts — adds a text-render case asserting the top line shows Inference: unreachable (not Inference: healthy) alongside the Inference (gateway) hop.

createRequire budget: reworked status-snapshot.test.ts off the createRequire seam onto typed dependency injection — collectSandboxStatusSnapshot now accepts injectable captureLiveInference and probeInferenceGateway deps — so it passes test-create-require-budget with native imports. Biome format re-runs clean as well.

[wscurran]

✨ Thanks for the PR. This fixes the health check regression where nemoclaw status and nemoclaw doctor probed the upstream provider instead of the inference.local route the agent actually uses — extending the fix from #3265 to all providers. Ready for maintainer review.

---

Related open issues:

• #6192 [All Platforms][CLI&UX] nemoclaw status/doctor report inference "healthy" by probing the upstream, not the inference.local route the agent uses
• #3265 nemoclaw status reports 'Inference: healthy' by probing Ollama directly on :11434 — bypasses auth proxy (:11435) and gateway

---

Related open issues:

• #6192 [All Platforms][CLI&UX] nemoclaw status/doctor report inference "healthy" by probing the upstream, not the inference.local route the agent uses
• #3265 nemoclaw status reports 'Inference: healthy' by probing Ollama directly on :11434 — bypasses auth proxy (:11435) and gateway

[apurvvkumaria]

Superseded by #6412, which consolidates the complete issue #6192 fix on current main. The typed dependency-injection/test-seam contribution by @souvikDevloper is explicitly retained and credited in #6412 through the signed commit Co-authored-by trailer, the PR description, and the contributor-credit comment. Thank you, Souvik, for the contribution.