fix(evaluator-sdk): align Fabric runtime calls to current nemo-fabric API (enable_relay, adapter ids)#844
Draft
SandyChapman wants to merge 1 commit into
Conversation
… API (enable_relay, adapter ids) Two call sites in the Fabric agent-eval runtime drifted against a newer nemo-fabric (limitations L7 and L8 from the gap analysis) and were masked because every fabric unit test fakes nemo_fabric. L7 — enable_relay: the dropped `config=` keyword now raises, and Fabric wants its own typed relay models. `_relay_config` now builds `RelayObservabilityConfig(atif=RelayAtifConfig(...), atof=RelayAtofConfig(...))` from nemo_fabric's own types and passes it via `enable_relay(observability=...)`, mirroring Fabric's Harbor integration (Fabric owns the schema). L8 — harness adapter ids: current nemo-fabric registers bare `nvidia.fabric.codex` / `nvidia.fabric.hermes` (transport moved to runtime.transport). Swept the retired `.cli`/`.sdk` suffixes across the SDK runtime docstring, examples, notebook, skill-eval, and hermetic tests, plus the nemo-evaluator plugin's FabricRunnerTarget docstring/tests and regenerated spec. Guard against recurrence: add test_fabric_surface.py, a real-nemo_fabric contract test (importorskip elsewhere) that exercises the runtime's own call sites — enable_relay's observability shape and bare-id resolution via the planner — and wire the Linux fabric-wheel-smoke CI job to run it (a bare import could not catch either drift). A new `fabric` path filter triggers that job on runtime/test edits as well as wheel bumps. Stacks on the Fabric-packaging work (#778); shares the same non-prerelease Fabric gate. The fabric unit tests are hermetic and validate the call shape; live ATIF relay capture needs a real Fabric + nemo-relay gateway and was not CI-verified (the smoke job installs wheels but runs no agent). Signed-off-by: Sandy Chapman <schapman@nvidia.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two call sites in the Fabric agent-eval runtime drifted against a newer nemo-fabric (limitations L7 and L8 from the gap analysis). They were masked because every fabric unit test fakes
nemo_fabric, so the fakes kept encoding the old API.L7 —
enable_relaydropped a keyword and changed typesruntime.pycalledcfg.enable_relay(output_dir=..., config=self._relay_config(...)), but current Fabric's signature isenable_relay(*, project=None, output_dir=None, observability=..., components=..., policy=None)— there is noconfig=, and it wants Fabric's ownRelayObservabilityConfig/RelayAtifConfig/RelayAtofConfig._relay_confignow buildsRelayObservabilityConfig(atif=RelayAtifConfig(...), atof=RelayAtofConfig(...))from Fabric's typed models and passes it viaenable_relay(observability=...)— the SDK only configures ATIF/ATOF observability, soobservability=is the semantically clean target. This mirrors nemo-fabric's own Harbor integration (Fabric owns the schema).L8 — harness adapter ids dropped their
.cli/.sdksuffixesCurrent nemo-fabric registers bare
nvidia.fabric.codex/nvidia.fabric.hermes(transport moved toruntime.transport, which the code already sets). Swept the retired suffixes across the SDK runtime docstring, examples, the notebook, skill-eval, and the hermetic tests, plus the nemo-evaluator plugin'sFabricRunnerTargetdocstring/tests (regeneratedopenapi.yaml).Why the drift wasn't caught + the guard
The hermetic tests fake
nemo_fabric, so they can't catch upstream API drift.#778'sfabric-wheel-smokejob installs the real wheels but only did a bareimport— it never touchedenable_relayor adapter resolution, the exact blind spot.test_fabric_surface.py: a real-nemo_fabriccontract test (importorskipeverywhere Fabric isn't installed) that drives the runtime's own call sites —_compose_config→enable_relay(observability=...), and bare-id resolution throughFabric().plan().fabric-wheel-smokejob to run it, and added afabricpath filter so the job also fires on runtime/test edits (not just dependency bumps — though a Fabric wheel bump is itself auv.lock/depschange).Validated locally against a real Fabric build (
script/dev-install-fabric.sh): both assertions pass, and the pre-fix forms fail as expected (oldconfig=→TypeError, oldnvidia.fabric.codex.cli→FabricConfigError).Stacking / gate
fabricextra #778 (base branchfabric-published-wheels/schapman) — the fixes are only correct against the newer Fabric that build(evaluator): install NeMo Fabric from published wheels via a Linuxfabricextra #778 installs, so it shares the same non-prerelease Fabric gate.capture_trajectory=True) needs a real Fabric + thenemo-relaygateway and was not CI-verified — the smoke job installs wheels but runs no agent.Out of scope (flagged)
_common.py(the containerized runtime) still emits the old component-wrapped relay telemetry for the Fabric CLI profile path — a separate mechanism gated behind its own image's Fabric version, not theenable_relaypath. Worth a follow-up to confirm newer Fabric's CLI relay parser still accepts that shape.Test plan
pytest packages/nemo_evaluator_sdk/tests/agent_eval/ -k fabric— 79 passed, 1 skippedpytest plugins/nemo-evaluator/tests/test_agent_evaluate.py— 38 passedtest_fabric_surface.pyagainst a real Fabric build — 2 passedlint-sdk-vendored(mirror in sync)