test(e2e): migrate inference routing provider coverage#3903
Open
jyaunches wants to merge 25 commits into
Open
Conversation
Contributor
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Enterprise Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThis PR completes the scenario framework migration of inference routing and provider E2E coverage. It introduces a Bash primitive library with reusable assertion helpers, nine new domain-specific validation scripts, updates suite configurations and parity mappings, and adds framework tests to validate coverage completeness. ChangesInference Routing E2E Coverage Migration
🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
Suggested labels
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Contributor
PR Review AdvisorRecommendation: info only This is an automated advisory review. A human maintainer must make the final merge decision. Limitations: Advisor execution failed: Could not configure advisor model openai/openai/gpt-5.5 Full advisor summaryPR Review AdvisorBase: PR review advisor failed: Could not configure advisor model openai/openai/gpt-5.5 Gate status
🔴 Blockers
🟡 Warnings
🔵 Suggestions
Acceptance coverage
Security review
Test / E2E status
✅ What looks good
Review completeness
|
Contributor
E2E Advisor RecommendationRequired E2E: Full advisor summaryE2E Recommendation AdvisorBase: Required E2E
Optional E2E
New E2E recommendations
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@specs/2026-05-20_inference-routing-provider-coverage/spec.md`:
- Line 1: Add a top-of-file SPDX license header to this new Markdown spec:
insert an HTML comment at the very beginning containing the copyright holder and
the SPDX identifier for Apache-2.0 (e.g., include a copyright line and
"SPDX-License-Identifier: Apache-2.0") so the file complies with the required
header rule.
In `@specs/2026-05-20_inference-routing-provider-coverage/tests.md`:
- Line 1: This Markdown spec is missing the required SPDX header; add an HTML
comment at the top of the file containing both the SPDX copyright and license
tags, e.g. include a first-line block like <!-- SPDX-FileCopyrightText: 2026
YourOrganizationName --> and <!-- SPDX-License-Identifier: Apache-2.0 -->
(replace YourOrganizationName with the appropriate copyright holder) so the file
(specs/2026-05-20_inference-routing-provider-coverage/tests.md) has the required
SPDX copyright and Apache-2.0 license header.
In `@specs/2026-05-20_inference-routing-provider-coverage/validation.md`:
- Line 1: The file validation.md is missing the required SPDX license header;
add the standard two-line SPDX header at the very top of the file for copyright
and Apache-2.0 license (include SPDX-FileCopyrightText with the project or
copyright owner and SPDX-License-Identifier: Apache-2.0) so the top of
specs/2026-05-20_inference-routing-provider-coverage/validation.md contains the
required SPDX metadata before any content.
In `@test/e2e/validation_suites/lib/inference_routing.sh`:
- Around line 100-120: The function e2e_inference_routing_assert_auth_proxy
doesn't vary the request for different modes and never marks success; update
e2e_inference_routing_assert_auth_proxy to (1) choose and inject the appropriate
authentication for the request based on the mode variable (e.g., add a valid
Authorization header when mode == "valid", omit or send an invalid token when
mode == "invalid" or "unauthenticated") by passing the header/context into the
call to _e2e_inference_status (or by calling a variant that accepts headers),
and (2) call e2e_pass after the case check succeeds; keep the existing status
checks for valid vs invalid modes and ensure the function still returns non-zero
on unknown mode.
In `@test/e2e/validation_suites/suites.yaml`:
- Around line 91-95: Prepend the required SPDX license header to the top of this
YAML file (above the existing "steps" block) so every source includes copyright
and license info; add a copyright line (e.g., "Copyright (c) <YEAR> <Owner>")
followed by "SPDX-License-Identifier: Apache-2.0" as the file header, ensuring
it appears before entries like "steps" and the ids "proxy-reachable" and
"auth-enforcement".
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 60cb7ccd-5c05-4844-9bfc-8cb084c912e0
📒 Files selected for processing (21)
specs/2026-05-20_inference-routing-provider-coverage/spec.mdspecs/2026-05-20_inference-routing-provider-coverage/tests.mdspecs/2026-05-20_inference-routing-provider-coverage/validation.mdtest/e2e/docs/parity-map.yamltest/e2e/runtime/resolver/coverage.tstest/e2e/runtime/run-scenario.shtest/e2e/scenario-framework-tests/e2e-coverage-report.test.tstest/e2e/scenario-framework-tests/e2e-lib-helpers.test.tstest/e2e/scenario-framework-tests/e2e-parity-map.test.tstest/e2e/scenario-framework-tests/e2e-scenario-additional-families.test.tstest/e2e/validation_suites/inference/kimi-compatibility/00-plugin-wiring.shtest/e2e/validation_suites/inference/kimi-compatibility/01-kimi-compatible-models-route.shtest/e2e/validation_suites/inference/model-router/00-healthy-endpoint.shtest/e2e/validation_suites/inference/model-router/01-provider-routed-completion.shtest/e2e/validation_suites/inference/ollama-auth-proxy/01-auth-enforcement.shtest/e2e/validation_suites/inference/routing/00-inference-local-chat-completion.shtest/e2e/validation_suites/inference/routing/01-provider-route-health.shtest/e2e/validation_suites/inference/switch/00-route-state-updated.shtest/e2e/validation_suites/inference/switch/01-switched-inference-local-chat.shtest/e2e/validation_suites/lib/inference_routing.shtest/e2e/validation_suites/suites.yaml
…ference-routing-provider-coverage
…ference-routing-provider-coverage
wscurran
added
E2E
Contributor
|
✨Related open issues: |
wscurran
added
the
enhancement: inference
This was referenced May 20, 2026
cv
approved these changes
May 20, 2026
12 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds inference-routing/provider E2E coverage primitives and domain suites for issue #3812, migrating legacy inference assertions into the scenario framework with parity metadata. The PR also records spec/test/validation artifacts and marks available validation results while leaving live PR/provider evidence blocked until PR context exists.
Related Issue
Fixes #3812
Changes
test/e2e/validation_suites/lib/inference_routing.shhelper coverage for chat completions, provider health, and Ollama auth-proxy checks.test/e2e/validation_suites/suites.yamlandtest/e2e/docs/parity-map.yamlso issue test(e2e): migrate inference routing and provider coverage #3812 legacy assertions are explicitly classified.specs/2026-05-20_inference-routing-provider-coverage/with completion and validation markers.Type of Change
Verification
npx prek run --all-filespassesnpm testpassesmake docsbuilds without warnings (doc changes only)Verification run:
npm test -- test/e2e/scenario-framework-tests/e2e-legacy-assertion-inventory.test.ts test/e2e/scenario-framework-tests/e2e-parity-map.test.ts test/e2e/scenario-framework-tests/e2e-lib-helpers.test.ts test/e2e/scenario-framework-tests/e2e-convention-lint.test.ts test/e2e/scenario-framework-tests/e2e-scenario-resolver.test.ts test/e2e/scenario-framework-tests/e2e-suite-runner.test.ts test/e2e/scenario-framework-tests/e2e-coverage-report.test.ts— passed, 71 tests.ubuntu-repo-cloud-openclaw,gpu-repo-local-ollama-openclaw,ubuntu-repo-cloud-hermes, andbrev-launchable-cloud-openclaw.Known verification gaps:
npx prek run --all-filesfailed in this worktree due broad pre-existing/environment failures, including missingnemoclaw/dist/blueprint/private-networks.js, missingnemoclaw/node_modules/json5, and unrelated CLI test timeouts/assertions.npm testwas not rerun after the broad hook failure; targeted scenario-framework tests passed.Signed-off-by: Julie Yaunches jyaunches@nvidia.com
Summary by CodeRabbit
Release Notes