Fail over when one assignment rejects an advertised model

[masterkain]

problem

an upstream assignment can advertise a model through discovery and catalog metadata while rejecting inference for that same model. sanitized failure shapes can include:

• structured model_not_found
• HTTP 404 invalid_request_error with param=model and no specific error code

when accounts with different model entitlements share a Pool, this assignment-specific serving rejection can terminate the request even though another assignment in the existing route plan may be able to serve the model

expected behavior

• keep /models as the existing Pool-level union filtered by API-key policy; serving health must not mutate catalog visibility
• classify only canonical assignment-specific model-unavailable outcomes, with catalog provenance required for the code-less invalid_request_error shape
• before any downstream-visible output, retry internally on the next eligible assignment already present in the route plan without rewriting the requested model
• produce one downstream request or connection with ordered assignment attempts; no client retry or reconnect should be required
• record safe model/assignment/route-class lifecycle state through the existing database-backed demotion and circuit mechanisms
• mark an attempt retryable_failed and increment retry count only when an actual handoff occurs
• preserve the final sanitized upstream failure when no later candidate exists
• preserve hard pins for previous-response continuity, file affinity, and already-live upstream websockets
• retain fallback candidates for soft session preferences
• apply failover to direct HTTP, pre-visible SSE terminal events, and pre-visible websocket terminal events
• never retry after visible SSE/websocket output, and leave compact routes outside this behavior
• keep recovery bounded through the existing configurable circuit thresholds and half-open probe lifecycle

catalog visibility and routing eligibility must remain separate: a model backed only by degraded visible assignments may remain listed while inference returns no_eligible_backend

admin observability

make the existing Routing footer block on /admin/upstreams open a read-only panel that shows, per visible Pool assignment:

• advertised model ids and observed/preserved catalog provenance
• per-assignment capabilities, preserving unknown values
• separately labeled HTTP, SSE, and websocket serving signals from bounded Pool-level circuit state

the panel must remain metadata-only, read-only, authorization-bounded, and constant-query with respect to assignment and model count

safety constraints

• no broad 404 => retry rule, message substring matching, or plan/account-label inference
• no fallback for malformed requests, globally invalid models, continuation misses, hard pins, compact routes, or post-visible failures
• no model alias rewrite, catalog exclusion metadata, new process-local health state, or UI-triggered probe/sync/mutation
• no raw provider body, prompt, token, credential, websocket frame, or unsanitized error message in persistence, logs, UI, tests, docs, or evidence

acceptance criteria

• HTTP, SSE, and websocket controlled two-assignment tests prove first-assignment rejection and second-assignment success within one client request/connection
• generic 404, missing-provenance, post-visible, compact, hard-pin, shortlist-exhaustion, and final-candidate controls preserve their defined terminal behavior
• accounting, demotion, circuit state, recovery thresholds, and multi-node visibility are truthful and tested
• degraded-only model visibility remains independent from routing eligibility
• the Routing panel presents safe per-assignment provenance, capabilities, and route-class serving signals without N+1 queries
• focused tests, precommit, quality, controlled smoke, and responsive accessibility checks pass

[masterkain]

Shipped. Mapping to the acceptance criteria:

Gateway behavior — landed in 3e7338a (feat(gateway): fail over assignment-dropped models across transports): shared classifier accepting structured model_not_found at any status plus the code-less HTTP 404 invalid_request_error/param=model shape only with catalog provenance; pre-visible-only internal retry on the next eligible route-plan assignment within one client request/connection across direct HTTP, first-event SSE, and pre-visible websocket; hard pins (continuity, file affinity, live websockets) never escape; compact routes excluded; retryable_failed + retry count only on actual handoff; final sanitized failure preserved when no candidate remains; lifecycle recorded through the existing circuit/demotion mechanisms (upstream_model_unavailable reason, bounded half-open probe recovery). Catalog visibility stays decoupled from routing eligibility.

Hardened in e60fb7c: compact SSE terminal classification, withheld-body-only delivery on exhausted plans (no duplicated relayed blocks), mixed-version owner-forwarding tolerance, narrow nested-envelope error-code resolution.

Two-assignment first-rejects/second-serves E2E tests exist per transport (HTTP/SSE/websocket) along with the generic-404, missing-provenance, post-visible, compact, hard-pin, shortlist-exhaustion, and final-candidate controls.

Supporting invariant — 6b58a89 (fix(catalog): preserve partial upstream sync results) keeps a temporarily unreachable assignment counting as a catalog source, so provenance-gated classification cannot degrade when one account's discovery fetch fails.

Admin observability — first shipped per this spec in 00060a2 (Routing panel: flat deduplicated model list with observed/preserved provenance, pool-scoped badges, per-route-class serving signals, constant-query). Subsequently revised by owner direction in ce1e88b: the panel is now the per-model 5-minute usage leaderboard, and the per-model serving-signal/capability chips were deliberately retired along with their projection — the serving state still lives in the database-backed circuit rows.

Full suite, credo --strict, dialyzer, and sobelow green on main.