Make SSH endpoint placement dynamic

[wang-tong0]

Summary

• remove the fixed primary/spare SSH endpoint convention and place champion, challenger, and pre-challenger workloads on any available scoring endpoint
• promote pre-deployed challengers in place with an identity-fenced assignment role update instead of tearing down and reloading the model
• transition winning SSH challengers to the champion role before publishing canonical champion state, with identity-fenced and crash-safe retries
• distinguish transient provider failures from permanently stale assignments so a removed or reassigned endpoint drops runtime reuse without blocking the evaluation winner
• persist the active battle before releasing the old champion runtime, with retry guards for teardown failures and single-endpoint model swaps
• deduplicate executor dispatch during the temporary battle/predeployment overlap used for crash-safe promotion
• retain compatibility with legacy assignment_role=active rows and reclaim expired endpoint reservations

Why

The fixed primary-slot convention caused unnecessary endpoint switching and model reloads. It also made cleanup and recovery brittle when a pre-deployed model was promoted or an endpoint changed while teardown was still converging.

Runtime roles are now independent from physical endpoint position. This keeps healthy deployments in place and makes both one-endpoint and multi-endpoint operation explicit.

Validation

• PYTHONPATH=/home/ubuntu/workspace/affine-cortex-dynamic-endpoints pytest -q (793 passed)
• ruff check on all changed Python files
• git diff --check

[catoneone]

Bug: single-endpoint deployments deadlock after the first challenger win

There is a gap in the assignment_role lifecycle:

1. A direct challenger deploy reserves the endpoint with assignment_role="challenger" (try_reserve_assignment); in-place promotion (promote_assignment) also writes "challenger".
2. When the challenger wins, _decide's WON branch only updates scheduler state (set_champion). Nothing rewrites the endpoint row's role to "champion" — the only writers of assignment_role on this branch are reserve and promote.
3. The next challenger's direct deploy hits the replacement fallback in _select_ssh_endpoints (affine/src/scheduler/main.py:184-195), which only matches assignment_role in ("champion", "active"). The only endpoint hosts the current champion but is still labeled "challenger", so selection raises EndpointReservationConflict → _start_battle releases the claim → repeats every tick. Battles never start.

The only self-healing path is the next task-pool refresh: it clears champion runtime state and the champion redeploy goes through the identity-matching path, whose reserve rewrites the role back to "champion". So a one-endpoint setup stalls for up to a full refresh period after every win — "A one-endpoint setup still works by replacing the champion deployment" only holds for the first battle. test_select_ssh_endpoints_challenger_replaces_legacy_active_role covers the legacy "active" label, but not this state, which every win produces.

Minimal repro (passes on this branch):

def test_post_win_single_endpoint_cannot_start_next_battle():
    # Endpoint exactly as left after a challenger win: current champion
    # (uid=2) holds the only endpoint, DB row still says role="challenger".
    endpoint = _EndpointStub(
        "only-endpoint",
        assigned_uid=2,
        assigned_hotkey="winner_hk",
        assigned_model="org/winner",
        assigned_revision="winner_rev",
        assignment_role="challenger",   # never relabeled after the win
        assignment_token="owner",
        assignment_status="ready",
        deployment_id="ssh:only-endpoint:affine-sglang-current",
    )
    target = _target()  # next challenger, different identity -> no match
    with pytest.raises(EndpointReservationConflict):
        _select_ssh_endpoints([endpoint], target, role="challenger")

Suggested fix (either direction works):

• After a win, do one identity-fenced role update of the winner's endpoint to "champion" (e.g. reuse promote_assignment with role="champion", accepting "challenger" as the prior role) in _decide's WON branch or as a promotion follow-up; or
• make the challenger replacement fallback match on "assignment identity equals the current champion's identity" instead of the role label.