Skip to content

Refactor GPU endpoint lifecycle operations#582

Merged
catoneone merged 3 commits into
mainfrom
feature/gpu-endpoint-add
Jul 16, 2026
Merged

Refactor GPU endpoint lifecycle operations#582
catoneone merged 3 commits into
mainfrom
feature/gpu-endpoint-add

Conversation

@wang-tong0

@wang-tong0 wang-tong0 commented Jul 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • replace the unrestricted af db set-endpoint command with af db register-static-endpoint
  • stage static endpoints as inactive, preserve assignment/provider runtime fields, reject autoscaler-owned names, probe health, then activate with a conditional write
  • serialize manual GPU add, replace, and remove requests through the running autoscaler daemon so normal scale ticks cannot race operator actions
  • split af gpu add-endpoint from replacement and remove the ambiguous --keep-old path
  • harden tunnel ownership checks, removed-endpoint cleanup, and autoscaler config metadata preservation

Why

The generic database endpoint command could overwrite scheduler assignment and autoscaler ownership fields. Manual GPU lifecycle commands also performed provider work outside the daemon, allowing them to race normal autoscaling and create duplicate or stale endpoint state. This change gives static endpoints and autoscaled endpoints separate ownership paths and makes the daemon the single executor for autoscaled lifecycle operations.

Safety

  • static registrations cannot modify autoscale_managed=True rows
  • runtime assignment and provider metadata are preserved during static config updates
  • activation happens only after the configured SSH/Docker/GPU or Targon API preflight succeeds
  • conditional writes reject ownership changes between staging, probing, and activation
  • manual autoscaler requests use a single leased request, heartbeat, durable result, and cleanup path

Validation

  • combined endpoint lifecycle and scheduler regression suite (347 passed)
  • pytest -q (824 passed)
  • ruff check on changed Python files; existing E402 command-registration imports in affine/cli/main.py ignored
  • git diff --check origin/main...HEAD

@wang-tong0
wang-tong0 marked this pull request as ready for review July 16, 2026 06:29
@catoneone
catoneone merged commit f72a677 into main Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants