Sync to upstream ai-dynamo main (squashed) + bake router policy profiles into images#17
Closed
Pernekhan wants to merge 2 commits into
Closed
Sync to upstream ai-dynamo main (squashed) + bake router policy profiles into images#17Pernekhan wants to merge 2 commits into
Pernekhan wants to merge 2 commits into
Conversation
github-actions
Bot
added
documentation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| except subprocess.TimeoutExpired: | ||
| pytest.fail("building dgd-apply-overrides timed out after 300 seconds") | ||
|
|
||
| if result.returncode != 0: |
| from dynamo.runtime.logging import configure_dynamo_logging | ||
|
|
||
| from .engine import LLMEngine | ||
| from .engine import BaseEngine, RawEngine |
| from dynamo.runtime.logging import configure_dynamo_logging | ||
|
|
||
| from .engine import LLMEngine | ||
| from .engine import BaseEngine, RawEngine |
| # Match runtime_image_tag inside the trtllm: block (first sub-block, | ||
| # e.g. cuda13.1) without colliding with vllm/sglang earlier in the file. | ||
| re.compile( | ||
| r"(?m)(^trtllm:\s*?\n(?:[ \t]+[^\n]*\n)*?[ \t]+runtime_image_tag:\s+)\S+", |
| # Anchored inside the trtllm: block so we don't pick up | ||
| # vllm/sglang runtime_image_tag lines earlier in the file. | ||
| "current_regex": re.compile( | ||
| r"(?m)^trtllm:\s*?\n(?:[ \t]+[^\n]*\n)*?[ \t]+runtime_image_tag:\s*(\S+)\s*$", |
| sys.stdout.flush() | ||
| sys.stderr.flush() | ||
| os._exit(0) | ||
| except BaseException as exc: |
| ): | ||
| global _fpm_trace_invalid_warning_emitted | ||
| if not _fpm_trace_invalid_warning_emitted: | ||
| _fpm_trace_invalid_warning_emitted = True |
| assert engine_args["skip_tokenizer_init"] is False | ||
|
|
||
| # Step 2: resolve. | ||
| spec = resolve_test_logits_processor_spec(lambda: _MockTokenizer()) |
| one (terminal) object; for streaming modalities yield intermediate | ||
| progress objects ending with the terminal one. | ||
| """ | ||
| ... |
| Final chunk must include: ``{"token_ids": [...], "index": 0, | ||
| "finish_reason": "...", "completion_usage": {...}}`` | ||
| """ | ||
| ... |
Single squashed commit bringing in the upstream state this fork's per-request rate limiting depends on (deepinfra/backend#3440): - Router policy classes (--router-policy-config): per-class worker-scaled prefill-busy thresholds, DRR fair-share, and per-worker queue limits; request_queue_limit_per_worker 0 rejects with structured HTTP 529 instead of queueing. - Per-request class selection via the x-dynamo-meta-policy-class header. - DYN_HTTP_OVERLOAD_STATUS_CODE env knob for the rejection status code. No deepinfra-specific patches; content is byte-identical to upstream c3c7e52. Squashed deliberately -- the fork diverges from upstream SHAs from here on. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Gs1wHPe3EHLHDddrWEXBd
Pernekhan
force-pushed
the
claude/dynamo-per-request-rate-limiting-gtuouu
branch
from
July 8, 2026 21:49
c3c7e52 to
7643049
Compare
Pernekhan
changed the title
Add deploy/router-policies/default.yaml (standard bounded-queue class + explicit no-queue class rejecting with 529 instead of queueing, selected per request via x-dynamo-meta-policy-class). frontend, sglang and trtllm images already ship deploy/ at /workspace/deploy; add the missing COPY to the vllm runtime image so the path is uniform. model-manager points frontends at these via DYN_ROUTER_POLICY_CONFIG (deepinfra/backend#3440). lib/kv-router/tests/router_policies.rs validates the baked profiles with the real RouterPolicyConfig parser so a bad edit fails CI instead of crash-looping frontends at startup. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012Gs1wHPe3EHLHDddrWEXBd
Pernekhan
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Overview:
Two commits:
ai-dynamo/dynamomain @c3c7e527— content byte-identical to that upstream state (git diff c3c7e527 764304932is empty). Squashed deliberately; we accept diverging from upstream SHAs.deploy/router-policies/), so frontends need no ConfigMaps or rendered config — the dynamo half of deepinfra/backend#3440.Details:
Baked policy profiles (
deploy/router-policies/default.yaml):standard: default family for all traffic; bounded queue (16 requests / 1M input tokens per discovered worker), busy threshold at dynamo's default (16.0 x max_num_batched_tokens, worker-scaled).no-queue: explicit header-selectable class (x-dynamo-meta-policy-class: no-queue, sent by deepapi for OpenRouter /early_throttle/ guest traffic);request_queue_limit_per_worker: 0→ rejected with structured HTTP 529 the moment all workers are busy, never queued.frontend, sglang and trtllm images already ship
deploy/at/workspace/deploy; the vllm runtime image gets a one-line COPY of justdeploy/router-policiesso the path is uniform:/workspace/deploy/router-policies/. model-manager points frontends at a profile viaDYN_ROUTER_POLICY_CONFIG(env, not flag, so pre-policy images ignore it).CI guard:
lib/kv-router/tests/router_policies.rsvalidates every baked profile with the realRouterPolicyConfigparser (assertsno-queueis explicit/header-selectable, limit 0, unknown class names fall back to the default family) — a bad edit failscargo testinstead of crash-looping frontends at startup.Upstream features the sync brings that backend#3440 depends on: router policy classes /
--router-policy-config,x-dynamo-meta-*header→metadata extraction, andDYN_HTTP_OVERLOAD_STATUS_CODE(ai-dynamo#11249). Note:feat-deepinfra-runtime-v2has the header extraction but not the policy-class system — its deepinfra patches need rebasing onto this sync before building rollout images.Where should the reviewer start?
Commit 2 (
4ea58a41) is the reviewable change:deploy/router-policies/default.yaml,container/templates/vllm_runtime.Dockerfile(one COPY line),lib/kv-router/tests/router_policies.rs.Commit 1 is the upstream delta — verify it's a faithful sync rather than reading it:
Related Issues
🚫 This PR is NOT linked to an issue:
🤖 Generated with Claude Code
https://claude.ai/code/session_012Gs1wHPe3EHLHDddrWEXBd