diff --git a/README.md b/README.md index b742543f..4a128539 100644 --- a/README.md +++ b/README.md @@ -88,7 +88,8 @@ for supported combinations and limitations. scheduling, multiple CUDA streams, reusable buffers, and CUDA Graph capture of decode recurrent cores. Asynchronous self-speculation also supports compatible paged FlashAttention backends and recurrent/coda CUDA Graphs; see the - [execution scope and constraints](docs/async-speculative.md#cuda-graph-scope-and-ownership). + [sync vs async timeline](docs/async-speculative.md#timeline-synchronous-versus-asynchronous-speculation) + and [execution scope and constraints](docs/async-speculative.md#cuda-graph-scope-and-ownership). - **Offline and online inference.** A Python API, a command-line interface, and an OpenAI-compatible completions endpoint with streaming, greedy decoding, and seeded top-k/top-p sampling. diff --git a/docs/async-speculative.md b/docs/async-speculative.md index b6fe4557..0c61088c 100644 --- a/docs/async-speculative.md +++ b/docs/async-speculative.md @@ -6,6 +6,127 @@ backends can consume its GPU-owned context lengths; `auto` selects against this capability as well as hardware, dtype and block size. Draft depth is configurable, with `d=2, D=4` as the initial validation configuration. The target depth must match the model. +## Timeline: synchronous versus asynchronous speculation + +Cross-round asynchronous speculation keeps the next token and position on the +GPU, so the next round can compute while the previous round's results return to +the CPU. The comparison below shows the dependency change within one stage. + + + +[Open full-size PNG](diagrams/async-speculative-timeline.png) · +[Open editable SVG](diagrams/async-speculative-timeline.svg) · +[Interactive diagram viewer](diagrams/index.html#async-speculative-timeline) + +The panels follow **the same request through two speculative rounds**, using the +same illustrative time scale and equal draft/verify widths. **Round 0 / Round 1** +identify speculative iterations; **R0–R3 identify zero-based recurrent depths**. +The example uses draft depth 2 and target depth 4. Draft repeats the shallow +R0 → R1 path for its inputs; verification extends their retained hidden states +through R2 → R3. Prelude and coda are included in the draft/verify blocks. +Durations are explanatory, not benchmark results or a predicted speedup. + +**Round boundaries.** `Round N start` marks the beginning of CPU submission; +`Round N end` marks completion of CPU Commit, including ticket retirement on the +asynchronous path. These are host-visible lifecycle boundaries, not GPU execution +boundaries. Shared markers explicitly name both events, such as synchronous +`Round 0 end / Round 1 start`. Asynchronous Round 1 starts before Round 0 ends. +`compute_done` marks device completion and `copy_done` marks result-transfer +completion; neither alone is the end of the round. The compact legend below the +timelines defines these terms without adding another comparison panel. + +**Match the purple blocks.** Accept/update determines the accepted prefix and +next input token/position. In the synchronous path, this spans CPU greedy +acceptance inside `collect()` and the scheduler's subsequent request progress +update. In the asynchronous path, acceptance and the next token/position +are computed on the GPU. CPU output delivery and EOS/length handling remain on +the CPU, represented separately by the blue **Commit** block. + +**What Commit does.** After readback, the CPU interprets the already-decided +result, records accepted output tokens in the Request, reconciles request progress +and reservations, and checks EOS/length limits. In the asynchronous path, the +engine then retires the round ticket so its buffer can be reused. This block +groups collection's host processing and the engine/scheduler update; it is a +diagram label, not a new runtime API. It does not redo GPU acceptance. The +synchronous path performs the corresponding CPU output/progress update too, +although its KV bookkeeping and submission lifetime differ. + +**Read the comparison from matching blocks.** Green Draft/Verify blocks represent +the same model work. Purple Accept blocks move from CPU to GPU; blue Commit blocks +remain on the CPU. In sync, Round 1 GPU computation starts only after Round 0's +readback, acceptance, commit and next submission. In async, Round 1 GPU computation +starts once GPU acceptance has updated +its input state, while Round 0's D2H and CPU commit proceed alongside it. + +The gold **T** guide marks the end of synchronous **Round 1 CPU commit** and +continues at the same horizontal coordinate into the asynchronous panel. The +horizontal arrow connects asynchronous Round 1's earlier commit to T, comparing +completion of the same two rounds. This is CPU result commitment, not the +earlier GPU-completion event. + +The example draws GPU acceptance/update slightly shorter than CPU acceptance, +so the first round's result delivery is only slightly earlier. Subsequent rounds +also overlap prior readback and CPU work, making the completion gap accumulate. +Shorter GPU acceptance is an illustrative assumption, not a measured property or +a guarantee: batch size, launch overhead and implementation determine the actual +cost. Draft/verify durations are equal across panels. The figure explains the +removed host dependency without predicting a speedup or first-result latency. + +Both panels use the same three lanes: + +| Lane | Synchronous | Asynchronous | +| --- | --- | --- | +| CPU | Submit, wait for computation and IDs, accept, apply request progress, then submit the next round | Submit ahead, collect results in order, apply outputs and retire tickets | +| `compute_stream` | Draft, verify, then produce/read back IDs inside `collect()`; wait for CPU progress before the next draft | Draft, verify, accept and update device state; the next draft consumes that state in stream order | +| `copy_stream` | Unused for this result readback | Wait for `compute_done`, copy compact results into pinned host memory, record `copy_done` | + +**Synchronous dependency.** Verification must finish before `collect()` produces +and reads the target/candidate IDs. CPU acceptance then determines the accepted +prefix and correction/bonus token. The scheduler commits progress and truncates +the rejected KV suffix before the next round can be submitted. The orange gap on +`compute_stream` is this CPU-dependent interval. The ID readback is drawn on +`compute_stream`, matching the stream context around `_collect_round()`; it is +not an independent asynchronous copy-stream operation. + +**Asynchronous dependency.** Round 1 can be submitted before Round 0 is collected. +Its GPU computation still follows Round 0's acceptance and token/position update. +Meanwhile, `copy_stream` returns Round 0's result. The dashed completion guides +align each CPU wait endpoint with the end of its corresponding D2H transfer. +This removes CPU delivery from the immediate inter-round dependency; it does not +allow dependent GPU rounds to run in parallel. + +The first two submissions use the runner's two round buffers. A buffer contains +host/device page tables and result storage; it is separate from persistent +per-request token, position and KV state. It becomes reusable only after output +application and ticket retirement. The illustration stops after Round 1; further +scheduling and buffer reuse are omitted. In sustained execution, the engine collects the +oldest ticket before scheduling beyond its two-ticket limit. CPU stop checks, +resource availability and reservation limits still constrain submission. + +**Why there is no `boundary_stream` lane.** In this steady-state speculative +path, prelude, recurrent and coda all execute in the speculative runner's +`compute_stream`, which is supplied by `ModelRunner.core_stream`. Ordinary +prefill/bootstrap execution is outside this comparison. A separate +`boundary_stream` would misrepresent the speculative path shown here. + +The diagram assumes greedy, warm execution with `multi_stream=True` and enough +KV/output capacity. Input preparation/H2D are folded into the activities; cold +capture and bootstrap are omitted. With `multi_stream=False`, result copies share +the compute stream and the depicted D2H/compute overlap disappears. Cancellation +or EOS can discard an already-submitted later round, but its resources remain +live until device use completes. + +| Diagram operation | Current implementation | +| --- | --- | +| Sync submission, collection and scheduler update | `StageEngine._step()` in [stage_engine.py](../vllm_rlt/engine/stage_engine.py) | +| Sync ID readback and CPU acceptance | `SpeculativeSubmission.collect()` / `SpeculativeRunner._collect_round()` in [speculative.py](../vllm_rlt/worker/speculative.py) | +| Async submission ahead and ordered collection | `StageEngine._step_async_speculative()` / `_collect_speculative()` in [stage_engine.py](../vllm_rlt/engine/stage_engine.py) | +| GPU acceptance and token/position update | `AsyncSpeculativeRunner._accept_and_update_state()` in [async_speculative.py](../vllm_rlt/worker/async_speculative.py) | +| Stream events, result copy and buffer retirement | `AsyncSpeculativeRunner.submit()` and `SpeculativeRoundTicket.collect()` / `retire()` in [async_speculative.py](../vllm_rlt/worker/async_speculative.py) | +| CPU output/progress application | `Scheduler.update_from_output()` in [scheduler.py](../vllm_rlt/core/scheduler.py) | + +## Configuration + `LLM` starts a spawned worker process. In Python scripts, create it inside `main()` and use the `if __name__ == "__main__":` guard shown below. The context manager closes the worker when inference finishes or raises. diff --git a/docs/diagrams/README.md b/docs/diagrams/README.md index 19cdb674..56061270 100644 --- a/docs/diagrams/README.md +++ b/docs/diagrams/README.md @@ -66,3 +66,23 @@ and available as individual tabs in `index.html`. Timing and outcomes are illustrative. Preserve safety conditions and distinguish request progress, device completion and remote-access lifetime when editing. + +## Synchronous versus asynchronous speculation + +[async-speculative-timeline.svg](async-speculative-timeline.svg) is the editable +2700 × 1620 source; `async-speculative-timeline.png` is its 2× raster export. +It is embedded in [the speculation guide](../async-speculative.md#timeline-synchronous-versus-asynchronous-speculation) +and available in `index.html`. Both panels follow one request through two rounds +on the same illustrative time scale, with CPU, `compute_stream` and `copy_stream` +lanes. Purple blocks match CPU/GPU acceptance; blue Commit blocks show CPU output +and progress updates. The T guide aligns synchronous Round 1 completion with the +asynchronous timeline; an arrow marks the earlier asynchronous Round 1 commit. +The example assumes slightly shorter GPU acceptance, a small first-commit gap, +and accumulating overlap across rounds. These durations are not measurements. +CPU wait endpoints align with their result-copy completion. Later submissions and +buffer reuse are omitted. R0–R3 denote recurrent depths; Round 0 / 1 +denote speculative iterations. + +Start/end markers delimit CPU submission through CPU Commit (including async +ticket retirement); GPU/copy completion events remain separate. The bottom legend +defines these boundaries and notes the illustrative timing assumptions. diff --git a/docs/diagrams/async-speculative-timeline.png b/docs/diagrams/async-speculative-timeline.png new file mode 100644 index 00000000..7690974f Binary files /dev/null and b/docs/diagrams/async-speculative-timeline.png differ diff --git a/docs/diagrams/async-speculative-timeline.svg b/docs/diagrams/async-speculative-timeline.svg new file mode 100644 index 00000000..bdc3e6fd --- /dev/null +++ b/docs/diagrams/async-speculative-timeline.svg @@ -0,0 +1,185 @@ + diff --git a/docs/diagrams/index.html b/docs/diagrams/index.html index fb002434..3859ae3d 100644 --- a/docs/diagrams/index.html +++ b/docs/diagrams/index.html @@ -18,6 +18,7 @@ +
Rendered offline; no network or Mermaid plugin required. For limitations, including arbitrary launch topologies and PD with speculation, see the architecture guide.