[Docs] Explain synchronous and asynchronous speculation timelines

README.md

@@ -88,7 +88,8 @@ for supported combinations and limitations.
   scheduling, multiple CUDA streams, reusable buffers, and CUDA Graph capture
   of decode recurrent cores. Asynchronous self-speculation also supports compatible
   paged FlashAttention backends and recurrent/coda CUDA Graphs; see the
-  [execution scope and constraints](docs/async-speculative.md#cuda-graph-scope-and-ownership).
+  [sync vs async timeline](docs/async-speculative.md#timeline-synchronous-versus-asynchronous-speculation)
+  and [execution scope and constraints](docs/async-speculative.md#cuda-graph-scope-and-ownership).
 - **Offline and online inference.** A Python API, a command-line interface,
   and an OpenAI-compatible completions endpoint with streaming, greedy decoding,
   and seeded top-k/top-p sampling.

docs/async-speculative.md

@@ -6,6 +6,127 @@ backends can consume its GPU-owned context lengths; `auto` selects against this
 capability as well as hardware, dtype and block size. Draft depth is configurable, with `d=2, D=4`
 as the initial validation configuration. The target depth must match the model.
 
+## Timeline: synchronous versus asynchronous speculation
+
+Cross-round asynchronous speculation keeps the next token and position on the
+GPU, so the next round can compute while the previous round's results return to
+the CPU. The comparison below shows the dependency change within one stage.
+
+![Synchronous versus asynchronous speculation on the same illustrative time scale](diagrams/async-speculative-timeline.png)
+
+[Open full-size PNG](diagrams/async-speculative-timeline.png) ·
+[Open editable SVG](diagrams/async-speculative-timeline.svg) ·
+[Interactive diagram viewer](diagrams/index.html#async-speculative-timeline)
+
+The panels follow **the same request through two speculative rounds**, using the
+same illustrative time scale and equal draft/verify widths. **Round 0 / Round 1**
+identify speculative iterations; **R0–R3 identify zero-based recurrent depths**.
+The example uses draft depth 2 and target depth 4. Draft repeats the shallow
+R0 → R1 path for its inputs; verification extends their retained hidden states
+through R2 → R3. Prelude and coda are included in the draft/verify blocks.
+Durations are explanatory, not benchmark results or a predicted speedup.
+
+**Round boundaries.** `Round N start` marks the beginning of CPU submission;
+`Round N end` marks completion of CPU Commit, including ticket retirement on the
+asynchronous path. These are host-visible lifecycle boundaries, not GPU execution
+boundaries. Shared markers explicitly name both events, such as synchronous
+`Round 0 end / Round 1 start`. Asynchronous Round 1 starts before Round 0 ends.
+`compute_done` marks device completion and `copy_done` marks result-transfer
+completion; neither alone is the end of the round. The compact legend below the
+timelines defines these terms without adding another comparison panel.
+
+**Match the purple blocks.** Accept/update determines the accepted prefix and
+next input token/position. In the synchronous path, this spans CPU greedy
+acceptance inside `collect()` and the scheduler's subsequent request progress
+update. In the asynchronous path, acceptance and the next token/position
+are computed on the GPU. CPU output delivery and EOS/length handling remain on
+the CPU, represented separately by the blue **Commit** block.
+
+**What Commit does.** After readback, the CPU interprets the already-decided
+result, records accepted output tokens in the Request, reconciles request progress
+and reservations, and checks EOS/length limits. In the asynchronous path, the
+engine then retires the round ticket so its buffer can be reused. This block
+groups collection's host processing and the engine/scheduler update; it is a
+diagram label, not a new runtime API. It does not redo GPU acceptance. The
+synchronous path performs the corresponding CPU output/progress update too,
+although its KV bookkeeping and submission lifetime differ.
+
+**Read the comparison from matching blocks.** Green Draft/Verify blocks represent
+the same model work. Purple Accept blocks move from CPU to GPU; blue Commit blocks
+remain on the CPU. In sync, Round 1 GPU computation starts only after Round 0's
+readback, acceptance, commit and next submission. In async, Round 1 GPU computation
+starts once GPU acceptance has updated
+its input state, while Round 0's D2H and CPU commit proceed alongside it.
+
+The gold **T** guide marks the end of synchronous **Round 1 CPU commit** and
+continues at the same horizontal coordinate into the asynchronous panel. The
+horizontal arrow connects asynchronous Round 1's earlier commit to T, comparing
+completion of the same two rounds. This is CPU result commitment, not the
+earlier GPU-completion event.
+
+The example draws GPU acceptance/update slightly shorter than CPU acceptance,
+so the first round's result delivery is only slightly earlier. Subsequent rounds
+also overlap prior readback and CPU work, making the completion gap accumulate.
+Shorter GPU acceptance is an illustrative assumption, not a measured property or
+a guarantee: batch size, launch overhead and implementation determine the actual
+cost. Draft/verify durations are equal across panels. The figure explains the
+removed host dependency without predicting a speedup or first-result latency.
+
+Both panels use the same three lanes:
+
+| Lane | Synchronous | Asynchronous |
+| --- | --- | --- |
+| CPU | Submit, wait for computation and IDs, accept, apply request progress, then submit the next round | Submit ahead, collect results in order, apply outputs and retire tickets |
+| `compute_stream` | Draft, verify, then produce/read back IDs inside `collect()`; wait for CPU progress before the next draft | Draft, verify, accept and update device state; the next draft consumes that state in stream order |
+| `copy_stream` | Unused for this result readback | Wait for `compute_done`, copy compact results into pinned host memory, record `copy_done` |
+
+**Synchronous dependency.** Verification must finish before `collect()` produces
+and reads the target/candidate IDs. CPU acceptance then determines the accepted
+prefix and correction/bonus token. The scheduler commits progress and truncates
+the rejected KV suffix before the next round can be submitted. The orange gap on
+`compute_stream` is this CPU-dependent interval. The ID readback is drawn on
+`compute_stream`, matching the stream context around `_collect_round()`; it is
+not an independent asynchronous copy-stream operation.
+
+**Asynchronous dependency.** Round 1 can be submitted before Round 0 is collected.
+Its GPU computation still follows Round 0's acceptance and token/position update.
+Meanwhile, `copy_stream` returns Round 0's result. The dashed completion guides
+align each CPU wait endpoint with the end of its corresponding D2H transfer.
+This removes CPU delivery from the immediate inter-round dependency; it does not
+allow dependent GPU rounds to run in parallel.
+
+The first two submissions use the runner's two round buffers. A buffer contains
+host/device page tables and result storage; it is separate from persistent
+per-request token, position and KV state. It becomes reusable only after output
+application and ticket retirement. The illustration stops after Round 1; further
+scheduling and buffer reuse are omitted. In sustained execution, the engine collects the
+oldest ticket before scheduling beyond its two-ticket limit. CPU stop checks,
+resource availability and reservation limits still constrain submission.
+
+**Why there is no `boundary_stream` lane.** In this steady-state speculative
+path, prelude, recurrent and coda all execute in the speculative runner's
+`compute_stream`, which is supplied by `ModelRunner.core_stream`. Ordinary
+prefill/bootstrap execution is outside this comparison. A separate
+`boundary_stream` would misrepresent the speculative path shown here.
+
+The diagram assumes greedy, warm execution with `multi_stream=True` and enough
+KV/output capacity. Input preparation/H2D are folded into the activities; cold
+capture and bootstrap are omitted. With `multi_stream=False`, result copies share
+the compute stream and the depicted D2H/compute overlap disappears. Cancellation
+or EOS can discard an already-submitted later round, but its resources remain
+live until device use completes.
+
+| Diagram operation | Current implementation |
+| --- | --- |
+| Sync submission, collection and scheduler update | `StageEngine._step()` in [stage_engine.py](../vllm_rlt/engine/stage_engine.py) |
+| Sync ID readback and CPU acceptance | `SpeculativeSubmission.collect()` / `SpeculativeRunner._collect_round()` in [speculative.py](../vllm_rlt/worker/speculative.py) |
+| Async submission ahead and ordered collection | `StageEngine._step_async_speculative()` / `_collect_speculative()` in [stage_engine.py](../vllm_rlt/engine/stage_engine.py) |
+| GPU acceptance and token/position update | `AsyncSpeculativeRunner._accept_and_update_state()` in [async_speculative.py](../vllm_rlt/worker/async_speculative.py) |
+| Stream events, result copy and buffer retirement | `AsyncSpeculativeRunner.submit()` and `SpeculativeRoundTicket.collect()` / `retire()` in [async_speculative.py](../vllm_rlt/worker/async_speculative.py) |
+| CPU output/progress application | `Scheduler.update_from_output()` in [scheduler.py](../vllm_rlt/core/scheduler.py) |
+
+## Configuration
+
 `LLM` starts a spawned worker process. In Python scripts, create it inside
 `main()` and use the `if __name__ == "__main__":` guard shown below. The context
 manager closes the worker when inference finishes or raises.

docs/diagrams/README.md

@@ -66,3 +66,23 @@ and available as individual tabs in `index.html`.
 
 Timing and outcomes are illustrative. Preserve safety conditions and distinguish
 request progress, device completion and remote-access lifetime when editing.
+
+## Synchronous versus asynchronous speculation
+
+[async-speculative-timeline.svg](async-speculative-timeline.svg) is the editable
+2700 × 1620 source; `async-speculative-timeline.png` is its 2× raster export.
+It is embedded in [the speculation guide](../async-speculative.md#timeline-synchronous-versus-asynchronous-speculation)
+and available in `index.html`. Both panels follow one request through two rounds
+on the same illustrative time scale, with CPU, `compute_stream` and `copy_stream`
+lanes. Purple blocks match CPU/GPU acceptance; blue Commit blocks show CPU output
+and progress updates. The T guide aligns synchronous Round 1 completion with the
+asynchronous timeline; an arrow marks the earlier asynchronous Round 1 commit.
+The example assumes slightly shorter GPU acceptance, a small first-commit gap,
+and accumulating overlap across rounds. These durations are not measurements.
+CPU wait endpoints align with their result-copy completion. Later submissions and
+buffer reuse are omitted. R0–R3 denote recurrent depths; Round 0 / 1
+denote speculative iterations.
+
+Start/end markers delimit CPU submission through CPU Commit (including async
+ticket retirement); GPU/copy completion events remain separate. The bottom legend
+defines these boundaries and notes the illustrative timing assumptions.