Repository navigation

[FEAT] Add CUDA Graphs for cross-round async self-speculation - #127

Merged

bjf-frz merged 24 commits into

InferMatrixLab:mainfrom

liuyao0322:codex/async-speculative-cuda-graphs

Oct 27, 2026

Merged

[FEAT] Add CUDA Graphs for cross-round async self-speculation#127
bjf-frz merged 24 commits into
InferMatrixLab:mainfrom
liuyao0322:codex/async-speculative-cuda-graphs

Commits on Oct 15, 2026
Commits on Oct 16, 2026
- feat: precapture speculative row buckets with masked KV writes
  liuyao0322
  committed
  dfbb0f7
- docs: trace padded BF16 divergences and audit KV acceptance
  liuyao0322
  committed
  eba6b30
Commits on Oct 19, 2026
- fix: cover async speculative verification rows when the graph row limit is unset
  
  authored and
  liuyao0322
  committed
  9f3f650
- feat: adaptive drafts per async speculative round
  
  authored and
  liuyao0322
  committed
  fe1ada6
- fix: count only nonempty adaptive draft decisions
  
  authored and
  liuyao0322
  committed
  79af715
- perf: exact full-batch speculative graph buckets, no per-replay table reset
  
  authored and
  liuyao0322
  committed
  5a086df
- perf: vectorize async speculative rounds; per-position draft acceptance
  
  authored and
  liuyao0322
  committed
  158d187
- feat: adapt async speculative drafts by default; --draft-policy
  
  authored and
  liuyao0322
  committed
  c3a97d3
Commits on Oct 20, 2026
- docs: update adaptive defaults and current H100 results
  liuyao0322
  committed
  2c77360
Commits on Oct 22, 2026
- Merge main and integrate speculative graphs with stage runtime
  liuyao0322
  committed
  805010d
Commits on Oct 23, 2026
- Prioritize reachable graph buckets and trim startup work
  liuyao0322
  committed
  866985c
- Prevent cyclic garbage collection during CUDA graph capture
  liuyao0322
  committed
  e80f540
Commits on Oct 25, 2026
- Merge main and stabilize adaptive draft cost fitting
  liuyao0322
  committed
  05891dd
- Reuse Triton slot routing and split adaptive K out of graph PR
  liuyao0322
  committed
  980c092
Commits on Oct 26, 2026
- Remove README changes from async graph PR
  liuyao0322
  committed
  427ba9d
- Merge main while preserving fixed-K asynchronous graph integration
  liuyao0322
  committed
  7b5739c
Commits on Oct 27, 2026
- Remove async speculative graph timeline image from PR
  liuyao0322
  committed
  e474798
- Merge main pipeline decoding with bucketed asynchronous CUDA graphs
  liuyao0322
  committed
  179ef5a

Navigation Menu

Repository navigation

[FEAT] Add CUDA Graphs for cross-round async self-speculation - #127

[FEAT] Add CUDA Graphs for cross-round async self-speculation#127
bjf-frz merged 24 commits into
InferMatrixLab:mainfrom
liuyao0322:codex/async-speculative-cuda-graphs

Commits on Oct 15, 2026

feat: capture fixed-depth asynchronous speculative cores and coda

docs: record matched BF16 async graph measurements and timelines

docs: add real-checkpoint HTTP latency and CLI validation

docs: qualify async graphs through concurrency 256 and burst serving

docs: remove local artifact paths from async graph report

fix: expose graph coverage limits and budget speculative graph pools

Commits on Oct 16, 2026

feat: precapture speculative row buckets with masked KV writes

docs: trace padded BF16 divergences and audit KV acceptance

Commits on Oct 19, 2026

fix: cover async speculative verification rows when the graph row limit is unset

feat: adaptive drafts per async speculative round

fix: count only nonempty adaptive draft decisions

perf: exact full-batch speculative graph buckets, no per-replay table reset

perf: vectorize async speculative rounds; per-position draft acceptance

feat: adapt async speculative drafts by default; --draft-policy

Commits on Oct 20, 2026

docs: update adaptive defaults and current H100 results

Commits on Oct 22, 2026

Merge main and integrate speculative graphs with stage runtime

Commits on Oct 23, 2026

Prioritize reachable graph buckets and trim startup work

Prevent cyclic garbage collection during CUDA graph capture

Commits on Oct 25, 2026

Merge main and stabilize adaptive draft cost fitting

Reuse Triton slot routing and split adaptive K out of graph PR

Commits on Oct 26, 2026

Remove README changes from async graph PR

Merge main while preserving fixed-K asynchronous graph integration

Commits on Oct 27, 2026

Remove async speculative graph timeline image from PR

Merge main pipeline decoding with bucketed asynchronous CUDA graphs