Repository navigation
Design the minimal vLLM integration boundary #7
Description
Define how RandKV should integrate with vLLM before coupling the policy to a fast-moving internal API. The output is a reviewed design plus the smallest testable prototype.
Acceptance criteria:
- Pin and document the target vLLM version/commit.
- Map prompt protection, recent-buffer protection, deterministic request identity, eviction cadence, and observability onto vLLM concepts.
- Explain whether selection can remain independent per KV head when vLLM allocates paged blocks; call out any semantic compromise explicitly.
- Identify the narrow public or internal hook and its upgrade risk.
- Include call-site-first configuration requiring no more than one RandKV option.
- Add CPU-testable policy/adapter boundary tests before requiring CUDA.
- Document cancellation, prefix caching, chunked prefill, and multi-request behavior.
Start with a design document or draft PR; do not optimize kernels in this issue.
Activity
DaBestCode commentedon Sep 7, 2026
I’m taking this as the maintainer implementation.
The prototype targets vLLM v0.28.0 at commit
2cf0a6915ce544dc493a0990f2ea38d81601128a. The key design decision is to reject block-only random eviction: vLLM shares a physical block table across KV heads, while RandKV samples independently per head. The boundary instead emits deterministic per-head retained positions and source offsets so a later PyTorch/CUDA compactor can pack them into shared destination blocks without changing policy semantics.The accompanying design documents the required split between monotonic absolute token position and physical retained length, plus prefix caching, cancellation, chunked prefill, tensor/pipeline parallel identity, multi-request isolation, preemption, and initially unsupported modes. The prototype remains dependency-free and CPU-testable; no vLLM or CUDA install is required.
I’ll link the implementation PR here once it is open.
DaBestCode commentedon Sep 7, 2026
Implementation PR is open: #11. It includes the pinned design document, dependency-free planner, and CPU-only boundary tests. I’ll hold the merge until CI and automated review feedback are resolved.