Skip to content

fix(attention): size the decode KV tile to the shared-memory budget - #840

Open
festeraeb wants to merge 1 commit into
FlashML-org:mainfrom
festeraeb:fix/decode-tile-smem
Open

festeraeb wants to merge 1 commit into
FlashML-org:mainfrom
festeraeb:fix/decode-tile-smem

Conversation

@festeraeb

Copy link
Copy Markdown

Fixes #839.

The stage-1 decode kernel keeps the Q tile and one K and one V tile in shared memory. The launch code budgeted only K and V and never went below 32 KV rows, so head_dim 512 overflows a 64 KiB part. This counts the Q tile and halves BLOCK_N, down to tl.dot's 16-row minimum, until the three tiles fit (16 + 16 + 16 KiB for head_dim 512). GPUs whose budget already fits (all sm_80+) take the same tiles as before.

Tested on an RTX 2060 (sm_75), driver 580.178, CUDA 13.3, main @ a2ce636 built from source with sglang-kernel uninstalled, checkpoint and command as in #839:

build result
main first decode: OutOfResources ... Required: 81920, Hardware limit: 65536
this branch decode passes; the next failure is the extend tile (Required: 98304), which #24 fixes
this branch + #24's _select_extend_tile hunk serves: 53.5 / 65.8 tok/s decode on 44- / 400-token answers, TTFT 1.7 s warm

🤖 Generated with Claude Code

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Decode attention overflows shared memory on 64 KiB GPUs (Turing) for head_dim 512 (Gemma-4)

1 participant