Engine aborts itself every 60 s during a prompt prefill: 24 stall reports in 10 days (0.1.44.1, 0.1.45, 0.1.48), 23 minidumps attached #3979

Activity

1. github-actions

   added

   needs-formatTitle or template does not follow CONTRIBUTING.md yetTitle or template does not follow CONTRIBUTING.md yet

   on Oct 24, 2026
2. github-actions commented on Oct 24, 2026

   

   github-actionsBot

   on Oct 24, 2026 – with GitHub Actions

   More actions

   Thanks for the issue. To keep the tracker readable, every item needs an area prefix in the title and the parts of the issue form filled in. This one is missing:

   • Start the title with its area: [Engine], [GUI], [TUI], [Benchmark], [Docs] or [Other], then a short description. Example: "[Engine] decode stops after 2 minutes".

   Edit the issue (title and description) and this note goes away on its own. If it is still missing parts in 5 days it will be closed, which is not a rejection: fix it and comment /reopen.

   Details: CONTRIBUTING.md
3. PiccataPro commented on Oct 26, 2026

   

   PiccataPro

   on Oct 26, 2026

   More actions

   Still reproduces on 0.1.50

   Two RTX 3080 20 GiB (GA102-200, PCI 10de:2206), Windows 11 WDDM, driver 610.88 (CUDA UMD 13.3), 128 GiB.
   VERSION file says 0.1.49.1, commit c33fa7ba; the stall headers self-report 0.1.48 (Oct 24) and 0.1.50 (Oct 25/26).

   Four watchdog kills in ~4.2 days (2026-10-24 → 10-26), 4,683 requests served. Same invariant you found: 0 layers served since the last finished step in 4/4, and host idle for 61594 / 170900 / 64134 / 68699 ms.

   A distinction I have not seen in the other reports: they split 2 + 2 by an independent signal

   When	Stage	nvlddmkm events within ±60 s
   10-24 (instance 15:13:52)	layer 1 of the prompt chunk from token 16384	0 — the entire day has zero
   10-24 (instance 19:21:22)	layer 1 of the prompt chunk from token 16384	0 — the entire day has zero
   10-25 23:39	layer 23 of the prompt chunk from token 41812	208, all at 23:39:38
   10-26 13:56	layer 23 of the prompt chunk from token 63642	141 at 13:56:14, +7 ID-153 at 13:56:33–39

   Clean 4/4 in both directions. So these are two different bugs sharing one symptom:

   Class A (layer 1, ×2) — disk, not GPU. The engine warns about exactly this at every start:

   --ple-io direct measures slow here (313 rows/s, about 1 MB/s of random 4 KiB reads):
   a prompt can spend minutes reading layer 1 and trip the stall watchdog (#1425, #1629)
   

   Recorded across restarts: 313 / 333 / 3756 / 3914 / 4401 / 4608 / 4670 / 4734 / 5682 rows/s (1–23 MB/s). The drive is a DRAM-less ZHITAI Ti600 with 170 GB free of 932 (82% full). Both stalls land on token 16384, exactly the --prefill auto:16384 chunk boundary. I have not tried --ple-io mmap yet.

   Class B (layer 23, ×2) — GPU. This is the one that matters for #3979.

   Supporting your "not a full card" conclusion

   • Allocation failures in the whole log: 0 (out of memory / cudaErrorMemoryAllocation / failed to allocate / insufficient memory — four patterns, zero hits).
   • Free VRAM at the time: nvidia-smi 1,254 / 1,456 MiB; the engine's own figures at load 825 MiB (CUDA0) / 637 MiB (CUDA1).
   • The prompt-path scratch is carved out before the request, not during it: 6.72 GiB borrowed per card from the expert cache (2,295 / 2,305 of 4,276 slots), verify windows 96.2 MiB × 2, MTP head 275.7 MiB, --pipeline-windows 2 reserve 272 MiB.

   Answering your question 3 about --vram-reserve-mib

   On this box VRAM free when this request began: 683 MiB appears byte-identically in all four stall reports — including the two with zero GPU involvement. It is a comparison against --vram-reserve-mib 1024, which this config has never met (it loads at 637–825 MiB), and the companion line WARNING: only … MiB of VRAM free at the start of this request (below 1024 MiB) fires 72 times in this log. I read it as a standing condition, not a discriminator, and I would not use it as the reserve accounting you are probing for.

   Cross-link to #3801: same hand-off, same driver signature

   #3801 (--pipeline-windows 2, RTX 3080 Ti + V100, Windows/WDDM) is the closest match and it is currently closed without a fix. Three things line up exactly:

   1. Their correlated events: "nvlddmkm IDs 13, 14, and 153" — identical to ours.
   2. Their V100 went to Device Manager Code 10 (STATUS_INSUFFICIENT_RESOURCES) and vanished from nvidia-smi. On 2026-10-26 11:57 we got the escalated version of that same state: a 0x116 VIDEO_TDR_ERROR with parameter 3 = 0xC000009A (STATUS_INSUFFICIENT_RESOURCES), preceded by ggml_cuda_init: failed to initialize CUDA: no CUDA-capable device is detected after a 53-token prompt degraded to 2.5 tok/s.
   3. Their narrowed arm: "the earliest narrowed hang was after all 22 CUDA0 layers in the final stage-0 D2H hand-off followed by cudaStreamSynchronize(m.cs)". With --layer-split 23 the CUDA0 stage is exactly layers 0–22, so our layer 23 stall is the same hand-off seen from the other side — we never see layer 23 start, they never see the stage-0 copy finish.

   Both of us run --pipeline-windows 2. #3801's reporter disabled pipelining in production. Given how cheap the test is: is dropping --pipeline-windows 2 still the recommended mitigation on Windows/WDDM for a two-card split, or does 0.1.50 contain work that makes the pipelined hand-off safe?

   Artifacts

   • strata-stall-18080.dmp — 2026-10-26 13:57, 26,459 bytes
   • strata-stall-12868.dmp — 2026-10-25 23:40, 26,459 bytes (identical size)
   • C:\Windows\Minidump\102626-10390-01.dmp — the 0x116
   • Full engine log (4.1 MB) on request

   Two notes for whoever triages: across 3 faults in 4,683 requests, every one is in a data-movement / dequantisation kernel (copy_i32 ×2, iq_dequant_gu_f16 ×1), never in a math kernel. And one earlier instance (2026-10-25 21:09) died with two faults in a row (copy_i32: illegal memory access then iq_dequant_gu_f16: illegal memory access) without ever tripping the 60 s watchdog — a context-wide failure rather than a wedge.