You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
[Engine] session-save-append: park-borrow / slot-mount / checkpoint-mount watermarks dirty from token 0, so a resumed session cannot earn its next append #4552
Problem: a session saved to a v3 append file earns its next append only if no request runs in between. Any new request — even one that only adds 7 tokens above the file's committed boundary, with the whole prior conversation prefix-matched — makes the next save full-rewrite with save gate: full rewrite - pool rows at or below the file's saved boundary may differ from what it committed. (Observed on the draft append PR's branch, #4551.)
Reproduce (against a live serve instance): save a ~29k-token session to a fresh file (full, as expected); send one new short chat message (a handful of tokens) to the same session id; save again. Expected: append of the new group. Actual: full rewrite of the whole file. The no-request re-save variant appends correctly, which isolates the cause to the resume path.
Expected: the gate's own docs say a save should append when nothing below the committed boundary changed. Actual: the refusal fires on every post-request save.
Mechanism (static reading, consistent with the observed refusals): --session-save-reclaim releases the saved conversation's checkpoints/rows (session_save_reclaim(...), freed.parked). The session's next request therefore resumes through a physical re-copy — park-borrow (watermark M4, generate.cpp:14703), slot replacement (M5, :12818), or checkpoint mount (M3, :13989) — each of which calls mark_kv_dirty(0). The refusal is correct under current accounting: re-copied rows can land in different physical pool cells than the file recorded, and appending over them is the wrong-file restore the gate exists to make impossible. But the consequence is that append eligibility survives only while nothing touches the conversation — i.e. never in the ladder's steady state (save → turn → save). The feature's benefit today is limited to re-save chains.
Evidence — proof B from #4551 (short-increment shape, same host): saves 2–4 after +7..17-token asks full-rewrote (1.053 → 1.0535 → 1.0540 → 1.0544 GB); save 5, a re-save with no request in between, appended 237,210,624 bytes in 2.11 s vs 31.5 s full.
Reproduction status: reproduced locally (live serve host, both increment shapes, several sessions).
Design question: provenance-aware, offset-aware dirty accounting for engine-internal mounts/borrows. An engine-internal checkpoint captured from this session's own pool, never evicted, re-filled into the same physical cells byte-for-byte, could dirty from the mount offset instead of 0; anything whose provenance can't be established keeps dirty-from-0. Is there an existing invariant — or a cheap one to add, e.g. the checkpoint recording the pool generation it was captured from — that makes that identity provable?
Refusal lines as observed (click to expand)
save gate: full rewrite - pool rows at or below the file's saved boundary may differ from what it committed
session saved 28981 tokens, 1 of 3 checkpoints, 1053519424 bytes to <sessions>/ap-probe6 in 8454.7 ms (state 146.1 ms)
session saved 29014 tokens, 1 of 5 checkpoints, 237210624 bytes to <sessions>/ap-probe6 in 2114.6 ms (state 146.8 ms) (appended)
The last line shows the eligible shape (no request in between) working end to end: the append reports its delta as bytes, consumed by the serve-side save ledger fields from #4551.
A steady-state shape that sharpens this problem (live host, append branch, --session-save-reclaim --session-save-append): the save the serve unload button runs is exactly this mechanism, hit at the worst moment.
Sequence as it happened, with the engine's own lines:
Session idles → ladder idle-save appends normally (several [appended] saves in the 174k→188k-token range).
Cache pressure churns the pool: conversation cache: parked … evictions=5, then the next turn mounts the conversation back: conversation cache: restored 195063 tokens (live) — the M3 mount, mark_kv_dirty(0).
The user clicks Unload. The ladder's unload hook saves before engine.unload() drops the park (the save is required — park dies with the engine). The save is a few tokens above the committed boundary but the mount dirtied from 0: save gate: full rewrite - pool rows at or below the file's saved boundary may differ from what it committed → a 5.79 GB tmp+rename rewrite of a session whose conversation changed by ~2k tokens.
So in the ladder's normal loop, the mount bump doesn't just limit "re-save chains" — the one save that must happen (before the engine goes away) is guaranteed to ride a dirty-from-0 watermark.
Two related observations from the same host, same flags:
The save file is consumed at the cycle boundary today. After unload, the next request's warm-restore handoff unlinks the file the previous cycle wrote, so the next save starts a fresh full file anyway. Even with dirty accounting fixed, append eligibility cannot currently cross an unload→load cycle — unless keeping the file as an append base (instead of consuming it) is an acceptable design for append mode (disk cost: the file stands on disk until the session ends or retention retires it).
The gate's refusal reason conflates three very different guards.no verified checkpoint chain for this file (v2 file, torn tail, or foreign identity) fires identically for: file absent, chain digest failure, and a plan refused because the runtime's own read limits (which follow the boot-time split search — carve sizes vary between boots on a dual-GPU box) say this runtime could not restore the file. A post-restart first save is the common case; operators currently can't tell which guard fired. Splitting the reason into absent / refused by this runtime's limits / chain invalid is a small change to the same gate that makes all of this diagnosable.
Activity
sakuumomo commentedon Oct 29, 2026
A steady-state shape that sharpens this problem (live host, append branch,
--session-save-reclaim --session-save-append): the save the serve unload button runs is exactly this mechanism, hit at the worst moment.Sequence as it happened, with the engine's own lines:
[appended]saves in the 174k→188k-token range).conversation cache: parked … evictions=5, then the next turn mounts the conversation back:conversation cache: restored 195063 tokens (live)— the M3 mount,mark_kv_dirty(0).engine.unload()drops the park (the save is required — park dies with the engine). The save is a few tokens above the committed boundary but the mount dirtied from 0:save gate: full rewrite - pool rows at or below the file's saved boundary may differ from what it committed→ a 5.79 GB tmp+rename rewrite of a session whose conversation changed by ~2k tokens.So in the ladder's normal loop, the mount bump doesn't just limit "re-save chains" — the one save that must happen (before the engine goes away) is guaranteed to ride a dirty-from-0 watermark.
Two related observations from the same host, same flags:
no verified checkpoint chain for this file (v2 file, torn tail, or foreign identity)fires identically for: file absent, chain digest failure, and a plan refused because the runtime's own read limits (which follow the boot-time split search — carve sizes vary between boots on a dual-GPU box) say this runtime could not restore the file. A post-restart first save is the common case; operators currently can't tell which guard fired. Splitting the reason intoabsent/refused by this runtime's limits/chain invalidis a small change to the same gate that makes all of this diagnosable.