Skip to content

fix(checkpoint): book the writer's real outcome when the wait bound expires - #1945

Open
wqymi wants to merge 1 commit into
mainfrom
fix/checkpoint-writer-timeout-followup
Open

fix(checkpoint): book the writer's real outcome when the wait bound expires#1945
wqymi wants to merge 1 commit into
mainfrom
fix/checkpoint-writer-timeout-followup

Conversation

@wqymi

@wqymi wqymi commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

Follow-up to #1938 (merged). That PR's core fix is correct and is not changed here: waitForWriter still returns a distinct "timeout" when its 300s bound expires with the writer still in flight, and prune's result !== "failure" guard still skips the failure counter. This PR closes two holes that fix left behind.

1. Hitting the bound was completely silent

waitForWriter returned "timeout" with no log, and prune's watcher returned with no log. On main-before-#1938 the operator at least saw checkpoint writer failed — cleared thresholds for retry at +300s — in fact that log line is the entire evidence base #1938 was diagnosed from. After #1938, a writer stuck past 5 minutes produced no record at all.

waitForWriter now logs checkpoint writer wait bound expired — writer still in flight with sessionID and boundMs.

2. The writer's real outcome was booked nowhere

prune's watcher fiber is the only holder of the per-fire accounting, and writerFailures (prune.ts:177) is closure-private to the prune layer, so the checkpoint settle watcher cannot reach it. Once the watcher returned on "timeout" and never re-awaited:

  • a writer that genuinely FAILS past 300s ticked no counter anywhere. MAX_WRITER_FAILURES became unreachable for exactly the slow regime fix(checkpoint): don't count a timed-out writer wait as a writer failure #1938 is about — and fix(checkpoint): don't count a timed-out writer wait as a writer failure #1938's own notes document ~13 sequential LLM round-trips at 20-25s each, so a late failure is the realistic case, not a corner. A permanently-broken-but-slow writer was retried forever with no give-up warning.
  • a post-timeout SUCCESS never cleared the counter. writerFailures is only cleared on an observed success, and resetThresholds does not clear it, so two earlier fast failures left it frozen at 2 and any later fast failure tripped "gave up" for a session whose writers demonstrably work.

The watcher now extends its wait across bound expiries and accounts for the settled result, capped at MAX_WRITER_WAIT_EXTENSIONS (12 × 5min ≈ 1h) so a writer that never settles cannot pin the fiber for the life of the process.

Why extend the wait rather than subscribe to WriterCachePerf

Wiring prune to the WriterCachePerf event (checkpoint.ts:955-961, which already carries status: completed | failed) was the first candidate — it is the only existing cross-layer signal of the writer's real outcome. Rejected as materially riskier for the value:

  • Bus PubSub is InstanceState-scoped (bus/index.ts:107-114, 159-161), so a layer-level subscription in prune's layer would be built outside any Instance context. Subscribing per-fire inside the instance context instead means a subscription lifecycle per checkpoint trigger.
  • subscribe is lazy (Stream.unwrap) and races immediate publishes — history/writer.ts:40-42 documents exactly this hazard — so a correct version has to subscribe before the wait and hand the payload across via a Deferred, plus dedupe against the watcher's own observation for a writer that settles inside the bound.
  • It would add Bus.Service to prune's public layer signature (~25 provider call sites).

Re-entering waitForWriter re-awaits a Deferred we already have a handle to: no new dependency, no new state, no new event plumbing. Two microsecond-wide re-entry windows are documented in the code rather than papered over — (1) the writer settles between our expiry and the re-entry, so the map entry is gone and we get "no-writer" and book nothing (identical to the pre-existing fast-writer race already documented at prune.ts:302-315); (2) a queued writer was drained into a fresh entry in that window, so we book its outcome — still a real outcome for that session.

Tests

prune.test.ts gains the two cases that pin the prune-side consequence #1938 is actually about. #1938's test only asserted waitForWriter's return value, but its thesis is "the failure counter must not tick" — which lives in prune. Both reuse the existing makeRetryHarness:

  • timeouts never tick the counter, and a late success clears itfailure, failure, timeout, timeout, success then a re-cross with three more failures. Reads 6 enqueues; would read 5 if a post-timeout success did not clear the counter.
  • a late failure is still counted, so the cap stays reachabletimeout, failure × 3. Reads 3 enqueues then stops at the cap; would read 1 if the watcher still gave up at the bound (thresholds never cleared → fire 2 never enqueues).

The waitForWriter timeout test is tightened: it now pins the bound (still pending at 4 minutes, so shrinking 300s to 1s — reintroducing the bug in a new shape — fails) and asserts isWriterRunning is still true after the expiry, which is the property that makes "timeout" honest rather than a renamed failure. That replaces expect(result).not.toBe("failure"), which was dead — implied by the toBe("timeout") on the line above.

Verification

  • bun typecheck → exit 0
  • checkpoint-writer-wait-timeout, prune, prune-skip-system, checkpoint-drain, checkpoint-watermark-transactional, checkpoint-thresholds, checkpoint-boundary54 pass / 0 fail (bun test --timeout 120000; the 5s default times out on a loaded machine for reasons unrelated to this change — the pristine base fails the same way)
  • Regression probe: with these changes in place, reverting only the waitForWriter source change makes the timeout test fail with Expected: "timeout" / Received: "failure". Restored after.

Also

  • Folds fix(checkpoint): don't count a timed-out writer wait as a writer failure #1938's now-stale waitForWriter comment (the 5-min padding is no longer what prevents misclassification — the distinct return value is, and the AgentOutcome → WriterOutcome table was missing timeout) into the current block.
  • checkpoint-align.ts's claim that a degenerate-session LLM rejection increments writerFailures "via the existing path" is now only conditionally true; corrected to state when it is and is not booked.

Deliberately not included

computeBoundary coverage pinning (checkpoint.ts:254-293) — it changes what a checkpoint covers and is a separate, riskier decision.

…xpires

Follow-up to #1938. That PR stopped a merely-slow writer from being booked as
a failure by returning a distinct "timeout" from waitForWriter, which prune's
`result !== "failure"` guard skips. Correct, but it left two holes.

1. Hitting the bound became completely silent. waitForWriter returned and
prune's watcher returned, neither logging — yet the +300s log line is exactly
the evidence #1938 was diagnosed from. waitForWriter now logs
"checkpoint writer wait bound expired — writer still in flight".

2. The real outcome was booked nowhere. prune's watcher fiber is the only
holder of the per-fire accounting and writerFailures is private to the prune
layer, so once the watcher returned on "timeout" a writer that genuinely
FAILED past 300s ticked no counter — making MAX_WRITER_FAILURES unreachable
for exactly the slow regime #1938 is about, so a permanently-broken-but-slow
writer retried forever with no give-up warning. Symmetrically, a writer that
SUCCEEDED past 300s never cleared a counter left at 1-2 by earlier fast
failures, so a later fast failure could trip "gave up" for a session whose
writers demonstrably work.

The watcher now extends its wait across bound expiries and accounts for the
settled result, capped at MAX_WRITER_WAIT_EXTENSIONS (~1h) so a writer that
never settles cannot pin the fiber for the life of the process. Two
microsecond-wide re-entry windows are documented rather than papered over.

Tests: prune.test.ts gains the two cases that pin the prune-side consequence
#1938 is actually about (timeouts never tick the counter and a late success
clears it; a late failure is still counted so the cap stays reachable) — the
existing test only asserted waitForWriter's return value. The timeout test now
also pins the BOUND (still pending at 4 minutes, so shrinking it to 1s fails)
and asserts the writer is still running after the expiry, replacing a dead
`not.toBe("failure")` implied by the line above it.

Also folds the stale 5-min-padding comment into the current block and corrects
checkpoint-align.ts's now-conditional claim about writerFailures.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant