fix: repair orphaned running tool parts; stop a directory 403 from killing the TUI - #1960
Open
wqymi wants to merge 1 commit into
Open
fix: repair orphaned running tool parts; stop a directory 403 from killing the TUI#1960wqymi wants to merge 1 commit into
wqymi wants to merge 1 commit into
Conversation
… killing the TUI Two independent robustness holes. Orphaned `running` tool parts: a tool part is persisted as `running` when the tool starts so the TUI can stream progress, and only the abort finalizer in SessionProcessor.cleanup rewrites it. Any exit path that skips that finalizer (process kill, crash, dev restart, a registration that raced teardown, a call that arrived after ctx.toolcalls was cleared) leaves the row `running` forever, so the transcript permanently shows tool calls that will never finish. Nothing repaired them: the model-message converter synthesizes an output-error for pending/running parts so the provider never sees a dangling tool_use, but it never touches the persisted row. - cleanup() now takes a second, DB-driven pass over the assistant message's own tool parts instead of trusting only the in-memory map. - New SessionPrompt.sweepOrphanToolParts repairs leftovers from a previous process at the next prompt. A currently executing tool part is also `running`, so the sweep is gated on session status being idle (no active runner) and on the main slice only (SessionProcessor publishes status for the main slice only, so a subagent slice can be live while the session reads idle). - Both finalizers share MessageV2.abortedToolState so interrupted parts get one consistent shape. Directory 403 killing the TUI: the instance middleware correctly rejects a directory outside the server cwd, but the SDK throws the parsed body with no status attached, so bootstrap treated it as fatal and exit()ed — a user who picked a non-whitelisted worktree lost their whole session. The rejection now carries a stable `code`, bootstrap classifies it as recoverable and rethrows instead of exiting, and the worktree switch restores the previous directory, re-syncs and toasts the rejected path. Genuinely fatal bootstrap failures still exit.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two independent robustness defects. Both are small and both are about surviving an
interruption, so they ride in one PR; the diffs do not overlap and can be reviewed
section by section.
A — orphaned
runningtool parts are never repairedA tool part is persisted as
runningthe moment the tool STARTS, deliberately, so theTUI can stream progress (
prompt.ts:1037/1461/1677,processor.ts:496). Only the abortfinalizer in
SessionProcessor.cleanuprewrites it, and that finalizer iterated thein-memory
ctx.toolcallsmap of the owning process (processor.ts:743-759). Everyexit path that skips it leaves the DB row at
runningforever: process kill, crash, devrestart, a second interruption after
ctx.toolcalls = {}, areadToolCallmiss, or the250 ms
Deferred.await(...).pipe(Effect.timeout("250 millis"))window. There was nostartup or session-load repair anywhere (
session/index.ts,storage/db.ts: zero hits).Live evidence, re-measured across the whole store: **168 orphaned tool parts (126
runningpending) across 92 sessions**; in one two-week-old sessioncompleted 2474 / error 105 / running 48, of which exactly one was genuinely in flight — i.e. a per-interruption leak.This is NOT the empty-content provider 400 — proven by wire-level construction
This shape was investigated as a candidate producer of
messages.<N>: user messages must have non-empty contentand excluded. The exclusion is now a construction, not anargument: a prompt captured from a real
LanguageModelV3.doStreamshows that an assistantmessage whose ONLY part is an orphaned
running/pendingtool part converts toyielding zero empty-content messages.
message-v2.ts:886-899(the correct lines; anearlier revision of this description cited the stale
:872-883) matches bothdiscriminants and substitutes a hard-coded non-empty literal that no data can empty, and
ai@6.0.168's empty-text strip (dist/index.mjs:1424) lives in therole:"user"branchand only inspects
type:"text"parts — it never drops atool-resultfrom arole:"tool"block. So no tool part in any state can yield
content: [].The timeline agrees: 74 recorded 400s span 2026-07-14 → 07-28 while the first orphan-only
assistant message in that session is 2026-07-28, so 62 of 74 predate the shape, and 6 of
the 7 sessions carrying the shape have zero 400s. The real producer was a bare-string
continuation turn in
ProviderTransform.ensureTrailingUserMessage— see #1948.So the defect fixed here is data hygiene and UI truthfulness, not provider legality: the
transcript permanently shows tool calls that will never finish. It never touches the
persisted row today.
Fix, two layers:
cleanup()gets a second, DB-driven pass over the assistant message's own toolparts (
MessageV2.parts(ctx.assistantMessage.id)), so a call whose registration racedteardown, arrived after the map was cleared, or whose
readToolCalllookup missed isstill finalized. Every tool part of that message belongs to the turn being torn down,
so any part still
pending/runningis unfinalized by definition. Idempotent.SessionPrompt.sweepOrphanToolPartsrepairs rows a previous process left behind,at the same recovery point as the existing
sweepOrphanAssistants(start ofprompt()for non-spawn/hook sources).
Safety of the sweep's scoping
A currently executing tool part is also
running, so a naive "rewrite every running row"sweep corrupts live turns. Two guards, both required, and the gate lives inside the
function rather than in the caller contract:
idle.busy/retrymean an active runner owns thesession, and a tool can only execute inside a runner's turn. Same argument the shipped
sweepOrphanAssistantscaller relies on.sessions.messagesdefault).SessionProcessorpublishes statusfor the main slice only (
if (isMain) status.set(...)), so a subagent slice can beexecuting tools while the session status reads
idle— its parts are out of scope.Both finalizers now share
MessageV2.abortedToolState, so an interrupted part has oneconsistent shape (
status: "error",error: "Tool execution aborted",metadata.interrupted: true, originaltime.startpreserved).B — a rejected directory 403 killed the TUI
server/routes/instance/middleware.tsrejects any directory outside the server cwd(except the app-owned orchestrator workspace) with a 403. The generated SDK throws the
parsed response body with no status attached, so
sync.tsx's bootstrap catch couldnot tell a policy rejection from a broken server and ran
await exit(e)— destroying therenderer. A user who picked a non-whitelisted worktree lost their entire session.
The whitelist is correct policy and is unchanged. Only its recognisability and client
handling change:
code: "directory_not_allowed"plus the rejecteddirectory(new leaf moduleinstance/access.tsowns the code + guard so the TUI doesnot import the server's instance/bootstrap graph);
bootstrapclassifies that error as recoverable and rethrows instead of exiting;everything else is still fatal;
Cannot switch to <dir>: outside this server's working directory.Tests
test/session/prompt-orphan-tool-parts.test.ts(6) — an orphanedrunningpart isrepaired when idle; it is not touched while the session is
busy, nor while it isretry;completedparts are untouched;abortedToolStateshape.test/cli/tui/bootstrap-directory-denied.test.tsx(2) — a 403 from the middleware neverreaches the fatal-exit path (
exitis never invoked and the error is handed to thecaller); a non-403 bootstrap failure still exits.
Revert probes (each src change reverted alone):
Expected: "running" / Received: "error"(exactly the live-turn corruption the gate prevents);
Expected: "error" / Received: "running";if (fatal)→expect(failure).toBeDefined()fails,Received: undefined(bootstrap exited instead of surfacing), while the "still exits" test keeps passing.
bun typecheckexit 0.Test suites compared against a pristine
origin/mainworktree with the identical command(
bun test test/session/ test/cron/ test/cli/tui/ --timeout 120000), the two runsexecuted side by side:
origin/main(60af8f1)+8 pass / +2 filesis exactly the two new test files. The failure sets are byteidentical (
diffof the sorted(fail)lines is empty) — 10 pre-existing failures in thesession/loopbusy/cancel/queue group, which flake under concurrent suite load and failthe same way without this change.
Live-verified against
mimo/mimo-v2.5in an isolated dev home: selecting a worktreeoutside the server cwd shows the toast, leaves the transcript intact, and the session
still answers a following turn.