Skip to content

fix: bound a running actor row's claim to be in progress (dead children shown as progressing/stalled) - #1965

Open
wqymi wants to merge 1 commit into
mainfrom
fix/roster-liveness-dead-child
Open

fix: bound a running actor row's claim to be in progress (dead children shown as progressing/stalled)#1965
wqymi wants to merge 1 commit into
mainfrom
fix/roster-liveness-dead-child

Conversation

@wqymi

@wqymi wqymi commented Jul 28, 2026

Copy link
Copy Markdown
Collaborator

Why this matters more than a display nit

The orchestrator's <active-sessions> roster exists for exactly one purpose: make the
orchestrator route work to an existing child instead of creating a new one — that is
#1741's whole point. So a dead child displayed as progressing is worse than no roster
entry at all
: the orchestrator routes work into a session that can never answer, and
progressing additionally suppresses re-dispatch because something appears to already be
in flight. Every consumer of deriveLiveness inherits this: the roster, session list
(which groups progressing/stalled under the heading "In progress"), session status,
and the fleet table. #1741's route-to-existing mechanism is only as good as the liveness
signal it reads, so an over-optimistic signal weakens it directly.

stalled is not a safe fallback either. It also lives under "In progress" and it also
marks the child as routable; it only says "no recent turn", not "nobody is home".

The defect

deriveLiveness (src/actor/schema.ts) placed unbounded trust in a running/pending
row. Two shapes:

  • A never-started child reads progressing forever. if (actor.turnCount === 0) return "progressing" returned early and bypassed the staleness window entirely. The comment's
    intent is right — a queued or cold-starting first turn must not be misread as a stall —
    but it implemented that leniency as unbounded.
  • A started child that died reads stalled forever, i.e. still "in progress".

The only repair for a row whose owner died is ActorRegistry's orphan sweep
(registry.ts: UPDATE actor_registry SET status='idle', last_outcome='failure', last_error='orphaned: process restarted' WHERE status IN ('pending','running') AND instance_id != $instanceID). That sweep runs once, at process init, and only for rows
carrying a different instance_id
. Until it runs — and for any row it cannot reach — the
derivation is the only thing standing between a corpse and the router, and it asserted
progress. Same defect class as #1960's orphaned running tool parts: a persisted transient
whose repair lives on exactly one path.

The fix

One new constant and one guard, both in deriveLiveness:

export const DEFAULT_LIVENESS_ABANDON_MS = 30 * 60_000

if (now - (actor.turnCount === 0 ? actor.time.created : actor.lastTurnTime) > abandonMs) return "idle"
if (actor.turnCount === 0) return "progressing"   // leniency KEPT, now bounded

Reference clock per row shape: a started child's evidence of life is its last turn; a
never-started one has none, so its spawn time (time.created) is the only honest
reference — which is what B needed: a longer bound measured from spawn, not removal of the
leniency.

Why 30 minutes, picked the way DEFAULT_LIVENESS_STALL_MS's comment picks 90s:
updateTurn is a per-step heartbeat, so a child that has genuinely begun cannot go 30m
without bumping last_turn_time — that is an order of magnitude past the 5-minute
stuck-detection cutoff (STUCK_THRESHOLD_MS) and unreachable by a live turn. A child that
has not begun is waiting on the concurrency gate, which admits work continuously, so 30m
of zero admissions means its process is gone.

Past the bound the row reads idle — already documented as "finished with no recorded
outcome (or an unknown state)" — and idle is the one bucket no consumer treats as
routable or in-flight. Deliberate direction of error: past the bound we would rather
report a still-queued child as idle (worst case: the orchestrator creates a duplicate,
which is wasteful) than as progressing (worst case: work routed into a corpse and
re-dispatch suppressed, which is unrecoverable).

Nothing is written to the database and no new repair mechanism is introduced; this is
purely the read side becoming honest. abandonMs is a 4th parameter with the same
override shape stallMs already has.

What I could NOT confirm, and the named follow-up

The report I worked from assumed the registry "never reconciles rows whose process is gone
— they never even became idle". Measured on the two real fixture rows, that is not
true
, and I am reporting it rather than fixing a defect that is not there. Both rows in
the isolated dev home read:

session mode status lastOutcome turnCount lastTurnTime lastError
ses_05c0af252ffeigpmiJbqaa8vUV ("Fix calc.py add() bug") peer idle failure 26 1785163372484 orphaned: process restarted
ses_05c08a1e9ffeFg2DEKBuhhBxuS ("Modernize docs/README.md install steps") peer idle failure 20 1785162766301 orphaned: process restarted

All six orphaned rows share time_completed = 1785163573647 — one sweep, at process init.
So the sweep works and reached exactly these rows; the roster that showed
stalled/progressing was rendered while they were still running under the then-live
instance, and reconciliation only came at the next process start. That reframes the
registry-side defect from "no reconciliation exists" to "reconciliation is bound to
process init, so it cannot see a row whose owner is the CURRENT instance
".

That residual half is genuinely larger than this PR:

FOLLOW-UP — per-actor liveness ownership. Peers run in-process (Actor.spawnPeer
runTurn fibers) and are registered with the parent's PROCESS_INSTANCE_ID, so
instance_id cannot distinguish a dead child from a live one inside one process. Actor
does not even expose instance_id (fromRow drops it), so no consumer can ask. Detecting
a within-instance dead actor needs a new ownership signal (a per-actor heartbeat or an
owning-fiber liveness probe) plus a reconcile pass at the read point, gated the way
#1960's sweep is gated (if ((yield* status.get(sessionID)).type !== "idle") return, main
slice only). That is a schema + lifecycle change and is deliberately not in this PR.
This PR ships the part that is correct on its own: the read side stops asserting progress
it cannot justify, which closes the routing hazard for both the pre-sweep window and any
row the sweep cannot reach.

Also worth flagging for reviewers: <active-sessions> and listPeerChildren are not on
main
(#1741 is still open, head 8847d324e) — zero grep hits. The roster itself
therefore cannot be tested here; the on-main consumers of the same signal are
tool/fleet.ts and tool/session.ts, and both are covered below.

Tests — one per defect, each revert-probed

test/actor/liveness.test.ts (+4) and test/tool/fleet.test.ts (+1). The fleet test uses
the two measured fixture rows' real field values (turnCount 26 / 20) replayed a day after
their last turn, and asserts through assembleFleet — the consumer — that neither lands in
a routable bucket.

Every pre-existing expectation is unchanged; the existing literals only gained the time
field the widened Pick now requires, and the existing "10-minute-old cold start is still
progressing" case still passes (10m < 30m), which is the point of keeping the leniency.

Probe 1 — restore the unbounded turnCount === 0 early return (defect B):

177 |     expect(deriveLiveness(stillborn, now)).not.toBe("progressing")
                                                     ^
error: expect(received).not.toBe(expected)
Expected: not "progressing"
(fail) deriveLiveness (T39 derivation rule) > never-started child spawned long ago does NOT read progressing [1.47ms]

137 |     expect(summary.counts.progressing).toBe(0)
                                             ^
error: expect(received).toBe(expected)
Expected: 0
Received: 1
(fail) assembleFleet > a row still claiming running a day after its last turn is not routable [7.93ms]

 21 pass  2 fail

Probe 2 — keep B's spawn bound, drop the bound for rows that HAVE run a turn (defect A):

218 |     const live = deriveLiveness(dead, now)
219 |     expect(live).not.toBe("stalled")
                           ^
error: expect(received).not.toBe(expected)
Expected: not "stalled"
(fail) deriveLiveness (T39 derivation rule) > started child still claiming running long after its last turn is not routable [0.34ms]

234 |     expect(deriveLiveness(row, now, DEFAULT_LIVENESS_STALL_MS, 5 * 60_000)).toBe("idle")
                                                                                  ^
error: expect(received).toBe(expected)
Expected: "idle"
Received: "stalled"
(fail) deriveLiveness (T39 derivation rule) > custom abandonMs overrides the default bound [2.02ms]

138 |     expect(summary.counts.stalled).toBe(0)
                                         ^
error: expect(received).toBe(expected)
Expected: 0
Received: 2
(fail) assembleFleet > a row still claiming running a day after its last turn is not routable [0.75ms]

 20 pass  3 fail

Suite numbers

bun typecheck exit 0.

Scoped run only (test/actor + everything that references the signal, found with
rg -l 'deriveLiveness|active-sessions|listPeerChildren' test), because concurrent
heavyweight suites on this host starve each other. Same command both sides
(bun test test/actor test/tool/fleet.test.ts test/tool/session-tool.test.ts --timeout 120000),
same worktree and node_modules, the three changed files reverted to origin/main for the
baseline:

pass skip fail tests files
pristine origin/main (60af8f1) 178 3 5 186 22
this branch 187 3 1 191 22

+5 tests is exactly the five new cases. My failure set is a strict subset of the
baseline's: both runs fail spawn no-deadlock (F56) > checkpoint-writer settles (no hang) when session permission is '*':'ask' …, and the baseline additionally fails four
session tool join cases on their 30s timeouts. The runs were sequential rather than
side-by-side, so the four extra baseline failures are load-dependent flakes, not a
regression — none of them is touched by this change, and none appears on this branch.

`deriveLiveness` placed unbounded trust in a `running`/`pending` registry row.
Two consequences, both of which make the orchestrator's child roster lie:

- a child that died BEFORE its first turn returned `progressing` forever, because
  `turnCount === 0` returned early and skipped the staleness window entirely;
- a child that died AFTER some turns kept reading `stalled`, and `stalled` is
  presented as "in progress" — routable, and implying work is already in flight.

ActorRegistry's orphan sweep is the only repair for such a row, and it runs once
at process init and only for rows carrying a different instance_id. Until it runs
(and for any row it cannot reach) the derivation must not assert progress.

Adds DEFAULT_LIVENESS_ABANDON_MS (30m), measured from the last turn for a started
child and from spawn time for one that never started. The turnCount-0 leniency is
kept — a queued or cold-starting first turn is still `progressing` — but bounded
instead of unbounded. Past the bound the row reads `idle`: the honest reading and
the only non-routable bucket.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant