feat(operation): resumable Procedure conduct, Tier 1 (Held/Resumed + execute_from + reconduct + try-conduct)#276
Merged
Merged
Conversation
…ng steps The Conductor now writes a SEPARATE journal entry (result="in_flight") BEFORE the effect for side-effecting steps (setpoint + action), then the ok/failed outcome after. A check records no marker: a check is a pure read, always safe to re-run. Why: a future resume must identify the one step that was mid-flight when a conduct halted. The marker lands before the write/action body and is durable even when the effect then raises or is cancelled, so a marker-without-outcome for a step_index pinpoints the interrupted step. This is the Tier 1 substrate for resumable conduct; no reader exists yet. This doubles the per-step append count for setpoint/action. Updated the conductor unit tests, the collect/discrete/continuous action-body tests, and the conduct-driven integration/scenario tests (which now filter in_flight markers out of their activity-row assertions). The softioc conductor test additionally asserts the marker round-trips into Postgres. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Tier 1 of resumable conduct: an operator can pause a halted Procedure conduct (hold_procedure, Running -> Held) and later resume it (resume_procedure, Held -> Running) rather than abort-and-reseed. Adds the ProcedureHeld / ProcedureResumed events, the HELD status value, the two evolver arms, and both command slices (decider + handler + route + MCP tool + tests). Naming: the pause state is `Held`, mirroring RunStatus.HELD, per a naming-r3 verdict. Procedure is an execution-FSM sibling of Run (a dozen "Mirrors Run..." docstrings); Agent's config-FSM `Suspended` was the rejected alternative. This also frees `Suspended` for the data-transfer twin's intervention-required semantics and matches the PackML operator=Held / external-blocker=Suspended split. ProcedureHeld diverges from slim RunHeld by carrying a required reason (a halted-conduct pause is a deliberate, high-information act; mirrors AgentSuspended.reason); ProcedureResumed carries the re_establishment_boundary the Conductor's execute_from will replay from (a boundary, NOT a continuity proof). Adds an AST evolver carry-forward fitness pinning that every non-genesis arm threads each additive field (the iteration denorms especially) so a new arm cannot silently wipe state on replay. Deferred to follow-up slices (intentional intermediate state): abort / truncate / end_iteration are not yet widened to accept Held (a Held Procedure is momentarily un-abortable; the abort PBT auto-flags Held as a disallowed source until then); resume's off-diagonal guard (refuse while the parent Run is Held) lands next. The proj_operation_procedure_ summary read model is deliberately left unsubscribed (its status CHECK admits only 5 values; a held Procedure reads as Running until a forward-only migration widens it) with a drift-catching test pinning the omission. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Completes the resumable-conduct Tier-1 widening: a paused (Held)
Procedure is no longer stranded. abort_procedure and truncate_procedure
now accept `Running | Held` (a paused conduct stays abortable, and a
paused-then-de-facto-dead one can be truncated retroactively);
end_iteration accepts `Held` so an iteration left open when the operator
paused can still be closed. start_iteration and complete_procedure and
append_activities stay Running-only (no new iteration / no completion /
no steps while paused; resume first). Mirrors Run BC's
`Running | Held` source sets.
The abort/truncate property-based tests' permitted-source sets gain
Held (so the enum-derived disallowed set drops it); positive Held
example tests added for all three verbs; the Cannot{Abort,Truncate,
EndIteration}Error docstrings and the abort/truncate runtime messages
now name `Running | Held` (matching RunCannotAbortError's "requires
Running or Held"). Also drops a few stale "deferred to 10c-c / if Held
lands" phase-tagged docstrings the FSM addition invalidated.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The off-diagonal guard for resumable conduct: a Phase-of-Run Procedure
cannot resume to Running and walk real setpoints while its parent Run is
itself Held. resume_procedure's handler is now a custom cross-aggregate
handler (like start_procedure) that loads the parent Run via a
one-directional Operation -> Run read and passes parent_run_held into
the pure decider, which raises ProcedureCannotResumeError(
parent_run_held=True). There is NO cascade from Run-resume into
Procedure-resume (that is a Layer-3 saga, deferred).
Standalone Procedures (no parent_run_id) skip the Run load and pass
parent_run_held=False. A Phase-of-Run Procedure whose parent_run_id
points at an empty Run stream raises RunNotFoundError (corruption, not
a happy path) -- same posture as start_procedure.
ProcedureCannotResumeError gains a parent_run_held discriminator for
operator-facing messaging ("its parent Run is Held; resume the Run
first") distinct from the status-guard message. Covered by decider
example + property tests (flag set, status-guard precedence) and handler
tests seeding a real parent Run in Held / Running / missing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…resume The resumable-conduct Tier-1 replay primitive. execute_from walks a PINNED manifest tail (manifest[boundary:]) rather than re-deriving the step list: re-drive setpoints (idempotent absolute writes), re-run checks as fresh gates, and HALT-for-operator on an acquisition (ActionStep) -- an interrupted acquisition is non-idempotent, so resume hands the redo-fresh-vs-reseed decision back to the operator instead of auto-skipping or auto-rerunning. Recorded step_index is the ABSOLUTE manifest position, so the replayed journal lines up with the original conduct, and re-driven setpoints inherit the pre-effect in-flight marker. Like execute(), it drives no FSM transition. `steps_from_manifest` is the exact inverse of `step_to_payload`: it parses the pinned ResolvedStepsRecorded.resolved_steps wire dicts back into Steps (pure, no Pydantic -- that lives at the HTTP boundary in step_from_wire), so resume NEVER re-derives from live Plan.wires / partition rules. A single-member `ResumePolicy.RE_ESTABLISH` enum documents the locked default; COMPARE (read-and-compare) stays an anti-hook-until-lease. Acceptance test: replay walks the pinned tail byte-for-byte -- two setpoints pinned via step_to_payload, parsed via steps_from_manifest, re-drive to two identical writes on a recording in-memory port. Plus boundary-skips-prefix, halt-on-action, re-run-check (match + mismatch), empty-tail no-op, negative-boundary reject, and the step/criterion round-trip pairs. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…lays its manifest
Wires Conductor.execute_from into an operator entry point. reconduct is the
resume twin of conduct_procedure: a thin orchestration slice (no decider)
over a new Conductor.reconduct method that composes resume + execute_from +
terminalize, exactly as Conductor.conduct composes start + execute +
terminalize. Putting the sibling-handler composition on the Conductor (a
non-slice module) keeps the slice independent (the cross-slice fitness
forbids a slice importing a sibling slice) and the composition in the one
place that already owns lifecycle-handler orchestration.
Conductor.reconduct replays a Held Procedure's PINNED manifest tail from
the re-establishment boundary, then terminalizes three ways:
- clean tail (incl. empty) -> auto-complete (Completed)
- acquisition halt -> leave Running, halt in the result body
(operator decides redo-fresh vs reseed)
- genuine step failure -> best-effort abort (Aborted)
A refused resume (not-Held / held parent Run / authz deny / not-found)
PROPAGATES as its mapped HTTP code rather than landing in the result body:
a refused resume is a guard outcome, not a replay outcome, and no replay
has happened yet.
The slice handler stays thin: authz -> load -> status guard (non-Held ->
409, BEFORE the manifest lookup so a Defined procedure is never a
misleading 500) -> find the pinned ResolvedStepsRecorded (a Held procedure
missing it is corruption -> 500) -> steps_from_manifest ->
conductor.reconduct -> project ConductorResult onto ReconductProcedureResult.
Adds `_manifest_replay.find_resolved_steps_record` (sibling of the recipe
finder) and `ResolvedStepsRecordNotFoundError` (500). `is_acquisition_halt`
stays a public conductor helper (the slice reads it for the result flag);
`derive_abort_reason` reverts to private (only conductor.py uses it now).
wire.py passes the hoisted resume handler into the Conductor and binds
reconduct over just the conductor.
Coverage: 9 handler unit tests exercise resume+replay+terminalize against
seeded Held+manifest states (clean / boundary / halt / step-failure /
missing-manifest / not-found / not-Held / parent-Run-Held / deny). Contract
tests cover the API-reachable guard/fault surfaces (404/409/422/500). NOTE:
the 200 happy path is not yet API-reachable -- the synchronous conduct pins
the manifest then runs to a terminal state, so there is no API path to a
Held+manifest state. Producing it (a conduct that pauses to Held on a halt,
or a mid-conduct cooperative hold) is a named follow-up; until then
reconduct is a tested consumer ahead of its producer.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
xmap
changed the title
…marker The Tier-1 pre-effect in-flight marker records a separate in_flight activity entry before each side-effecting step (setpoint / action), which consumes one extra event id per such step. The softioc+postgres integration tests seed a FixedIdGenerator with an exact id list and were not updated, so they exhausted mid-conduct; in the abort test the swallowed best-effort abort then never appended ProcedureAborted. Add one marker id per side-effecting step, ordered before its outcome so the ORDER BY event_id activity assertions still hold. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…e steps)
"manifest" was vestigial vocabulary. The event is already
ResolvedStepsRecorded with field resolved_steps, but the Conductor
parameter, a helper, two modules, and prose still said "manifest", a
noun that also collides with the federation and data BCs' own manifests.
Reuse the already-locked steps / resolved-steps / payload family instead
of the orphaned noun:
- execute_from(manifest=) / reconduct(manifest=) -> steps= (matches the
sibling execute(steps=); closes a same-data, different-name asymmetry)
- steps_from_manifest() -> steps_from_payload() (joins the existing
step_to_payload / _step_from_payload serialize pair)
- conduct_procedure/manifest.py -> resolved_steps.py
- _manifest_replay.py -> _resolved_steps_replay.py (mirrors the
_recipe_replay.py sibling: named after its event subject)
- prose / Field descriptions: "conduct manifest" / "pinned manifest" ->
"resolved steps" / "pinned step list"; openapi snapshot regenerated.
No behavior change. naming-r3 review PASS.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…recoverable failure Closes the resumable-conduct producer gap: until now a halted conduct could only abort, so reconduct (the resume path, PR #276) had nothing to resume and its 200 happy path was not API-reachable. try_conduct is the conduct verb-family's third member (conduct = run-to-terminal, reconduct = resume-and-replay, try-conduct = pause-to-Held-on-recoverable-failure). On a RECOVERABLE step failure (a setpoint or check: re-drivable / re-runnable on resume) the Conductor pauses the Procedure to Held instead of aborting, leaving the pinned ResolvedStepsRecorded so an operator can reconduct from a boundary. An acquisition (action) failure keeps aborting (an interrupted acquisition is non-idempotent, Tier 2 territory; a Held tail starting with an acquisition could only halt-for-operator on reconduct). Lifecycle failures and a mid-execute cancellation keep conduct's posture verbatim. Conductor changes: - new Conductor.try_conduct: the abort-vs-hold twin of conduct(); injects a hold_procedure handler and branches the failure terminalize on _is_recoverable_failure (source_kind in {setpoint, check}). - ConductorResult.held: True only when the hold transition itself succeeded, so a caller can tell a resumable Held outcome from a terminal Aborted one (both carry succeeded=False + failure). A hold that fails leaves the Procedure Running (held=False), mirroring conduct's best-effort abort. - _derive_abort_reason -> _derive_failure_reason (now serves abort + hold). Always-200 response gains a `held` discriminator (REST + MCP); without it Held and Aborted are indistinguishable in the body. Slice composition lives on the non-slice Conductor and two BC-level modules so try_conduct imports no sibling slice (the cross-slice-independence fitness): - _conduct_wire.py: the shared step-list wire models + converters, hoisted out of conduct's route (conduct's route + tool now import from here). - _conduct_preparation.py: the shared recipe re-expansion + pseudoaxis + resolved-steps pin pipeline, hoisted out of conduct's handler (the pure pin decider decide_resolved_steps_recorded moved here; conduct/resolved_steps.py removed). conduct's behavior is byte-for-byte unchanged (verified). Coverage: 10 handler unit tests (recoverable setpoint/check -> Held + manifest pinned; action -> Aborted; clean / empty -> Completed; hold-fails -> Running; start-rejected + complete-fails -> lifecycle failure; deny; not-found), contract + MCP tests, and two new reconduct contract tests that drive the now-reachable 200 paths end-to-end (try-conduct -> Held -> reconduct: empty tail completes; acquisition tail halts-for-operator). Gate-reviewed (naming-r3 PASS on the conduct/reconduct/try-conduct family; correctness + standards clean). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
xmap
changed the title
Resumable conduct landed the Held/Resumed FSM and try_conduct made a Held
Procedure operator-reachable, but the summary read model still showed it as
Running: the projection was left unsubscribed because the status CHECK admitted
only the 5 non-Held values. This completes the read-model story.
- Migration 20260621020000 widens the proj_operation_procedure_summary status
CHECK to admit 'Held' (drop + re-add the inline-named constraint; loosening
a CHECK is non-destructive, so no backfill and no data-safety opt-out).
Resumed maps back to 'Running', so 'Held' is the only new persisted value.
- ProcedureSummaryProjection subscribes ProcedureHeld -> status='Held' +
last_status_reason (the hold reason, like Aborted) and ProcedureResumed ->
status='Running' clearing last_status_reason (Running is not reason-bearing).
- ListProcedures status filter widened to admit 'Held' (the read-path Literal
had the same wrong "locked at 5 day one" assumption the projection did).
The projection's old docstring claim that the CHECK was "locked with the full
enum day one (5 statuses) so no future migration is needed even if Held/Resumed
land" was internally inconsistent (Held was never among those 5) and is
corrected. The "hoist at the 5th status arm" note is re-evaluated and the
explicit per-event SQL constants are kept: the arms are non-uniform (Truncated
sets interrupted_at, Resumed clears the reason), so a parameterized SQL would
read worse.
Coverage: 2 new projection unit tests (Held sets status+reason; Resumed clears
reason) + the flipped subscription test, and a real-Postgres integration test
that writes 'Held' through the projection and round-trips it via list_procedures
-- the only place the widened CHECK is actually exercised (unit tests mock the
connection). Contract status-filter test extended to all 6 statuses. Gate
reviewed (migration constraint name + safety, projection arg positions,
filter/read-path consistency, no other 5-status assumptions).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…enance + cleanup) Addresses the independent multi-axis gate review of PR #276. One confirmed MAJOR plus the verified MINOR/NIT cleanup. MAJOR -- actuation_kind provenance survives hold->resume. A conduct that touched a SIMULATED route before pausing to Held, then reconducted from a boundary past that prefix over a physical tail, completed as Physical and slipped past the promote_dataset Simulated/Hybrid gate. Fix: - ProcedureHeld (+ HoldProcedure) carry the conduct's observed-so-far actuation_kind (additive; legacy folds via payload.get -> None). - Conductor.try_conduct passes the observed kind to the hold. - the evolver Held arm MERGES it into Procedure.actuation_kind (merge, not set, so a later manual hold with no kind cannot wipe a prior conduct's kind); Resumed carries it forward. New pure merge_actuation_kinds helper on the aggregate (string-based: the aggregate owns no ActuationKind import, the cross-BC snapshot seam). Carry-forward fitness gains ProcedureHeld as an actuation_kind writer. - reconduct merges the folded prefix kind (passed by the handler) with the replay-tail kind before BOTH complete and abort, and reports the merged kind on the result so the response matches the terminal event. Regression test: Simulated prefix + physical tail -> Completed as Hybrid (not Physical), so the gate still bites. Two residuals documented (boundary past the failure index; acquisition-halt tail kind unpersisted), both narrower than the bug and aligned with the Tier-2 deferral. MINOR: - reconduct rejects re_establishment_boundary past the pinned step count (the resume decider only floors at 0; the bound lives where the manifest is known) -> 400; boundary == count stays valid (complete-with-nothing). - reconduct reuses the shared _conduct_wire.ConductorFailureResponse instead of a private duplicate, so the OpenAPI carries one canonical schema. - fix a stale docstring pointing at the removed resolved_steps.py. NIT: refresh the stale "hoist at the 5th status arm" projection-test note; add reconduct contract tests (boundary-past-count 400, genuine-step-failure abort) + try_conduct/reconduct missing-handler RuntimeError tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…coded scalar The pool registers a jsonb codec (pool.py set_type_codec encoder=json.dumps), so the event store passes event.payload as a raw dict and it round-trips as a real jsonb OBJECT. The activity store instead did an EXTRA json.dumps before binding, so the codec serialized the already-serialized string AGAIN, storing a double-encoded jsonb SCALAR string. Server-side `payload->>'key'` then returns NULL, which silently turned the conductor's pre-effect in-flight-marker filters (`payload->>'result' IS DISTINCT FROM 'in_flight'`) into no-ops across 17 integration/scenario test files. 16 tolerated the leaked marker rows; the 3 acquisition softioc tests that read exact rows did not -- the long-standing red on CI's (non-required) DB shard 1 that the local gate could not catch (softioc is CI-only). Fix: pass `row.payload` (the dict), matching the event store; the codec encodes it once into a real jsonb object. Reads now decode to dicts (like the event store), so the activity-payload `json.loads(row["payload"])` in 8 test files becomes direct dict access. Adds a real-Postgres regression test asserting `payload->>'result'` extracts the value (not NULL) so the marker filter actually excludes the marker -- the check that would have caught this locally. No production reader of the activities payload exists, so the read-shape change is test-only. The decision_inferences adapter has the same json.dumps-into-jsonb pattern; left as-is (nothing queries its jsonb server-side) and called out in the entries.py comment as a latent follow-up. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Integrates 18 commits from main (2-BM modeling, agents, the ports read-aloud naming pass, TomoWISE scaffold). Only conflict was infra/atlas/migrations/ atlas.sum (both sides appended migration entries); resolved by regenerating the hash from the migration files. Renamed this branch's migration 20260621020000_proc_summary_status_admit_held -> 20260621060000 to avoid a same-timestamp collision with main's 20260621020000_add_entries_run_observations _is_simulated and order it last; updated the 3 doc/test references. Regenerated openapi.json from the merged code (textual JSON auto-merge is unreliable). Full local gate green: ruff, pyright, tach, 878 operation unit, 26840 arch (openapi-drift + migration hash + projection fitness), 70 operation integration. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Coverage report
Click to see where and how coverage changed
The report is truncated to 25 files out of 43. To see the full report, please visit the workflow summary page. This report was generated by python-coverage-comment-action |
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tier 1 of resumable Procedure conduct (Tier 0 —
step_index+ResolvedStepsRecordedresolved-step list — merged in #258). Makes a halted Procedure conduct honestly pausable/resumable instead of abort-and-reseed, adds the producer that makes the resume path reachable end-to-end, and surfaces the paused state in the read model. Ten commits, each green through the full pre-commit gate with a gate-review before each.What lands
result="in_flight"journal append before the effect, for side-effecting steps only (setpoint/action; a check is a pure read). It survives a crash/cancel, which is the substrate a resume uses to identify the one mid-flight step.ProcedureHeld/ProcedureResumedevents,HELDstatus, evolver arms carrying every additive field, and thehold_procedure/resume_procedureslices. Plus an AST evolver carry-forward fitness.Running | Held(so a paused Procedure stays abortable and an open iteration can close).resume_procedurereads the parent Run and refuses while it isHeld; no Run-resume→Procedure-resume cascade.Conductor.execute_from(steps, boundary, policy)— replays the pinned resolved-step tail verbatim (re-drive setpoints, re-run checks fresh, halt-for-operator on an acquisition), never re-deriving the step list.reconduct_procedureorchestration — wiresexecute_frominto an operator entry point (POST /procedures/{id}/reconduct+ MCP). A thin slice overConductor.reconduct; terminalizes clean tail → Completed / acquisition halt → leave Running / step failure → abort. A refused resume propagates as its mapped HTTP code.try_conduct_procedureorchestration (the producer) — conduct's pause-capable twin (POST /procedures/{id}/try-conduct+ MCP): on a recoverable step failure (setpoint/check) it pauses to Held instead of aborting, leaving the pinned resolved steps forreconduct. An acquisition failure still aborts.Conductor.try_conductowns the composition;ConductorResult.held(on the always-200 body) is True only when the hold transition succeeded. This closes the producer gap: reconduct's 200 happy path is now API-reachable. Shared wire models + the recipe/pseudoaxis/pin pipeline were hoisted to BC-level_conduct_wire.py/_conduct_preparation.pyso the new slice imports no sibling; conduct's behavior is byte-for-byte unchanged.proj_operation_procedure_summarystatus CHECK to admit'Held'(drop + re-add the inline-named constraint; non-destructive);ProcedureSummaryProjectionsubscribesProcedureHeld→ status='Held' + reason andProcedureResumed→ status='Running' (clearing the reason); theListProceduresstatus filter is widened to match. A real-Postgres integration test writes'Held'and round-trips it throughlist_procedures— the only place the widened CHECK is actually exercised. (Corrects a stale docstring + filter Literal that both wrongly assumed the 5 statuses were "locked day one.")Plus two folded commits: a naming pass retiring "manifest" for "resolved steps" in code, and a CI fix reseeding the
FixedIdGeneratorfor the in-flight marker's extra appends in the softioc+postgres integration tests.Naming (naming-r3 resolved)
The pause state is
Held/Resumed(mirrorsRunStatus.HELD), notSuspended— which stays free for the data-transfer twin's intervention-required semantics and the PackML operator=Held / external-blocker=Suspended split.ProcedureHeldcarries a requiredreason. The conduct verb-family isconduct/reconduct/try-conduct(naming-r3 PASS on all three).Deliberately out of scope / deferred
See
project_resumable_conduct_design.mdfor the full design + gate-review history.🤖 Generated with Claude Code