Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions .github/releases/v1.0.91.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
## opencode 1.0.91

Stable release from `main` branch. Orchestrators keep working when a workflow completes or runs out of node budget, and every workflow gets at least 100 nodes.

---

### 🐛 Bug Fixes

- **A 100-node floor for every workflow (#904).** `max_total_nodes` still defaults to 100, and 100 is now also the minimum: a lower value, such as a self-imposed `20`, is raised to 100 on create, replan, extend and recover. Higher values are still accepted. Existing workflows that stored a lower cap get the 100-node budget on their next graph action.
- **No more stopping to ask after a terminal graph (#904).** When a workflow had completed, `control(recover)` told the parent to "pause live scheduling first", which a completed graph cannot do, and the orchestration policy said to stop and ask the user after a ceiling rejection. Parents therefore paused and asked permission for verification and review that the original request already covered. The recover error and the node-ceiling hint now point at a bounded successor workflow, and the policy says a terminal graph or exhausted budget ends the graph, not the task: the parent continues the remaining in-scope work in a successor workflow, or directly, reusing valid outputs and leaving the old graph as evidence. It asks the user only when continuing would widen scope, need new permissions, take an irreversible external action, or repeat a plan that has already failed twice for the same reason.

---

### 🧪 Test Summary

```text
Local toolchain: Bun 1.4.3 (pinned 1.4.2; CI uses the pin)
DAG and tool tests: 1525 passed, 0 failed (after updating the workflow parameter snapshot)
Local recovery and create-validation tests: 21 passed, 0 failed
Runtime typecheck: passed
```

---

### 🔍 Verification

- Required merge checks are Typecheck, Unit Tests (linux), E2E Tests (linux), and E2E Tests (windows).
- The release pipeline validates template provenance, three-platform CLI builds, executable and install smoke checks, release notes, and SHA256SUMS before publication.
- No dependency, runtime-pin, HTTP, event, or database changes are included. Recovery still accepts only paused, failed or cancelled workflows; the replan-attempt ceiling is unchanged.
- Regression coverage includes a configured cap of 2 enforced as 100 on create and on replan, the successor hint on a replan ceiling rejection, and the completed-workflow recover message. A live orchestrator run was not exercised.

---

**Full changelog:** [`graphagent-v1.0.90`...`graphagent-v1.0.91`](https://github.com/LeXwDeX/GraphAgent/compare/graphagent-v1.0.90...graphagent-v1.0.91)
24 changes: 18 additions & 6 deletions packages/core/src/plugin/command/orchestration-policy.md
Original file line number Diff line number Diff line change
Expand Up @@ -264,9 +264,9 @@ repair and verification. A green build alone does not prove acceptance.

Useful follow-ups include a direct bounded repair, one delegated check, a
targeted `extend`, or a paused `control(replan)` when dependencies need to
change. Reuse valid outputs where useful. A new workflow is an option when the
existing graph cannot represent the work, not a requirement of a changed risk
label. A non-`ACCEPT` verdict does not mandate more agents or a full template.
change. Reuse valid outputs where useful. A new workflow is the expected path when the
existing graph cannot represent the work (terminal, or out of node budget), not
a requirement of a changed risk label. A non-`ACCEPT` verdict does not mandate more agents or a full template.

Report blockers and the actual workflow state when stopping or asking for a
decision. Do not claim rejected work passed. A replan or extend rejected by
Expand All @@ -283,8 +283,20 @@ prefer `control(extend_timeout)` to grant more time in place — no replan, no
lost child session. If ending the work, settle live scheduling rather than
abandoning active children. Naturally completed workflows can be extended, but
cannot be paused or replanned. For a ceiling breach, do not retry the identical
plan; report the remaining findings and stop or change approach within the
user's authorization.
plan; change approach instead.

## Keep Going Past a Terminal Graph

A terminal workflow, a node-ceiling rejection, or a refused `recover`/`extend`
ends that graph, not the task. When work the user's request already covers
remains (verification, review, acceptance, a bounded repair), continue it
yourself: start a bounded successor workflow scoped to the remaining steps, or
do it directly. Reuse the predecessor's valid outputs, cite its failure
evidence, and leave the old workflow untouched. Do not pause, end the turn, or
ask the user for permission to do this; the original request is the
authorization. Ask only when the next step would widen scope, need new
permissions or credentials, take an irreversible external action, or repeat a
plan that has already failed twice for the same reason.

## Actionable Checkpoints

Expand Down Expand Up @@ -314,7 +326,7 @@ operations targeting the remaining findings. Direct repair may be simpler once
child writers have stopped. You MUST NOT create cyclic `depends_on` or bypass
runtime budgets. Additional waves need a concrete evidence gap, not a quota.

Declare a finite `max_node_replan_attempts`. When the ceiling is exhausted, stop with `BLOCKED`, report the remaining findings, and do not retry the identical plan.
Declare a finite `max_node_replan_attempts`. When the ceiling is exhausted, do not retry the identical plan: change approach (direct repair or a successor workflow with a different decomposition). Only stop with `BLOCKED` when no in-scope approach remains.

## Replan Protocol (pause-first)

Expand Down
11 changes: 6 additions & 5 deletions packages/core/src/plugin/command/workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -386,7 +386,7 @@ a reporting leaf naturally completed the graph).

### Escalation: change approach after repeated failures

**Change your approach** when a node or workflow repeatedly fails. Triggers include `orchestrator_unresponsive` (the turn ended with the workflow still RUNNING and stalled), a replan-attempt ceiling rejection, or repeated review failures. A rejected replan or extend parks the workflow as paused and recoverable, not failed. Call `control(pause)` before stopping to ask the user. Try a different decomposition, model, or prompt, or split the node into smaller steps. Repeating the same failing plan wastes budget. Never cancel the graph just to avoid `orchestrator_unresponsive`; pause and ask the user instead of destroying in-flight work.
**Change your approach** when a node or workflow repeatedly fails. Triggers include `orchestrator_unresponsive` (the turn ended with the workflow still RUNNING and stalled), a replan-attempt ceiling rejection, or repeated review failures. A rejected replan or extend parks the workflow as paused and recoverable, not failed. Try a different decomposition, model, or prompt, or split the node into smaller steps. Repeating the same failing plan wastes budget. A rejected or impossible graph action is not a reason to stop: if the workflow is terminal or out of node budget and in-scope work remains, start a bounded successor workflow for just the remaining steps (reusing valid outputs) or do them directly, without asking the user (see Keep Going Past a Terminal Graph in the orchestration policy). Ask the user only when continuing would widen scope, need new permissions, or take an irreversible external action; call `control(pause)` on a live workflow first. Never cancel the graph just to avoid `orchestrator_unresponsive`; pause instead of destroying in-flight work.

### Crash recovery: a recovered workflow arrives paused

Expand Down Expand Up @@ -436,13 +436,14 @@ The value set above is what the runtime produces today; `push_exhausted`
exists in the schema but is reserved and currently never emitted.

Budget exhaustion is a separate class: `replan attempt ceiling exceeded` or
`Total node ceiling exceeded` means the budget is spent — change the approach
or stop, do not retry the identical plan (see Escalation).
`Total node ceiling exceeded` means this graph's budget is spent — do not retry
the identical plan; continue the remaining in-scope work in a bounded successor
workflow or directly (see Escalation).

Where to apply the repair:

- **Local retry in the same workflow**: pause live scheduling, read `status`, then call `control(recover)` with the current `expected_graph_rev` and selected `node_ids`. Failed workflows can recover directly. The runtime creates new attempts for these nodes and their downstream closure, preserves unrelated nodes, checks reusable managed artifacts, and resumes scheduling atomically. Old attempts and results remain readable; no BUILD or replacement YAML is needed.
- **Cancelled workflow**: recovery additionally requires `resume_cancelled: true`, reflecting an explicit request to resume cancelled work. Completed and archived workflows do not use this recovery action.
- **Cancelled workflow**: recovery additionally requires `resume_cancelled: true`, reflecting an explicit request to resume cancelled work. Completed and archived workflows do not use this recovery action: continue remaining in-scope work with `extend` when allowed, otherwise a bounded successor workflow.
- **Change the work definition**: use existing `replan`/`extend` for a different graph. Recovery retries the stored definitions; it does not rewrite prompts, reset budgets, roll back code, or undo external actions.

An isolated failure often calls for targeted recovery rather than starting
Expand Down Expand Up @@ -525,7 +526,7 @@ large. Higher concurrency is useful only for genuinely independent work:

- `max_concurrency`: default 5. Balance independent work against host and provider capacity.
- `max_node_replan_attempts`: default 5. Increase only if you expect iterative quality-driven convergence (review → revise → review cycles on a single artifact).
- `max_total_nodes`: default 100. Increase for large-scale decompositions.
- `max_total_nodes`: default and minimum 100 (lower values are raised to 100). Increase for large-scale decompositions.
- `worker_config.timeout_ms`: default 10 minutes. Increase for long-running nodes (compilation, large test suites). Verifier and aggregator nodes are the most common timeout victims: a node that consumes several parallel reports and re-checks their claims against code runs sequentially and routinely needs 20–30 minutes (e.g. `timeout_ms: 1800000`) — declaring the fan-out lanes' budget for the fan-in lane is a recurring failure pattern.

## Single-Workspace Discipline
Expand Down
7 changes: 5 additions & 2 deletions packages/opencode/src/dag/dag.ts
Original file line number Diff line number Diff line change
Expand Up @@ -264,7 +264,8 @@ function normalizeWorkflowConfig(config: WorkflowConfig): WorkflowConfig & { nod
mode: config.mode ?? "standard",
max_concurrency: config.max_concurrency ?? DEFAULT_WORKFLOW_CONFIG.maxConcurrency,
max_node_replan_attempts: config.max_node_replan_attempts ?? DEFAULT_WORKFLOW_CONFIG.maxNodeReplanAttempts,
max_total_nodes: config.max_total_nodes ?? DEFAULT_WORKFLOW_CONFIG.maxTotalNodes,
// The default doubles as a floor (see DagValidation.effectiveMaxTotalNodes).
max_total_nodes: Math.max(config.max_total_nodes ?? DEFAULT_WORKFLOW_CONFIG.maxTotalNodes, DEFAULT_WORKFLOW_CONFIG.maxTotalNodes),
max_timeout_extensions: config.max_timeout_extensions ?? DEFAULT_WORKFLOW_CONFIG.maxTimeoutExtensions,
node_defaults: defaults,
nodes: config.nodes.map((node) => normalizeNodeConfig(node, defaults)),
Expand Down Expand Up @@ -1182,7 +1183,9 @@ export const layer = Layer.effect(
if (workflow.status !== "paused" && workflow.status !== "failed" && workflow.status !== "cancelled") {
return yield* Effect.fail(
new Error(
`Recovery requires a paused or failed workflow; ${workflow.status} cannot recover (pause live scheduling first)`,
workflow.status === "completed" || workflow.status === "archived"
? `Recovery requires a paused or failed workflow; ${workflow.status} cannot recover. It is terminal: if in-scope work remains, extend it when allowed or start a bounded successor workflow that reuses its valid outputs — no user approval is needed for work the original request already covers`
: `Recovery requires a paused or failed workflow; ${workflow.status} cannot recover (pause live scheduling first)`,
),
)
}
Expand Down
8 changes: 5 additions & 3 deletions packages/opencode/src/dag/validation.ts
Original file line number Diff line number Diff line change
Expand Up @@ -273,7 +273,7 @@ const GraphBudgetFields = {
description: "Max replan restarts per node ID. Default: 5",
}),
max_total_nodes: Schema.optional(Schema.Number).annotate({
description: "Cumulative node cap across the workflow lifetime. Default: 100",
description: "Cumulative node cap across the workflow lifetime. Default and minimum: 100; lower values are raised to 100",
}),
} as const

Expand Down Expand Up @@ -1052,8 +1052,10 @@ function bindingDiagnostics(nodes: readonly NodeConfig[]): Diagnostic[] {
)
}

/** The default is also the floor: a start document may raise the budget but
* never shrink it, so a tight self-imposed cap cannot strand remaining work. */
export function effectiveMaxTotalNodes(maxTotalNodes: number | undefined) {
return maxTotalNodes ?? DEFAULT_WORKFLOW_CONFIG.maxTotalNodes
return Math.max(maxTotalNodes ?? DEFAULT_WORKFLOW_CONFIG.maxTotalNodes, DEFAULT_WORKFLOW_CONFIG.maxTotalNodes)
}

/** Which document carries the ceiling. Only `start` can change it: extend,
Expand All @@ -1079,7 +1081,7 @@ function ceilingExceeded(
hint:
action === "start"
? "Reduce the graph or raise config.max_total_nodes deliberately"
: `The ceiling is fixed by the start document (this ${action} would reach ${totalNodes} nodes; limit ${max}); a ${action} document cannot carry config.max_total_nodes. Shrink the fragment or start a new workflow with a higher config.max_total_nodes`,
: `The ceiling is fixed by the start document (this ${action} would reach ${totalNodes} nodes; limit ${max}); a ${action} document cannot carry config.max_total_nodes. Shrink the fragment, or continue in-scope work in a bounded successor workflow (with a higher config.max_total_nodes if needed) without stopping for user approval`,
}),
]
}
Expand Down
10 changes: 6 additions & 4 deletions packages/opencode/test/dag/dag-create-validation.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -99,9 +99,10 @@ describe("Dag.create structural validation", () => {
Effect.gen(function* () {
const error = yield* createExpectingError({
max_total_nodes: 2,
nodes: [node("a"), node("b"), node("c")],
nodes: Array.from({ length: 101 }, (_, i) => node(`n${i}`)),
})
expect(error.message).toContain("Total node ceiling exceeded: 3 nodes > 2 max")
// A lower configured cap is raised to the 100-node floor.
expect(error.message).toContain("Total node ceiling exceeded: 101 nodes > 100 max")
// Only a start document owns the graph budget; the start hint may say so.
expect((error as Dag.StructuralValidationError).diagnostics[0]?.hint).toContain("raise config.max_total_nodes")
}).pipe(Effect.scoped, Effect.provide(dagLayer)) as Effect.Effect<never>,
Expand All @@ -122,15 +123,16 @@ describe("Dag.create structural validation", () => {
})
.pipe(Effect.orDie)
const error = yield* dag
.replan(dagID, { nodes: [node("b"), node("c")] })
.replan(dagID, { nodes: Array.from({ length: 100 }, (_, i) => node(`b${i}`)) })
.pipe(Effect.catch((e: Error) => Effect.succeed(e)))
expect(error).toBeInstanceOf(Dag.StructuralValidationError)
const ceiling = (error as Dag.StructuralValidationError).diagnostics.find((d) =>
d.message.includes("Total node ceiling exceeded: 3 nodes > 2 max"),
d.message.includes("Total node ceiling exceeded: 101 nodes > 100 max"),
)!
expect(ceiling.hint).not.toContain("raise max_total_nodes")
expect(ceiling.hint).toContain("fixed by the start document")
expect(ceiling.hint).toContain("Shrink the fragment")
expect(ceiling.hint).toContain("bounded successor workflow")
}).pipe(Effect.scoped, Effect.provide(dagLayer)) as Effect.Effect<never>,
)
})
Expand Down
11 changes: 9 additions & 2 deletions packages/opencode/test/dag/dag-local-recovery.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,7 @@ const complete = Effect.fn(function* (dag: Dag.Interface, id: string, nodeID: st
})

describe("DAG local recovery", () => {
it.effect("rejects live scheduling, incomplete selections, corrupt definitions, and exhausted lifetime node budgets", () =>
it.effect("rejects live scheduling, incomplete selections, and corrupt definitions", () =>
Effect.gen(function* () {
const { dag, id } = yield* setup([node("A"), node("B")])
const rev = (yield* dag.store.getWorkflow(id))!.graphRev
Expand All @@ -100,7 +100,7 @@ describe("DAG local recovery", () => {
}
const { db } = yield* Database.Service
const original = Dag.parseWorkflowConfig(snapshot.workflow!.config)!
for (const config of ["{broken", JSON.stringify({ ...original, nodes: [] }), JSON.stringify({ ...original, max_total_nodes: 2 })]) {
for (const config of ["{broken", JSON.stringify({ ...original, nodes: [] })]) {
yield* db.update(WorkflowTable).set({ config }).where(eq(WorkflowTable.id, id)).run().pipe(Effect.orDie)
expect((yield* dag.recover(id, { nodeIDs: ["A", "B"], expectedGraphRev: rev }).pipe(Effect.result))._tag).toBe("Failure")
expect(yield* dag.store.getNodes(id)).toEqual(snapshot.nodes)
Expand Down Expand Up @@ -161,6 +161,13 @@ describe("DAG local recovery", () => {
yield* complete(dag, id, retryD, "done")
yield* dag.complete(id)
expect((yield* dag.store.getWorkflow(id))?.status).toBe("completed")
// A completed graph points the parent at a successor workflow instead of
// a pause it cannot perform, so it keeps working rather than stopping.
const terminal = yield* dag
.recover(id, { nodeIDs: [retryC], expectedGraphRev: (yield* dag.store.getWorkflow(id))!.graphRev })
.pipe(Effect.flip)
expect(terminal.message).toContain("bounded successor workflow")
expect(terminal.message).not.toContain("pause live scheduling first")
const { db } = yield* Database.Service
const events = yield* EventV2.Service
const snapshot = { workflow: (yield* dag.store.getWorkflow(id))!, nodes: yield* dag.store.getNodes(id) }
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -1061,7 +1061,7 @@ exports[`tool parameters JSON Schema (wire shape) workflow 1`] = `
"type": "number",
},
"max_total_nodes": {
"description": "Cumulative node cap across the workflow lifetime. Default: 100",
"description": "Cumulative node cap across the workflow lifetime. Default and minimum: 100; lower values are raised to 100",
"type": "number",
},
"name": {
Expand Down Expand Up @@ -1123,7 +1123,7 @@ exports[`tool parameters JSON Schema (wire shape) workflow 1`] = `
"type": "number",
},
"max_total_nodes": {
"description": "Cumulative node cap across the workflow lifetime. Default: 100",
"description": "Cumulative node cap across the workflow lifetime. Default and minimum: 100; lower values are raised to 100",
"type": "number",
},
"name": {
Expand Down
Loading