You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit fe7de5d
Browse filesBrowse the repository at this point in the historyBrowse files
feat(agent): add run limits for spend, steps, and duration
Adds a "Limit run" switch to the Agent block's advanced settings with
max spend (USD), max steps, and max duration, and a choice of what happens
when the agent still wants tools past a limit: return a final answer
without tools (default) or fail the block. A new stopReason output reports
whether a limit stopped the run.
A step is one model response whose tool calls run. Limits are enforced at
the two seams every provider already shares, so no provider tool loop
changes: prepareConversationGeneration turns the agent's tools off in each
protocol's own final-answer shape once a limit is reached, and
executeProviderTool refuses tools past a limit. Calls that offer no agent
tools (structured-output passes, Bedrock's forced structured_output tool)
are never limited, so a run that answers within its limits is unaffected.
One budget spans the selected model and its fallbacks; spend counts each
attempt's own billed cost, excluding what it carried in from earlier
attempts. Refusals are not journaled, so a retry with a fresh budget runs
the call.
Also renames isConversationContextError to isPermanentGenerationError,
since it now covers the run-limit and usage-limit stops, and shares the
billed user-stop construction between both stops.
Copy file name to clipboardExpand all lines: apps/docs/content/docs/workflows/blocks/agent.mdx
+3-1Lines changed: 3 additions & 1 deletion
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -121,7 +121,7 @@ Set **Response Format** to a JSON Schema when later blocks need named fields. Na
121
121
122
122
Malformed JSON can fall back to plain text with `_responseFormatWarning`. Sim does not perform a general post-parse schema check on every provider result, so validate fields downstream before consequential actions.
123
123
124
-
Avoid schema fields named `model`, `tokens`, `toolCalls`, `providerTiming`, or `cost`. Sim's standard metadata overwrites those fields after parsing, even when a metadata value is absent.
124
+
Avoid schema fields named `model`, `tokens`, `toolCalls`, `providerTiming`, or `cost`. Sim's standard metadata overwrites those fields after parsing, even when a metadata value is absent. With **Limit run** on, `stopReason` is overwritten too.
125
125
126
126
```json
127
127
{
@@ -144,6 +144,7 @@ Some settings live under advanced, or appear only for models that support them:
144
144
145
145
-**Temperature.** Controls variation on models that support it. Lower values tend to reduce variation; test accuracy on representative inputs rather than treating temperature as a factuality guarantee.
146
146
-**Max output tokens.** Caps the response length. Defaults to the model's full limit.
147
+
- **Limit run.** Stops a long tool-using run at a **max spend** in USD, a **max number of steps**, or a **max duration** in seconds. A step is one model response whose tool calls run. Leave a field empty for no limit. A limit acts only when the agent wants to keep using tools, so a call or tool that is already running finishes first, and a run that finishes before reaching a limit is unaffected. Spend is measured the same way as `<agent.cost>`, including tool costs, so a model Sim does not bill for, such as one on your own API key, adds only its tool costs. One budget covers the selected model and its fallbacks, and each try of **Retry on fail** starts a new one. **When a limit is reached** chooses what happens: **Return best answer** (the default) asks the model for a final answer without tools, which can spend slightly past the limit, and fails the block only if the model keeps calling tools anyway; **Fail block** fails the block if the agent asks for more tools. `<agent.stopReason>` reports whether a limit stopped the run.
147
148
-**Reasoning effort / Thinking level.** Controls supported models' reasoning budget. Higher settings can increase latency and token use; evaluate the result on your task.
148
149
-**Prompt caching.** Reuses a stable system prompt and tool definitions on supported Claude models. Sim uses a five-minute cache. Cache-write pricing, read discounts, and minimum prefix length depend on the model; check [Anthropic's current requirements](https://platform.claude.com/docs/en/build-with-claude/prompt-caching#cache-limitations). Enable it for repeated runs sharing a prefix; a one-off run can pay a write premium without a later cache hit.
149
150
-**API key.** Your key for the chosen provider. Hidden when hosted Sim supplies a key for the selected model, including Jev.
@@ -163,6 +164,7 @@ After the agent runs, later blocks read its result by name:
163
164
|`<agent.toolCalls>`| The tools the agent called, with their inputs and results |
164
165
|`<agent.model>`| The model that ran the step |
165
166
|`<agent.cost>`| Estimated cost in USD, as an object with `input`, `output`, and `total`; read `<agent.cost.total>` for the total |
167
+
|`<agent.stopReason>`| With **Limit run** on: `completed`, or the limit that stopped the run: `spend_limit`, `step_limit`, or `time_limit`|
166
168
167
169
The cost object can include additional fields such as tool costs or pricing. The Logs UI and run-resource API use credits. See [Costs](/platform/costs) for units and billing behavior.
@@ -422,6 +428,56 @@ Return ONLY the JSON array.`,
422
428
not: true,
423
429
},
424
430
},
431
+
{
432
+
id: 'runLimits',
433
+
title: 'Limit run',
434
+
type: 'switch',
435
+
description:
436
+
'A step is one model response whose tool calls run. Limits stop the agent from starting new steps; a call or tool already running finishes first. Spend is measured like the cost output, tools included.',
437
+
defaultValue: false,
438
+
mode: 'advanced',
439
+
condition: {
440
+
field: 'model',
441
+
value: MODELS_WITHOUT_CHAT_CONTROLS,
442
+
not: true,
443
+
},
444
+
},
445
+
{
446
+
id: 'maxRunCost',
447
+
title: 'Max spend (USD)',
448
+
type: 'short-input',
449
+
placeholder: 'No limit',
450
+
mode: 'advanced',
451
+
condition: RUN_LIMIT_FIELD_CONDITION,
452
+
},
453
+
{
454
+
id: 'maxSteps',
455
+
title: 'Max steps',
456
+
type: 'short-input',
457
+
placeholder: 'No limit',
458
+
mode: 'advanced',
459
+
condition: RUN_LIMIT_FIELD_CONDITION,
460
+
},
461
+
{
462
+
id: 'maxRunDuration',
463
+
title: 'Max duration (seconds)',
464
+
type: 'short-input',
465
+
placeholder: 'No limit',
466
+
mode: 'advanced',
467
+
condition: RUN_LIMIT_FIELD_CONDITION,
468
+
},
469
+
{
470
+
id: 'onRunLimit',
471
+
title: 'When a limit is reached',
472
+
type: 'dropdown',
473
+
options: [
474
+
{label: 'Return best answer',id: 'finalize'},
475
+
{label: 'Fail block',id: 'fail'},
476
+
],
477
+
defaultValue: 'finalize',
478
+
mode: 'advanced',
479
+
condition: RUN_LIMIT_FIELD_CONDITION,
480
+
},
425
481
{
426
482
id: 'responseFormat',
427
483
title: 'Response Format',
@@ -614,6 +670,24 @@ Return ONLY the JSON array.`,
614
670
type: 'boolean',
615
671
description: 'Cache the system prompt and tool definitions on models that support it',
616
672
},
673
+
runLimits: {
674
+
type: 'boolean',
675
+
description: 'Stop the agent at the spend, step, and duration limits below',
676
+
},
677
+
maxRunCost: {type: 'number',description: 'Most this block may be billed, in USD'},
678
+
maxSteps: {
679
+
type: 'number',
680
+
description: 'Most model responses whose tool calls may run',
681
+
},
682
+
maxRunDuration: {
683
+
type: 'number',
684
+
description: 'Seconds after which the agent starts no new steps',
685
+
},
686
+
onRunLimit: {
687
+
type: 'string',
688
+
description:
689
+
'When the agent still wants tools past a limit: finalize to answer without them, or fail to fail the block',
0 commit comments