Skip to content

Commit fe7de5d

Browse files
committed
feat(agent): add run limits for spend, steps, and duration
Adds a "Limit run" switch to the Agent block's advanced settings with max spend (USD), max steps, and max duration, and a choice of what happens when the agent still wants tools past a limit: return a final answer without tools (default) or fail the block. A new stopReason output reports whether a limit stopped the run. A step is one model response whose tool calls run. Limits are enforced at the two seams every provider already shares, so no provider tool loop changes: prepareConversationGeneration turns the agent's tools off in each protocol's own final-answer shape once a limit is reached, and executeProviderTool refuses tools past a limit. Calls that offer no agent tools (structured-output passes, Bedrock's forced structured_output tool) are never limited, so a run that answers within its limits is unaffected. One budget spans the selected model and its fallbacks; spend counts each attempt's own billed cost, excluding what it carried in from earlier attempts. Refusals are not journaled, so a retry with a fresh budget runs the call. Also renames isConversationContextError to isPermanentGenerationError, since it now covers the run-limit and usage-limit stops, and shares the billed user-stop construction between both stops.
1 parent 9a66fee commit fe7de5d

29 files changed

Lines changed: 681 additions & 51 deletions

File tree

‎apps/docs/content/docs/workflows/blocks/agent.mdx‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -121,7 +121,7 @@ Set **Response Format** to a JSON Schema when later blocks need named fields. Na
121121

122122
Malformed JSON can fall back to plain text with `_responseFormatWarning`. Sim does not perform a general post-parse schema check on every provider result, so validate fields downstream before consequential actions.
123123

124-
Avoid schema fields named `model`, `tokens`, `toolCalls`, `providerTiming`, or `cost`. Sim's standard metadata overwrites those fields after parsing, even when a metadata value is absent.
124+
Avoid schema fields named `model`, `tokens`, `toolCalls`, `providerTiming`, or `cost`. Sim's standard metadata overwrites those fields after parsing, even when a metadata value is absent. With **Limit run** on, `stopReason` is overwritten too.
125125

126126
```json
127127
{
@@ -144,6 +144,7 @@ Some settings live under advanced, or appear only for models that support them:
144144

145145
- **Temperature.** Controls variation on models that support it. Lower values tend to reduce variation; test accuracy on representative inputs rather than treating temperature as a factuality guarantee.
146146
- **Max output tokens.** Caps the response length. Defaults to the model's full limit.
147+
- **Limit run.** Stops a long tool-using run at a **max spend** in USD, a **max number of steps**, or a **max duration** in seconds. A step is one model response whose tool calls run. Leave a field empty for no limit. A limit acts only when the agent wants to keep using tools, so a call or tool that is already running finishes first, and a run that finishes before reaching a limit is unaffected. Spend is measured the same way as `<agent.cost>`, including tool costs, so a model Sim does not bill for, such as one on your own API key, adds only its tool costs. One budget covers the selected model and its fallbacks, and each try of **Retry on fail** starts a new one. **When a limit is reached** chooses what happens: **Return best answer** (the default) asks the model for a final answer without tools, which can spend slightly past the limit, and fails the block only if the model keeps calling tools anyway; **Fail block** fails the block if the agent asks for more tools. `<agent.stopReason>` reports whether a limit stopped the run.
147148
- **Reasoning effort / Thinking level.** Controls supported models' reasoning budget. Higher settings can increase latency and token use; evaluate the result on your task.
148149
- **Prompt caching.** Reuses a stable system prompt and tool definitions on supported Claude models. Sim uses a five-minute cache. Cache-write pricing, read discounts, and minimum prefix length depend on the model; check [Anthropic's current requirements](https://platform.claude.com/docs/en/build-with-claude/prompt-caching#cache-limitations). Enable it for repeated runs sharing a prefix; a one-off run can pay a write premium without a later cache hit.
149150
- **API key.** Your key for the chosen provider. Hidden when hosted Sim supplies a key for the selected model, including Jev.
@@ -163,6 +164,7 @@ After the agent runs, later blocks read its result by name:
163164
| `<agent.toolCalls>` | The tools the agent called, with their inputs and results |
164165
| `<agent.model>` | The model that ran the step |
165166
| `<agent.cost>` | Estimated cost in USD, as an object with `input`, `output`, and `total`; read `<agent.cost.total>` for the total |
167+
| `<agent.stopReason>` | With **Limit run** on: `completed`, or the limit that stopped the run: `spend_limit`, `step_limit`, or `time_limit` |
166168

167169
The cost object can include additional fields such as tool costs or pricing. The Logs UI and run-resource API use credits. See [Costs](/platform/costs) for units and billing behavior.
168170

‎apps/sim/blocks/blocks/agent.ts‎

Lines changed: 80 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
import { createLogger } from '@sim/logger'
22
import { AgentIcon } from '@/components/icons'
33
import { getModelFallbackSubBlock, MODEL_FALLBACK_INPUTS } from '@/blocks/model-fallbacks'
4-
import type { BlockConfig } from '@/blocks/types'
4+
import type { BlockConfig, SubBlockConfig } from '@/blocks/types'
55
import { AuthMode, IntegrationType } from '@/blocks/types'
66
import {
77
getAgentModelOptions,
@@ -42,6 +42,12 @@ const MODELS_WITHOUT_MEMORY = getModelsWithoutMemory()
4242
const EVALUATION_MODELS = getEvaluationModels()
4343
const MODELS_WITHOUT_CHAT_CONTROLS = [...MODELS_WITH_DEEP_RESEARCH, ...EVALUATION_MODELS]
4444

45+
const RUN_LIMIT_FIELD_CONDITION: SubBlockConfig['condition'] = {
46+
field: 'runLimits',
47+
value: true,
48+
and: { field: 'model', value: MODELS_WITHOUT_CHAT_CONTROLS, not: true },
49+
}
50+
4551
// Helper function to get the tool ID from a block type
4652
const getToolIdFromBlock = (blockType: string): string | undefined => {
4753
try {
@@ -422,6 +428,56 @@ Return ONLY the JSON array.`,
422428
not: true,
423429
},
424430
},
431+
{
432+
id: 'runLimits',
433+
title: 'Limit run',
434+
type: 'switch',
435+
description:
436+
'A step is one model response whose tool calls run. Limits stop the agent from starting new steps; a call or tool already running finishes first. Spend is measured like the cost output, tools included.',
437+
defaultValue: false,
438+
mode: 'advanced',
439+
condition: {
440+
field: 'model',
441+
value: MODELS_WITHOUT_CHAT_CONTROLS,
442+
not: true,
443+
},
444+
},
445+
{
446+
id: 'maxRunCost',
447+
title: 'Max spend (USD)',
448+
type: 'short-input',
449+
placeholder: 'No limit',
450+
mode: 'advanced',
451+
condition: RUN_LIMIT_FIELD_CONDITION,
452+
},
453+
{
454+
id: 'maxSteps',
455+
title: 'Max steps',
456+
type: 'short-input',
457+
placeholder: 'No limit',
458+
mode: 'advanced',
459+
condition: RUN_LIMIT_FIELD_CONDITION,
460+
},
461+
{
462+
id: 'maxRunDuration',
463+
title: 'Max duration (seconds)',
464+
type: 'short-input',
465+
placeholder: 'No limit',
466+
mode: 'advanced',
467+
condition: RUN_LIMIT_FIELD_CONDITION,
468+
},
469+
{
470+
id: 'onRunLimit',
471+
title: 'When a limit is reached',
472+
type: 'dropdown',
473+
options: [
474+
{ label: 'Return best answer', id: 'finalize' },
475+
{ label: 'Fail block', id: 'fail' },
476+
],
477+
defaultValue: 'finalize',
478+
mode: 'advanced',
479+
condition: RUN_LIMIT_FIELD_CONDITION,
480+
},
425481
{
426482
id: 'responseFormat',
427483
title: 'Response Format',
@@ -614,6 +670,24 @@ Return ONLY the JSON array.`,
614670
type: 'boolean',
615671
description: 'Cache the system prompt and tool definitions on models that support it',
616672
},
673+
runLimits: {
674+
type: 'boolean',
675+
description: 'Stop the agent at the spend, step, and duration limits below',
676+
},
677+
maxRunCost: { type: 'number', description: 'Most this block may be billed, in USD' },
678+
maxSteps: {
679+
type: 'number',
680+
description: 'Most model responses whose tool calls may run',
681+
},
682+
maxRunDuration: {
683+
type: 'number',
684+
description: 'Seconds after which the agent starts no new steps',
685+
},
686+
onRunLimit: {
687+
type: 'string',
688+
description:
689+
'When the agent still wants tools past a limit: finalize to answer without them, or fail to fail the block',
690+
},
617691
...MODEL_FALLBACK_INPUTS,
618692
tools: { type: 'json', description: 'Available tools configuration' },
619693
skills: { type: 'json', description: 'Selected skills configuration' },
@@ -634,6 +708,11 @@ Return ONLY the JSON array.`,
634708
description: 'Provider timing information',
635709
},
636710
cost: { type: 'json', description: 'Cost of the API call' },
711+
stopReason: {
712+
type: 'string',
713+
description: 'Why the agent stopped: completed, spend_limit, step_limit, or time_limit',
714+
condition: { field: 'runLimits', value: true },
715+
},
637716
interactionId: {
638717
type: 'string',
639718
description: 'Interaction ID for multi-turn deep research follow-ups',

‎apps/sim/executor/execution/block-executor.ts‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1423,6 +1423,9 @@ export class BlockExecutor {
14231423
providerTiming: executionOutput.providerTiming,
14241424
cost: executionOutput.cost,
14251425
model: executionOutput.model,
1426+
...(executionOutput.stopReason !== undefined && {
1427+
stopReason: executionOutput.stopReason,
1428+
}),
14261429
}
14271430
parsedForFormat = true
14281431
} catch (error) {

‎apps/sim/executor/handlers/agent/handler.ts‎

Lines changed: 29 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -57,7 +57,7 @@ import {
5757
import { getAllBlocks, getBlock } from '@/blocks'
5858
import { assembleCustomBlockInputMapping, isCustomBlockType } from '@/blocks/custom/build-config'
5959
import type { BlockOutput } from '@/blocks/types'
60-
import { normalizeFileInput } from '@/blocks/utils'
60+
import { normalizeFileInput, parseOptionalNumberInput } from '@/blocks/utils'
6161
import {
6262
assertPermissionsAllowed,
6363
validateBlockType,
@@ -126,6 +126,7 @@ import {
126126
getInlineHydrationMaxBytes,
127127
} from '@/providers/file-attachments.server'
128128
import { isAutoModel, isEvaluationModel, SIM_AUTO_MODEL_ID } from '@/providers/models'
129+
import { AgentRunLimits } from '@/providers/run-limits'
129130
import { isRunAbortError } from '@/providers/streaming-tool-loop-shared'
130131
import {
131132
type ProviderToolInputProvenance,
@@ -231,6 +232,7 @@ interface ExecuteAcrossModelsConfig {
231232
resultRegistry: ResolvedSecretTraceRegistry | undefined
232233
providerErrorRegistry: ResolvedSecretTraceRegistry | undefined
233234
agentConversation?: AgentTurnSession
235+
runLimits?: AgentRunLimits
234236
}
235237

236238
interface FormattedAgentTools {
@@ -519,6 +521,7 @@ export class AgentBlockHandler implements BlockHandler {
519521
}
520522

521523
const streamingConfig = this.getStreamingConfig(ctx, block)
524+
const runLimits = this.createRunLimits(filteredInputs)
522525
const agentConversation =
523526
filteredInputs.memoryType &&
524527
filteredInputs.memoryType !== 'none' &&
@@ -649,6 +652,7 @@ export class AgentBlockHandler implements BlockHandler {
649652
resultRegistry,
650653
providerErrorRegistry,
651654
agentConversation,
655+
runLimits,
652656
})
653657
if (servedRegistry) ctx.resolvedSecretTraceRegistry = servedRegistry
654658

@@ -667,6 +671,7 @@ export class AgentBlockHandler implements BlockHandler {
667671
if (this.isStreamingExecution(result)) {
668672
const streamingResult = result as StreamingExecution
669673
streamingResult.diagnosticResolvedSecretTraceRegistry = providerErrorRegistry
674+
runLimits?.reportOn(streamingResult.execution.output)
670675
if (filteredInputs.memoryType && filteredInputs.memoryType !== 'none') {
671676
return this.wrapStreamForMemoryPersistence(
672677
ctx,
@@ -689,12 +694,30 @@ export class AgentBlockHandler implements BlockHandler {
689694
)
690695
}
691696

697+
if (runLimits && isRecordLike(result)) runLimits.reportOn(result)
692698
return result
693699
} finally {
694700
settlePrivateAgentSelectors()
695701
}
696702
}
697703

704+
/** The limits a block set while Limit run is on; parsed once and shared by every model it tries. */
705+
private createRunLimits(inputs: AgentInputs): AgentRunLimits | undefined {
706+
if (inputs.runLimits !== true) return undefined
707+
const maxRunDuration = parseOptionalNumberInput(inputs.maxRunDuration, 'Max duration', {
708+
min: 1,
709+
})
710+
return new AgentRunLimits({
711+
maxCost: parseOptionalNumberInput(inputs.maxRunCost, 'Max spend', { min: 0.01 }),
712+
maxSteps: parseOptionalNumberInput(inputs.maxSteps, 'Max steps', {
713+
integer: true,
714+
min: 1,
715+
}),
716+
maxDurationMs: maxRunDuration === undefined ? undefined : maxRunDuration * 1000,
717+
onLimit: inputs.onRunLimit === 'fail' ? 'fail' : 'finalize',
718+
})
719+
}
720+
698721
private resolveToolUsageControls(
699722
tools: ToolInput[],
700723
canonicalModes?: Record<string, 'basic' | 'advanced'>
@@ -2755,7 +2778,8 @@ export class AgentBlockHandler implements BlockHandler {
27552778
config.responseFormat,
27562779
resultRegistry,
27572780
config.providerErrorRegistry,
2758-
config.agentConversation
2781+
config.agentConversation,
2782+
config.runLimits
27592783
)
27602784
if ((hasNext || config.retryPrimaryOnStreamStart) && this.isStreamingExecution(result)) {
27612785
result = await this.primeStreamingExecution(result as StreamingExecution)
@@ -2974,7 +2998,8 @@ export class AgentBlockHandler implements BlockHandler {
29742998
responseFormat: any,
29752999
modelRuntimeRegistry: ResolvedSecretTraceRegistry | undefined,
29763000
providerErrorRegistry: ResolvedSecretTraceRegistry | undefined,
2977-
agentConversation?: AgentTurnSession
3001+
agentConversation?: AgentTurnSession,
3002+
runLimits?: AgentRunLimits
29783003
): Promise<BlockOutput | StreamingExecution> {
29793004
const providerId = providerRequest.provider
29803005
const model = providerRequest.model
@@ -3056,6 +3081,7 @@ export class AgentBlockHandler implements BlockHandler {
30563081
agentMemoryContext: agentConversation
30573082
? { historyTokens: providerRequest.memoryHistoryTokens }
30583083
: undefined,
3084+
runLimits,
30593085
}
30603086
)
30613087

‎apps/sim/executor/handlers/agent/types.ts‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,12 @@ export interface AgentInputs {
5050
verbosity?: string
5151
thinkingLevel?: string
5252
promptCaching?: boolean
53+
// Run limits, applied only while `runLimits` is on
54+
runLimits?: boolean
55+
maxRunCost?: string | number
56+
maxSteps?: string | number
57+
maxRunDuration?: string | number
58+
onRunLimit?: 'finalize' | 'fail'
5359
files?: unknown
5460
/** Ordered models tried when the request to `model` fails; see `normalizeFallbackModels`. */
5561
fallbackModels?: Array<Partial<FallbackModelEntry> & { model: string }>

‎apps/sim/providers/anthropic/core.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,7 +21,7 @@ import {
2121
createReadableStreamFromAnthropicStream,
2222
} from '@/providers/anthropic/utils'
2323
import {
24-
isConversationContextError,
24+
isPermanentGenerationError,
2525
prepareConversationGeneration,
2626
} from '@/providers/conversation-generation'
2727
import {
@@ -1006,7 +1006,7 @@ export async function executeAnthropicProviderRequest(
10061006
}
10071007
)
10081008

1009-
if (isAbortError(error) || request.abortSignal?.aborted || isConversationContextError(error)) {
1009+
if (isAbortError(error) || request.abortSignal?.aborted || isPermanentGenerationError(error)) {
10101010
throw error
10111011
}
10121012

‎apps/sim/providers/azure-openai/index.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -26,7 +26,7 @@ import {
2626
isResponsesEndpoint,
2727
} from '@/providers/azure-openai/utils'
2828
import {
29-
isConversationContextError,
29+
isPermanentGenerationError,
3030
prepareConversationGeneration,
3131
} from '@/providers/conversation-generation'
3232
import {
@@ -716,7 +716,7 @@ async function executeChatCompletionsRequest(
716716
}
717717
)
718718

719-
if (isAbortError(error) || request.abortSignal?.aborted || isConversationContextError(error)) {
719+
if (isAbortError(error) || request.abortSignal?.aborted || isPermanentGenerationError(error)) {
720720
throw error
721721
}
722722

‎apps/sim/providers/bedrock/index.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -31,7 +31,7 @@ import {
3131
} from '@/providers/bedrock/utils'
3232
import { getCachedProviderClient } from '@/providers/client-cache'
3333
import {
34-
isConversationContextError,
34+
isPermanentGenerationError,
3535
prepareConversationGeneration,
3636
} from '@/providers/conversation-generation'
3737
import {
@@ -1060,7 +1060,7 @@ export const bedrockProvider: ProviderConfig = {
10601060
if (
10611061
isAbortError(error) ||
10621062
request.abortSignal?.aborted ||
1063-
isConversationContextError(error)
1063+
isPermanentGenerationError(error)
10641064
) {
10651065
throw error
10661066
}

‎apps/sim/providers/cerebras/index.ts‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ import { MAX_TOOL_ITERATIONS } from '@/providers'
77
import { formatMessagesForProvider } from '@/providers/attachments'
88
import type { CerebrasResponse } from '@/providers/cerebras/types'
99
import {
10-
isConversationContextError,
10+
isPermanentGenerationError,
1111
prepareConversationGeneration,
1212
} from '@/providers/conversation-generation'
1313
import {
@@ -676,7 +676,7 @@ export const cerebrasProvider: ProviderConfig = {
676676
if (
677677
isAbortError(error) ||
678678
request.abortSignal?.aborted ||
679-
isConversationContextError(error)
679+
isPermanentGenerationError(error)
680680
) {
681681
throw error
682682
}

‎apps/sim/providers/conversation-generation.ts‎

Lines changed: 8 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -21,6 +21,7 @@ import {
2121
isConversationHistoryNotice,
2222
} from '@/providers/conversation-metadata'
2323
import { getConversationModelLimits } from '@/providers/conversation-model'
24+
import { applyRunLimits, RunLimitError } from '@/providers/run-limits'
2425
import { assertRunWithinUsageLimit, RunUsageLimitError } from '@/providers/run-usage-limit'
2526
import { isAbortError } from '@/providers/streaming-tool-loop-shared'
2627
import type { Message, ProviderRequest } from '@/providers/types'
@@ -117,13 +118,15 @@ function contextError(message: string): Error {
117118

118119
/**
119120
* Provider wrappers must preserve the permanent failures this module raises, context faults and
120-
* the run's usage-limit stop, so fallback cannot restart them and the stop keeps its billed cost.
121+
* the run's usage-limit and run-limit stops, so fallback cannot restart them and a stop keeps its
122+
* billed cost.
121123
*/
122-
export function isConversationContextError(error: unknown): boolean {
124+
export function isPermanentGenerationError(error: unknown): boolean {
123125
return (
124126
error instanceof AgentContextLimitError ||
125127
error instanceof AgentContextProtocolError ||
126-
error instanceof RunUsageLimitError
128+
error instanceof RunUsageLimitError ||
129+
error instanceof RunLimitError
127130
)
128131
}
129132

@@ -288,10 +291,11 @@ function nativeSummaryMessage(summary: Message, protocol: ConversationProtocol):
288291
export async function prepareConversationGeneration<T>(
289292
request: ProviderRequest,
290293
protocol: ConversationProtocol,
291-
payload: T
294+
requestedPayload: T
292295
): Promise<T> {
293296
const runtime = getConversationRequestContext(request)
294297
await assertRunWithinUsageLimit(runtime)
298+
const payload = applyRunLimits(runtime, request, protocol, requestedPayload)
295299
if (!runtime?.agentConversation) return payload
296300
request.abortSignal?.throwIfAborted()
297301
if (!isRecordLike(payload))

0 commit comments

Comments
 (0)