[Claimed #1777] fix: add variable substitution to keys tool

[github-actions]

Mirrored from external contributor PR #1777 after approval by @pirate.

Original author: @trillville
Original PR: #1777
Approved source head SHA: 9c30883ba601e39d6e17ddfe91dfd72f00d4f7cd

@trillville, please continue any follow-up discussion on this mirrored PR. When the external PR gets new commits, this same internal PR will be marked stale until the latest external commit is approved and refreshed here.

Original description

• Check the documentation for relevant information
• Search existing issues to avoid duplicates

Fixes #1776

Fix

• Accept variables parameter in keysTool (matching typeTool)
• Call substituteVariables() before page.type() in the method === "type" branch
• Pass variables to keysTool in createAgentTools
• Return original token in result to avoid exposing sensitive values to LLM

---

Summary by cubic

Fixes variable substitution in the keys tool when method="type" so %variableName% tokens are resolved before typing. Aligns behavior with the type tool and prevents secret leakage in tool outputs.

• Bug Fixes
  • Substitute %variableName% in keys(method="type") via substituteVariables before page.type().
  • Accept and pass variables to keysTool; update input schema to list available variables so the agent knows %var% syntax.
  • Return the original token string in the tool result to avoid exposing substituted secrets.

^{Written for commit 9c30883. Summary will update on new commits. Review in cubic}

[changeset-bot]

⚠️ No Changeset found

Latest commit: 9c30883

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[github-actions]

This mirrored PR was closed without merge. The original external PR #1777 has been reopened and relabeled as awaiting approval.

[greptile-apps]

Greptile Summary

This PR fixes variable substitution (%variableName% token replacement) in the keys tool when method="type", aligning it with the existing type tool's behavior. Previously, keysTool did not accept or apply variables, meaning secrets passed via the variables API were typed literally as tokens rather than being resolved.

Key changes:

• keys.ts: Accepts an optional variables parameter, dynamically builds the valueDescription to advertise available variables (when present), and calls substituteVariables() before page.type() in the method === "type" branch. Returns the original token string in the tool result to avoid leaking resolved secret values to the LLM.
• index.ts: Passes variables through to keysTool in createAgentTools, matching how other variable-aware tools (actTool, fillFormTool, typeTool) are wired.

Issue found: The valueDescription string advertises %variableName% substitution for the shared value field (used for both method="type" and method="press"), but substituteVariables() is only called inside the method === "type" branch. If the LLM is influenced by the description to use a variable token with method="press", the literal token string (e.g. %apiKey%) will be forwarded to page.keyPress(), causing an unexpected failure rather than a substitution.

Confidence Score: 3/5

• Safe to merge with minor concern — the logic issue only manifests if the LLM misuses variable tokens with method="press", which is an unlikely but possible edge case driven by a misleading schema description.
• The core fix (substituting variables before page.type in method="type") is correct and consistent with the type tool pattern. Secret values are properly kept out of LLM-visible results. However, the valueDescription is shared across both methods and advertises variable substitution without clarifying it only applies to method="type", which could cause silent failures if the LLM uses tokens with method="press".
• packages/core/lib/v3/agent/tools/keys.ts — the valueDescription on lines 9–11 needs to scope the variable substitution hint to method="type" only.

Important Files Changed

Filename	Overview
packages/core/lib/v3/agent/tools/keys.ts	Adds variable substitution support for method="type"; the value field description advertises substitution for both methods but the substitution only runs in the type branch, which could mislead the LLM into using tokens with method="press".
packages/core/lib/v3/agent/tools/index.ts	One-line change to forward variables to keysTool; straightforward and correct.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A["keysTool called\n(v3, variables?)"] --> B{variables\nprovided?}
    B -- Yes --> C["Build valueDescription\nwith available var names"]
    B -- No --> D["Use default\nvalueDescription"]
    C --> E["Return tool(...)"]
    D --> E

    E --> F["execute called\n(method, value, repeat)"]
    F --> G{method?}

    G -- type --> H["substituteVariables(value, variables)\n→ actualValue"]
    H --> I["page.type(actualValue, delay:100)\n× times"]
    I --> J["recordAgentReplayStep\nwith original value token"]
    J --> K["return { success, method,\nvalue ← original token, times }"]

    G -- press --> L["page.keyPress(value, delay:100)\n× times\n⚠️ NO substitution"]
    L --> M["recordAgentReplayStep"]
    M --> N["return { success, method, value, times }"]

Loading

_{Last reviewed commit: 9c30883}

[greptile-apps]

Variable substitution description applies to both methods

The valueDescription hints that %variableName% substitution is available for the value field, but substituteVariables() is only called inside the method === "type" branch (line 43). If the LLM reads this description and uses method="press" with a token like %apiKey%, the literal string %apiKey% will be forwarded directly to page.keyPress(), which will either fail or press unexpected keys — the substitution silently does not occur.

The description should be scoped to clarify that variable substitution only works with method="type":

Suggested change

	const valueDescription = hasVariables
	? `The text to type, or the key/combo to press (Enter, Tab, Cmd+A). Use %variableName% to substitute a variable value. Available: ${Object.keys(variables).join(", ")}`
	: "The text to type, or the key/combo to press (Enter, Tab, Cmd+A)";
	const valueDescription = hasVariables
	? `The text to type (method="type"), or the key/combo to press (method="press") (Enter, Tab, Cmd+A). When using method="type", use %variableName% to substitute a variable value. Available: ${Object.keys(variables).join(", ")}`
	: "The text to type, or the key/combo to press (Enter, Tab, Cmd+A)";

[cubic-dev-ai]

2 issues found across 2 files

Confidence score: 2/5

• There is a high-confidence security risk in packages/core/lib/v3/agent/tools/keys.ts: on the error path, substituted secret values can be exposed to the LLM instead of staying as %variableName% tokens.
• packages/core/lib/v3/agent/tools/keys.ts also has a behavior mismatch where valueDescription advertises %variableName% substitution, but substituteVariables() only runs for method === "type", so other methods may fail or behave inconsistently.
• Given the concrete secret-leak impact (severity 7/10, confidence 8/10) plus a second medium-severity logic issue, this looks risky to merge without a fix.
• Pay close attention to packages/core/lib/v3/agent/tools/keys.ts - sanitize exception paths to avoid leaking resolved secrets and align variable substitution behavior with the documented tool contract.

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="packages/core/lib/v3/agent/tools/keys.ts">

<violation number="1" location="packages/core/lib/v3/agent/tools/keys.ts:10">
P2: The `valueDescription` tells the LLM that `%variableName%` substitution is available for the `value` field, but `substituteVariables()` is only called in the `method === "type"` branch. If the agent uses `method="press"` with a variable token (e.g., `%apiKey%`), the literal `%apiKey%` string will be passed to `page.keyPress()`, which will either fail or press unintended keys. Scope the variable-substitution hint to `method="type"` only.</violation>

<violation number="2" location="packages/core/lib/v3/agent/tools/keys.ts:45">
P1: Custom agent: **Exception and error message sanitization**

The error path can leak substituted secret values to the LLM. The success path correctly returns the original `%variableName%` token, but if `page.type(actualValue)` throws, the catch block returns the raw `(error as Error).message` without sanitizing out the substituted secret. Wrap the `page.type()` call so that any error in this branch returns a sanitized message that uses the original `value` token, not `actualValue`.</violation>
</file>

Architecture diagram

sequenceDiagram
    participant LLM as Agent (LLM)
    participant Tool as keysTool
    participant Utils as substituteVariables()
    participant Browser as Playwright Page
    participant Recorder as V3 Recorder

    Note over LLM,Tool: Tool initialized with available "variables" map

    LLM->>Tool: execute({ method, value, repeat })
    
    alt method === "type"
        Tool->>Utils: NEW: substituteVariables(value, variables)
        Utils-->>Tool: actualValue (e.g. "secret123")
        
        loop repeat count
            Tool->>Browser: CHANGED: page.type(actualValue)
        end
        
        Tool->>Recorder: recordAgentReplayStep(value)
        Note right of Tool: NEW: instruction uses original token <br/> to prevent secret leakage in logs
        
        Tool-->>LLM: { success: true, value }
        Note right of Tool: NEW: returns original token (e.g. %password%) <br/> to LLM context
        
    else method === "press"
        loop repeat count
            Tool->>Browser: page.press(value)
        end
        Tool->>Recorder: recordAgentReplayStep(value)
        Tool-->>LLM: { success: true, value }
    end

Loading

_{Reply with feedback, questions, or to request a fix. Tag @cubic-dev-ai to re-run a review.}

[cubic-dev-ai]

P1: Custom agent: Exception and error message sanitization

The error path can leak substituted secret values to the LLM. The success path correctly returns the original %variableName% token, but if page.type(actualValue) throws, the catch block returns the raw (error as Error).message without sanitizing out the substituted secret. Wrap the page.type() call so that any error in this branch returns a sanitized message that uses the original value token, not actualValue.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/core/lib/v3/agent/tools/keys.ts, line 45:

<comment>The error path can leak substituted secret values to the LLM. The success path correctly returns the original `%variableName%` token, but if `page.type(actualValue)` throws, the catch block returns the raw `(error as Error).message` without sanitizing out the substituted secret. Wrap the `page.type()` call so that any error in this branch returns a sanitized message that uses the original `value` token, not `actualValue`.</comment>

<file context>
@@ -36,14 +39,17 @@ Use method="press" for navigation keys (Enter, Tab, Escape, Backspace, arrows) a
+          const actualValue = substituteVariables(value, variables);
           for (let i = 0; i < times; i++) {
-            await page.type(value, { delay: 100 });
+            await page.type(actualValue, { delay: 100 });
           }
           v3.recordAgentReplayStep({
</file context>

Suggested change

	await page.type(actualValue, { delay: 100 });
	try {
	await page.type(actualValue, { delay: 100 });
	} catch (e) {
	return {
	success: false,
	error: `Failed to type "${value}"`,
	};
	}

[cubic-dev-ai]

P2: The valueDescription tells the LLM that %variableName% substitution is available for the value field, but substituteVariables() is only called in the method === "type" branch. If the agent uses method="press" with a variable token (e.g., %apiKey%), the literal %apiKey% string will be passed to page.keyPress(), which will either fail or press unintended keys. Scope the variable-substitution hint to method="type" only.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At packages/core/lib/v3/agent/tools/keys.ts, line 10:

<comment>The `valueDescription` tells the LLM that `%variableName%` substitution is available for the `value` field, but `substituteVariables()` is only called in the `method === "type"` branch. If the agent uses `method="press"` with a variable token (e.g., `%apiKey%`), the literal `%apiKey%` string will be passed to `page.keyPress()`, which will either fail or press unintended keys. Scope the variable-substitution hint to `method="type"` only.</comment>

<file context>
@@ -1,21 +1,24 @@
+export const keysTool = (v3: V3, variables?: Variables) => {
+  const hasVariables = variables && Object.keys(variables).length > 0;
+  const valueDescription = hasVariables
+    ? `The text to type, or the key/combo to press (Enter, Tab, Cmd+A). Use %variableName% to substitute a variable value. Available: ${Object.keys(variables).join(", ")}`
+    : "The text to type, or the key/combo to press (Enter, Tab, Cmd+A)";
+
</file context>

Suggested change

	? `The text to type, or the key/combo to press (Enter, Tab, Cmd+A). Use %variableName% to substitute a variable value. Available: ${Object.keys(variables).join(", ")}`
	? `The text to type (method="type"), or the key/combo to press (method="press") (Enter, Tab, Cmd+A). When using method="type", use %variableName% to substitute a variable value. Available: ${Object.keys(variables).join(", ")}`

[a7med3liamin]

Hey @pirate — I opened #1978 which fixes the cache replay side of this same issue. Your PR (#1813) fixes variable substitution during live execution of the keys tool, but replayAgentKeysStep in AgentCache has a separate code path that still types raw %variableName% tokens instead of resolving them. #1978 addresses that. Minimal change — 1 import, passing variables through, and a substituteVariables() call.

[pirate]

got it, thanks for the fix @a7med3liamin, I will close this PR and reclaim the new one