From bffad42a3ac24a2ee09005452e9eeed9bfcf6e1f Mon Sep 17 00:00:00 2001 From: DjDeveloperr Date: Thu, 7 May 2026 00:50:51 -0400 Subject: [PATCH 1/6] Add device-bound test API and Maestro YAML runner --- README.md | 6 + docs/api/rest.md | 36 + docs/cli/commands.md | 10 + docs/guide/testing.md | 16 +- packages/simdeck-test/dist/index.d.ts | 255 +++--- packages/simdeck-test/dist/index.js | 1045 ++++++++++++------------- packages/simdeck-test/src/index.ts | 895 ++++++++++++--------- server/Cargo.lock | 21 + server/Cargo.toml | 2 + server/src/api/routes.rs | 364 ++++++++- server/src/main.rs | 531 ++++++++++++- skills/simdeck/SKILL.md | 8 +- 12 files changed, 2115 insertions(+), 1074 deletions(-) diff --git a/README.md b/README.md index c8c159c0..81b07452 100644 --- a/README.md +++ b/README.md @@ -240,6 +240,12 @@ healthy, and only stops daemons it started itself. Pass `udid` to `connect()` to make it the default for session methods; each method still accepts an explicit UDID as the first argument when needed. +Run common Maestro YAML flows against the same daemon-backed simulator API: + +```sh +simdeck maestro test flow.yaml --artifacts-dir artifacts/maestro +``` + ## NativeScript Inspector NativeScript apps can connect directly to the running server from JS and expose diff --git a/docs/api/rest.md b/docs/api/rest.md index dcf0c9f8..c4ebe7c8 100644 --- a/docs/api/rest.md +++ b/docs/api/rest.md @@ -198,7 +198,10 @@ Touch, edge-touch, and multi-touch coordinates are normalized from `0.0` to `1.0 | `GET` | `/api/simulators/{udid}/accessibility-point?x=120&y=240` | Element at a point | | `POST` | `/api/simulators/{udid}/query` | Query tree by selector | | `POST` | `/api/simulators/{udid}/wait-for` | Wait until selector appears | +| `POST` | `/api/simulators/{udid}/wait-for-not` | Wait until selector disappears | | `POST` | `/api/simulators/{udid}/assert` | Assert selector exists | +| `POST` | `/api/simulators/{udid}/assert-not` | Assert selector is absent | +| `POST` | `/api/simulators/{udid}/scroll-until-visible` | Scroll until selector appears | | `POST` | `/api/simulators/{udid}/batch` | Run multiple control steps | | `POST` | `/api/simulators/{udid}/inspector/request` | Call an in-app inspector method | @@ -219,6 +222,39 @@ Point query parameters: Every tree response reports the `source` used and may include a `fallbackReason`. +Selector endpoints accept compact accessibility selectors: + +```json +{ + "selector": { + "text": "Continue", + "id": "continue-button", + "elementType": "Button", + "enabled": true, + "regex": false + }, + "source": "auto", + "maxDepth": 8, + "limit": 20 +} +``` + +Selectors can match `text`, `id`, `label`, `value`, `elementType`, `index`, `enabled`, `checked`, `focused`, and `selected`. Set `regex: true` to use regular expression matching for string fields. + +`POST /api/simulators/{udid}/query` returns compact matches. `wait-for` and `assert` use the same body shape for positive checks. `wait-for-not` and `assert-not` perform negative checks. + +`POST /api/simulators/{udid}/scroll-until-visible` scrolls and polls until a selector appears: + +```json +{ + "selector": { "text": "Settings" }, + "direction": "down", + "timeoutMs": 10000 +} +``` + +`direction` accepts `up`, `down`, `left`, and `right`. + ## DevTools And WebKit | Method | Path | Purpose | diff --git a/docs/cli/commands.md b/docs/cli/commands.md index f30a19c4..1c7871e7 100644 --- a/docs/cli/commands.md +++ b/docs/cli/commands.md @@ -140,6 +140,16 @@ simdeck batch \ Use `wait-for` or `assert` steps instead of fixed sleeps when possible. +## Maestro YAML + +Run common Maestro flows through SimDeck's daemon-backed iOS Simulator API: + +```sh +simdeck maestro test flow.yaml --artifacts-dir artifacts/maestro +``` + +The compatibility runner supports the core local commands: `launchApp`, `openLink`, `tapOn`, `inputText`, `eraseText`, `pressKey`, `assertVisible`, `assertNotVisible`, `scrollUntilVisible`, `swipe`, `takeScreenshot`, and `waitForAnimationToEnd`. + ## Evidence ```sh diff --git a/docs/guide/testing.md b/docs/guide/testing.md index 28c59e8c..930353df 100644 --- a/docs/guide/testing.md +++ b/docs/guide/testing.md @@ -25,7 +25,7 @@ try { } ``` -`connect()` starts the project daemon if needed, reuses a healthy daemon, and only stops daemons it started itself. Pass `udid` to `connect()` to make it the default for session methods; methods still accept an explicit UDID as their first argument. +`connect()` starts the project daemon if needed, reuses a healthy daemon, and only stops daemons it started itself. Pass `udid` to `connect()` to make it the default for session methods; methods still accept an explicit UDID as their first argument. Use `sim.device("")` to create a session bound to another simulator. ## Useful Test Methods @@ -39,10 +39,22 @@ try { | `typeText()`, `key()`, `keySequence()` | Text and keyboard input | | `button()`, `home()`, `appSwitcher()` | System controls | | `tree()`, `query()`, `waitFor()`, `assert()` | UI state checks | +| `waitForNot()`, `assertNot()` | Negative UI state checks | +| `scrollUntilVisible()` | Scroll until a selector exists | | `screenshot()`, `record()`, `logs()` | Evidence capture | | `batch()` | Multi-step actions | -Selectors can match `id`, `label`, `value`, or `type`. +Selectors can match `text`, `id`, `label`, `value`, `type`, `index`, `enabled`, `checked`, `focused`, or `selected`. Set `regex: true` to treat string selector fields as regular expressions. + +## Maestro-Compatible YAML + +The CLI includes a compatibility runner for common Maestro YAML flows: + +```sh +simdeck maestro test flow.yaml --artifacts-dir artifacts/maestro +``` + +Supported commands include `launchApp`, `openLink`, `tapOn`, `inputText`, `eraseText`, `pressKey`, `assertVisible`, `assertNotVisible`, `scrollUntilVisible`, `swipe`, `takeScreenshot`, and `waitForAnimationToEnd`. Unsupported Maestro commands fail clearly so the flow can be adjusted or the compatibility layer can be expanded. ## Repository Tests diff --git a/packages/simdeck-test/dist/index.d.ts b/packages/simdeck-test/dist/index.d.ts index c9326e5f..f7182b04 100644 --- a/packages/simdeck-test/dist/index.d.ts +++ b/packages/simdeck-test/dist/index.d.ts @@ -1,147 +1,162 @@ export type SimDeckLaunchOptions = { - cliPath?: string; - projectRoot?: string; - keepDaemon?: boolean; - isolated?: boolean; - port?: number; - videoCodec?: "auto" | "hardware" | "software" | "h264-software"; - udid?: string; + cliPath?: string; + projectRoot?: string; + keepDaemon?: boolean; + isolated?: boolean; + port?: number; + videoCodec?: "auto" | "hardware" | "software" | "h264-software"; + udid?: string; }; export type QueryOptions = { - source?: - | "auto" - | "nativescript" - | "react-native" - | "flutter" - | "uikit" - | "native-ax" - | "android-uiautomator"; - maxDepth?: number; - includeHidden?: boolean; + source?: "auto" | "nativescript" | "react-native" | "flutter" | "swiftui" | "uikit" | "native-ax" | "android-uiautomator"; + maxDepth?: number; + includeHidden?: boolean; }; export type ElementSelector = { - id?: string; - label?: string; - value?: string; - type?: string; + text?: string; + id?: string; + label?: string; + value?: string; + type?: string; + index?: number; + enabled?: boolean; + checked?: boolean; + focused?: boolean; + selected?: boolean; + regex?: boolean; }; export type TapOptions = QueryOptions & { - durationMs?: number; - waitTimeoutMs?: number; - pollMs?: number; + durationMs?: number; + waitTimeoutMs?: number; + pollMs?: number; }; export type SwipeOptions = { - durationMs?: number; - steps?: number; + durationMs?: number; + steps?: number; }; export type GestureOptions = SwipeOptions & { - delta?: number; + delta?: number; }; export type TypeTextOptions = { - delayMs?: number; + delayMs?: number; }; export type KeySequenceOptions = { - delayMs?: number; + delayMs?: number; }; export type LogsOptions = { - backfill?: boolean; - seconds?: number; - limit?: number; - levels?: string[]; - processes?: string[]; - q?: string; + backfill?: boolean; + seconds?: number; + limit?: number; + levels?: string[]; + processes?: string[]; + q?: string; }; export type ScreenshotOptions = { - bezel?: boolean; - withBezel?: boolean; + bezel?: boolean; + withBezel?: boolean; }; export type ScreenRecordingOptions = { - seconds?: number; + seconds?: number; }; type DeviceMethod = { - (udid: string, ...args: TArgs): TResult; - (...args: TArgs): TResult; + (udid: string, ...args: TArgs): TResult; + (...args: TArgs): TResult; }; export type SimDeckSession = { - endpoint: string; - pid: number; - projectRoot: string; - list(): Promise; - boot: DeviceMethod<[], Promise>; - shutdown: DeviceMethod<[], Promise>; - erase: DeviceMethod<[], Promise>; - install: DeviceMethod<[appPath: string], Promise>; - uninstall: DeviceMethod<[bundleId: string], Promise>; - launch: DeviceMethod<[bundleId: string], Promise>; - openUrl: DeviceMethod<[url: string], Promise>; - tap: DeviceMethod<[x: number, y: number], Promise>; - tapElement: DeviceMethod< - [selector: ElementSelector, options?: TapOptions], - Promise - >; - touch: DeviceMethod<[x: number, y: number, phase: string], Promise>; - swipe: DeviceMethod< - [ - startX: number, - startY: number, - endX: number, - endY: number, - options?: SwipeOptions, - ], - Promise - >; - gesture: DeviceMethod< - [preset: string, options?: GestureOptions], - Promise - >; - typeText: DeviceMethod< - [text: string, options?: TypeTextOptions], - Promise - >; - key: DeviceMethod<[keyCode: number, modifiers?: number], Promise>; - keySequence: DeviceMethod< - [keyCodes: number[], options?: KeySequenceOptions], - Promise - >; - button: DeviceMethod<[button: string, durationMs?: number], Promise>; - home: DeviceMethod<[], Promise>; - dismissKeyboard: DeviceMethod<[], Promise>; - appSwitcher: DeviceMethod<[], Promise>; - rotateLeft: DeviceMethod<[], Promise>; - rotateRight: DeviceMethod<[], Promise>; - toggleAppearance: DeviceMethod<[], Promise>; - pasteboardSet: DeviceMethod<[text: string], Promise>; - pasteboardGet: DeviceMethod<[], Promise>; - chromeProfile: DeviceMethod<[], Promise>; - logs: DeviceMethod<[options?: LogsOptions], Promise>; - tree: DeviceMethod<[options?: QueryOptions], Promise>; - query: DeviceMethod< - [selector: ElementSelector, options?: QueryOptions], - Promise - >; - assert: DeviceMethod< - [selector: ElementSelector, options?: QueryOptions], - Promise - >; - waitFor: DeviceMethod< - [ - selector: ElementSelector, - options?: QueryOptions & { - timeoutMs?: number; - pollMs?: number; - }, - ], - Promise - >; - batch: DeviceMethod< - [steps: unknown[], continueOnError?: boolean], - Promise - >; - screenshot: DeviceMethod<[options?: ScreenshotOptions], Promise>; - record: DeviceMethod<[options?: ScreenRecordingOptions], Promise>; - close(): void; + endpoint: string; + pid: number; + projectRoot: string; + udid?: string; + device(udid: string): SimDeckSession; + list(): Promise; + boot: DeviceMethod<[], Promise>; + shutdown: DeviceMethod<[], Promise>; + erase: DeviceMethod<[], Promise>; + install: DeviceMethod<[appPath: string], Promise>; + uninstall: DeviceMethod<[bundleId: string], Promise>; + launch: DeviceMethod<[bundleId: string], Promise>; + openUrl: DeviceMethod<[url: string], Promise>; + tap: DeviceMethod<[x: number, y: number], Promise>; + tapElement: DeviceMethod<[ + selector: ElementSelector, + options?: TapOptions + ], Promise>; + touch: DeviceMethod<[x: number, y: number, phase: string], Promise>; + swipe: DeviceMethod<[ + startX: number, + startY: number, + endX: number, + endY: number, + options?: SwipeOptions + ], Promise>; + gesture: DeviceMethod<[ + preset: string, + options?: GestureOptions + ], Promise>; + typeText: DeviceMethod<[ + text: string, + options?: TypeTextOptions + ], Promise>; + key: DeviceMethod<[keyCode: number, modifiers?: number], Promise>; + keySequence: DeviceMethod<[ + keyCodes: number[], + options?: KeySequenceOptions + ], Promise>; + button: DeviceMethod<[button: string, durationMs?: number], Promise>; + home: DeviceMethod<[], Promise>; + dismissKeyboard: DeviceMethod<[], Promise>; + appSwitcher: DeviceMethod<[], Promise>; + rotateLeft: DeviceMethod<[], Promise>; + rotateRight: DeviceMethod<[], Promise>; + toggleAppearance: DeviceMethod<[], Promise>; + pasteboardSet: DeviceMethod<[text: string], Promise>; + pasteboardGet: DeviceMethod<[], Promise>; + chromeProfile: DeviceMethod<[], Promise>; + logs: DeviceMethod<[options?: LogsOptions], Promise>; + tree: DeviceMethod<[options?: QueryOptions], Promise>; + query: DeviceMethod<[ + selector: ElementSelector, + options?: QueryOptions + ], Promise>; + assert: DeviceMethod<[ + selector: ElementSelector, + options?: QueryOptions + ], Promise>; + assertNot: DeviceMethod<[ + selector: ElementSelector, + options?: QueryOptions + ], Promise>; + waitFor: DeviceMethod<[ + selector: ElementSelector, + options?: QueryOptions & { + timeoutMs?: number; + pollMs?: number; + } + ], Promise>; + waitForNot: DeviceMethod<[ + selector: ElementSelector, + options?: QueryOptions & { + timeoutMs?: number; + pollMs?: number; + } + ], Promise>; + scrollUntilVisible: DeviceMethod<[ + selector: ElementSelector, + options?: QueryOptions & { + timeoutMs?: number; + pollMs?: number; + direction?: "up" | "down" | "left" | "right"; + durationMs?: number; + steps?: number; + } + ], Promise>; + batch: DeviceMethod<[ + steps: unknown[], + continueOnError?: boolean + ], Promise>; + screenshot: DeviceMethod<[options?: ScreenshotOptions], Promise>; + record: DeviceMethod<[options?: ScreenRecordingOptions], Promise>; + close(): void; }; -export declare function connect( - options?: SimDeckLaunchOptions, -): Promise; +export declare function connect(options?: SimDeckLaunchOptions): Promise; export {}; diff --git a/packages/simdeck-test/dist/index.js b/packages/simdeck-test/dist/index.js index 8963c7f7..8b03c9d6 100644 --- a/packages/simdeck-test/dist/index.js +++ b/packages/simdeck-test/dist/index.js @@ -6,567 +6,554 @@ import net from "node:net"; import os from "node:os"; import path from "node:path"; export async function connect(options = {}) { - const cliPath = options.cliPath ?? "simdeck"; - const result = options.isolated - ? await startIsolatedDaemon(cliPath, options) - : runJson(cliPath, ["daemon", "start"], { + const cliPath = options.cliPath ?? "simdeck"; + const result = options.isolated + ? await startIsolatedDaemon(cliPath, options) + : runJson(cliPath, ["daemon", "start"], { + cwd: options.projectRoot, + }); + const endpoint = result.url; + const createSession = (defaultUdid) => { + const simulatorPath = (udid, suffix) => `/api/simulators/${encodeURIComponent(udid)}${suffix}`; + const requireUdid = (udid) => { + const resolved = udid ?? defaultUdid; + if (!resolved) { + throw new Error("This SimDeck session method requires a UDID. Pass one as the first argument or call connect({ udid })."); + } + return resolved; + }; + const resolveNoArgDeviceCall = (args) => ({ + udid: requireUdid(typeof args[0] === "string" ? args[0] : undefined), + }); + const resolveStringArgDeviceCall = (args) => { + if (args.length >= 2 && + typeof args[0] === "string" && + typeof args[1] === "string") { + return { udid: args[0], value: args[1], rest: args.slice(2) }; + } + return { + udid: requireUdid(), + value: args[0], + rest: args.slice(1), + }; + }; + const resolveObjectArgDeviceCall = (args) => { + if (typeof args[0] === "string") { + return { udid: args[0], value: args[1], rest: args.slice(2) }; + } + return { udid: requireUdid(), value: args[0], rest: args.slice(1) }; + }; + const resolveOptionalObjectDeviceCall = (args) => { + if (typeof args[0] === "string") { + return { udid: args[0], options: args[1] }; + } + return { udid: requireUdid(), options: args[0] }; + }; + const session = { + endpoint, + pid: result.pid, + projectRoot: result.projectRoot, + udid: defaultUdid, + device: (udid) => createSession(udid), + list: () => requestJson(endpoint, "GET", "/api/simulators"), + boot: (...args) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestJson(endpoint, "POST", simulatorPath(udid, "/boot"), null); + }, + shutdown: (...args) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestJson(endpoint, "POST", simulatorPath(udid, "/shutdown"), null); + }, + erase: (...args) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestJson(endpoint, "POST", simulatorPath(udid, "/erase"), null); + }, + install: (...args) => { + const { udid, value: appPath } = resolveStringArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/install"), { + appPath, + }); + }, + uninstall: (...args) => { + const { udid, value: bundleId } = resolveStringArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/uninstall"), { + bundleId, + }); + }, + launch: (...args) => { + const { udid, value: bundleId } = resolveStringArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/launch"), { + bundleId, + }); + }, + openUrl: (...args) => { + const { udid, value: url } = resolveStringArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/open-url"), { + url, + }); + }, + tap: (...args) => { + const [udid, x, y] = typeof args[0] === "string" + ? [args[0], args[1], args[2]] + : [requireUdid(), args[0], args[1]]; + return requestOk(endpoint, simulatorPath(udid, "/tap"), { + x, + y, + normalized: true, + }); + }, + tapElement: (...args) => { + const { udid, value: selector, rest, } = resolveObjectArgDeviceCall(args); + const [tapOptions] = rest; + return requestOk(endpoint, simulatorPath(udid, "/tap"), { + selector: selectorPayload(selector), + ...tapOptions, + }); + }, + touch: (...args) => { + const [udid, x, y, phase] = typeof args[0] === "string" + ? [args[0], args[1], args[2], args[3]] + : [ + requireUdid(), + args[0], + args[1], + args[2], + ]; + return requestOk(endpoint, simulatorPath(udid, "/touch"), { + x, + y, + phase, + }); + }, + swipe: (...args) => { + const [udid, startX, startY, endX, endY, swipeOptions = {}] = typeof args[0] === "string" + ? [ + args[0], + args[1], + args[2], + args[3], + args[4], + args[5], + ] + : [ + requireUdid(), + args[0], + args[1], + args[2], + args[3], + args[4], + ]; + return requestJson(endpoint, "POST", simulatorPath(udid, "/batch"), { + steps: [ + { + action: "swipe", + startX, + startY, + endX, + endY, + ...swipeOptions, + }, + ], + }); + }, + gesture: (...args) => { + const { udid, value: preset, rest } = resolveStringArgDeviceCall(args); + const [gestureOptions = {}] = rest; + return requestJson(endpoint, "POST", simulatorPath(udid, "/batch"), { + steps: [ + { + action: "gesture", + preset, + ...gestureOptions, + }, + ], + }); + }, + typeText: (...args) => { + const { udid, value: text, rest } = resolveStringArgDeviceCall(args); + const [typeOptions = {}] = rest; + return requestJson(endpoint, "POST", simulatorPath(udid, "/batch"), { + steps: [ + { + action: "type", + text, + ...typeOptions, + }, + ], + }); + }, + key: (...args) => { + const [udid, keyCode, modifiers = 0] = typeof args[0] === "string" + ? [args[0], args[1], args[2]] + : [requireUdid(), args[0], args[1]]; + return requestOk(endpoint, simulatorPath(udid, "/key"), { + keyCode, + modifiers, + }); + }, + keySequence: (...args) => { + const { udid, value: keyCodes, rest, } = resolveObjectArgDeviceCall(args); + const [keySequenceOptions = {}] = rest; + return requestOk(endpoint, simulatorPath(udid, "/key-sequence"), { + keyCodes, + ...keySequenceOptions, + }); + }, + button: (...args) => { + const { udid, value: button, rest } = resolveStringArgDeviceCall(args); + const [durationMs = 0] = rest; + return requestOk(endpoint, simulatorPath(udid, "/button"), { + button, + durationMs, + }); + }, + home: (...args) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/home"), null); + }, + dismissKeyboard: (...args) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/dismiss-keyboard"), null); + }, + appSwitcher: (...args) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/app-switcher"), null); + }, + rotateLeft: (...args) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/rotate-left"), null); + }, + rotateRight: (...args) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/rotate-right"), null); + }, + toggleAppearance: (...args) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/toggle-appearance"), null); + }, + pasteboardSet: (...args) => { + const { udid, value: text } = resolveStringArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/pasteboard"), { + text, + }); + }, + pasteboardGet: async (...args) => { + const { udid } = resolveNoArgDeviceCall(args); + const result = await requestJson(endpoint, "GET", simulatorPath(udid, "/pasteboard")); + return result.text ?? ""; + }, + chromeProfile: (...args) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestJson(endpoint, "GET", simulatorPath(udid, "/chrome-profile")); + }, + logs: async (...args) => { + const { udid, value: logsOptions } = typeof args[0] === "string" + ? { udid: args[0], value: args[1] } + : { + udid: requireUdid(), + value: args[0], + }; + const result = await requestJson(endpoint, "GET", simulatorPath(udid, `/logs?${logsQuery(logsOptions)}`)); + return result.entries ?? []; + }, + tree: (...args) => { + const { udid, value: treeOptions } = typeof args[0] === "string" + ? { udid: args[0], value: args[1] } + : { + udid: requireUdid(), + value: args[0], + }; + return requestJson(endpoint, "GET", simulatorPath(udid, `/accessibility-tree?${treeQuery(treeOptions)}`)); + }, + query: async (...args) => { + const { udid, value: selector, rest, } = resolveObjectArgDeviceCall(args); + const [treeOptions] = rest; + const result = await requestJson(endpoint, "POST", simulatorPath(udid, "/query"), { + selector: selectorPayload(selector), + ...treeOptions, + }); + return result.matches; + }, + assert: (...args) => { + const { udid, value: selector, rest, } = resolveObjectArgDeviceCall(args); + const [assertOptions] = rest; + return requestJson(endpoint, "POST", simulatorPath(udid, "/assert"), { + selector: selectorPayload(selector), + ...assertOptions, + }); + }, + assertNot: (...args) => { + const { udid, value: selector, rest, } = resolveObjectArgDeviceCall(args); + const [assertOptions] = rest; + return requestJson(endpoint, "POST", simulatorPath(udid, "/assert-not"), { + selector: selectorPayload(selector), + ...assertOptions, + }); + }, + waitFor: (...args) => { + const { udid, value: selector, rest, } = resolveObjectArgDeviceCall(args); + const [waitOptions] = rest; + return requestJson(endpoint, "POST", simulatorPath(udid, "/wait-for"), { + selector: selectorPayload(selector), + ...waitOptions, + }); + }, + waitForNot: (...args) => { + const { udid, value: selector, rest, } = resolveObjectArgDeviceCall(args); + const [waitOptions] = rest; + return requestJson(endpoint, "POST", simulatorPath(udid, "/wait-for-not"), { + selector: selectorPayload(selector), + ...waitOptions, + }); + }, + scrollUntilVisible: (...args) => { + const { udid, value: selector, rest, } = resolveObjectArgDeviceCall(args); + const [scrollOptions] = rest; + return requestJson(endpoint, "POST", simulatorPath(udid, "/scroll-until-visible"), { + selector: selectorPayload(selector), + ...scrollOptions, + }); + }, + batch: (...args) => { + const { udid, value: steps, rest, } = resolveObjectArgDeviceCall(args); + const [continueOnError = false] = rest; + return requestJson(endpoint, "POST", simulatorPath(udid, "/batch"), { + steps, + continueOnError, + }); + }, + screenshot: (...args) => { + const { udid, options } = resolveOptionalObjectDeviceCall(args); + const params = new URLSearchParams(); + if (options?.withBezel ?? options?.bezel) { + params.set("bezel", "true"); + } + const query = params.toString(); + return requestBuffer(endpoint, simulatorPath(udid, `/screenshot.png${query ? `?${query}` : ""}`)); + }, + record: (...args) => { + const { udid, options } = resolveOptionalObjectDeviceCall(args); + return requestBuffer(endpoint, simulatorPath(udid, "/screen-recording"), "POST", { + seconds: options?.seconds ?? 5, + }); + }, + close: () => { + if (options.keepDaemon) { + return; + } + if (result.child) { + result.child.kill(); + if (result.isolatedRoot) { + fs.rmSync(result.isolatedRoot, { recursive: true, force: true }); + } + return; + } + if (result.started) { + spawnSync(cliPath, ["daemon", "stop"], { cwd: options.projectRoot }); + } + }, + }; + return session; + }; + return createSession(options.udid); +} +async function startIsolatedDaemon(cliPath, options) { + const port = options.port ?? (await freePortPair()); + const projectRoot = fs.mkdtempSync(path.join(os.tmpdir(), "simdeck-test-project-")); + const metadataPath = path.join(os.tmpdir(), `simdeck-test-${process.pid}-${Date.now()}-${crypto.randomUUID()}.json`); + const accessToken = crypto.randomBytes(32).toString("hex"); + const child = spawn(cliPath, [ + "daemon", + "run", + "--project-root", + projectRoot, + "--metadata-path", + metadataPath, + "--port", + String(port), + "--bind", + "127.0.0.1", + "--access-token", + accessToken, + "--video-codec", + options.videoCodec ?? "software", + ], { cwd: options.projectRoot, - }); - const endpoint = result.url; - const defaultUdid = options.udid; - const simulatorPath = (udid, suffix) => - `/api/simulators/${encodeURIComponent(udid)}${suffix}`; - const requireUdid = (udid) => { - const resolved = udid ?? defaultUdid; - if (!resolved) { - throw new Error( - "This SimDeck session method requires a UDID. Pass one as the first argument or call connect({ udid }).", - ); + stdio: ["ignore", "pipe", "pipe"], + }); + const output = captureChildOutput(child); + const url = `http://127.0.0.1:${port}`; + try { + await waitForHealth(url, child, output); } - return resolved; - }; - const resolveNoArgDeviceCall = (args) => ({ - udid: requireUdid(typeof args[0] === "string" ? args[0] : undefined), - }); - const resolveStringArgDeviceCall = (args) => { - if ( - args.length >= 2 && - typeof args[0] === "string" && - typeof args[1] === "string" - ) { - return { udid: args[0], value: args[1], rest: args.slice(2) }; + catch (error) { + child.kill(); + fs.rmSync(projectRoot, { recursive: true, force: true }); + throw error; } return { - udid: requireUdid(), - value: args[0], - rest: args.slice(1), - }; - }; - const resolveObjectArgDeviceCall = (args) => { - if (typeof args[0] === "string") { - return { udid: args[0], value: args[1], rest: args.slice(2) }; - } - return { udid: requireUdid(), value: args[0], rest: args.slice(1) }; - }; - const resolveOptionalObjectDeviceCall = (args) => { - if (typeof args[0] === "string") { - return { udid: args[0], options: args[1] }; - } - return { udid: requireUdid(), options: args[0] }; - }; - const session = { - endpoint, - pid: result.pid, - projectRoot: result.projectRoot, - list: () => requestJson(endpoint, "GET", "/api/simulators"), - boot: (...args) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestJson(endpoint, "POST", simulatorPath(udid, "/boot"), null); - }, - shutdown: (...args) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestJson( - endpoint, - "POST", - simulatorPath(udid, "/shutdown"), - null, - ); - }, - erase: (...args) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestJson(endpoint, "POST", simulatorPath(udid, "/erase"), null); - }, - install: (...args) => { - const { udid, value: appPath } = resolveStringArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/install"), { - appPath, - }); - }, - uninstall: (...args) => { - const { udid, value: bundleId } = resolveStringArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/uninstall"), { - bundleId, - }); - }, - launch: (...args) => { - const { udid, value: bundleId } = resolveStringArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/launch"), { - bundleId, - }); - }, - openUrl: (...args) => { - const { udid, value: url } = resolveStringArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/open-url"), { + ok: true, + projectRoot, + pid: child.pid ?? 0, url, - }); - }, - tap: (...args) => { - const [udid, x, y] = - typeof args[0] === "string" - ? [args[0], args[1], args[2]] - : [requireUdid(), args[0], args[1]]; - return requestOk(endpoint, simulatorPath(udid, "/tap"), { - x, - y, - normalized: true, - }); - }, - tapElement: (...args) => { - const { udid, value: selector, rest } = resolveObjectArgDeviceCall(args); - const [tapOptions] = rest; - return requestOk(endpoint, simulatorPath(udid, "/tap"), { - selector: selectorPayload(selector), - ...tapOptions, - }); - }, - touch: (...args) => { - const [udid, x, y, phase] = - typeof args[0] === "string" - ? [args[0], args[1], args[2], args[3]] - : [requireUdid(), args[0], args[1], args[2]]; - return requestOk(endpoint, simulatorPath(udid, "/touch"), { - x, - y, - phase, - }); - }, - swipe: (...args) => { - const [udid, startX, startY, endX, endY, swipeOptions = {}] = - typeof args[0] === "string" - ? [args[0], args[1], args[2], args[3], args[4], args[5]] - : [requireUdid(), args[0], args[1], args[2], args[3], args[4]]; - return requestJson(endpoint, "POST", simulatorPath(udid, "/batch"), { - steps: [ - { - action: "swipe", - startX, - startY, - endX, - endY, - ...swipeOptions, - }, - ], - }); - }, - gesture: (...args) => { - const { udid, value: preset, rest } = resolveStringArgDeviceCall(args); - const [gestureOptions = {}] = rest; - return requestJson(endpoint, "POST", simulatorPath(udid, "/batch"), { - steps: [ - { - action: "gesture", - preset, - ...gestureOptions, - }, - ], - }); - }, - typeText: (...args) => { - const { udid, value: text, rest } = resolveStringArgDeviceCall(args); - const [typeOptions = {}] = rest; - return requestJson(endpoint, "POST", simulatorPath(udid, "/batch"), { - steps: [ - { - action: "type", - text, - ...typeOptions, - }, - ], - }); - }, - key: (...args) => { - const [udid, keyCode, modifiers = 0] = - typeof args[0] === "string" - ? [args[0], args[1], args[2]] - : [requireUdid(), args[0], args[1]]; - return requestOk(endpoint, simulatorPath(udid, "/key"), { - keyCode, - modifiers, - }); - }, - keySequence: (...args) => { - const { udid, value: keyCodes, rest } = resolveObjectArgDeviceCall(args); - const [keySequenceOptions = {}] = rest; - return requestOk(endpoint, simulatorPath(udid, "/key-sequence"), { - keyCodes, - ...keySequenceOptions, - }); - }, - button: (...args) => { - const { udid, value: button, rest } = resolveStringArgDeviceCall(args); - const [durationMs = 0] = rest; - return requestOk(endpoint, simulatorPath(udid, "/button"), { - button, - durationMs, - }); - }, - home: (...args) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/home"), null); - }, - dismissKeyboard: (...args) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestOk( - endpoint, - simulatorPath(udid, "/dismiss-keyboard"), - null, - ); - }, - appSwitcher: (...args) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/app-switcher"), null); - }, - rotateLeft: (...args) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/rotate-left"), null); - }, - rotateRight: (...args) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/rotate-right"), null); - }, - toggleAppearance: (...args) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestOk( - endpoint, - simulatorPath(udid, "/toggle-appearance"), - null, - ); - }, - pasteboardSet: (...args) => { - const { udid, value: text } = resolveStringArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/pasteboard"), { - text, - }); - }, - pasteboardGet: async (...args) => { - const { udid } = resolveNoArgDeviceCall(args); - const result = await requestJson( - endpoint, - "GET", - simulatorPath(udid, "/pasteboard"), - ); - return result.text ?? ""; - }, - chromeProfile: (...args) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestJson( - endpoint, - "GET", - simulatorPath(udid, "/chrome-profile"), - ); - }, - logs: async (...args) => { - const { udid, value: logsOptions } = - typeof args[0] === "string" - ? { udid: args[0], value: args[1] } - : { udid: requireUdid(), value: args[0] }; - const result = await requestJson( - endpoint, - "GET", - simulatorPath(udid, `/logs?${logsQuery(logsOptions)}`), - ); - return result.entries ?? []; - }, - tree: (...args) => { - const { udid, value: treeOptions } = - typeof args[0] === "string" - ? { udid: args[0], value: args[1] } - : { udid: requireUdid(), value: args[0] }; - return requestJson( - endpoint, - "GET", - simulatorPath(udid, `/accessibility-tree?${treeQuery(treeOptions)}`), - ); - }, - query: async (...args) => { - const { udid, value: selector, rest } = resolveObjectArgDeviceCall(args); - const [treeOptions] = rest; - const result = await requestJson( - endpoint, - "POST", - simulatorPath(udid, "/query"), - { - selector: selectorPayload(selector), - ...treeOptions, - }, - ); - return result.matches; - }, - assert: (...args) => { - const { udid, value: selector, rest } = resolveObjectArgDeviceCall(args); - const [assertOptions] = rest; - return requestJson(endpoint, "POST", simulatorPath(udid, "/assert"), { - selector: selectorPayload(selector), - ...assertOptions, - }); - }, - waitFor: (...args) => { - const { udid, value: selector, rest } = resolveObjectArgDeviceCall(args); - const [waitOptions] = rest; - return requestJson(endpoint, "POST", simulatorPath(udid, "/wait-for"), { - selector: selectorPayload(selector), - ...waitOptions, - }); - }, - batch: (...args) => { - const { udid, value: steps, rest } = resolveObjectArgDeviceCall(args); - const [continueOnError = false] = rest; - return requestJson(endpoint, "POST", simulatorPath(udid, "/batch"), { - steps, - continueOnError, - }); - }, - screenshot: (...args) => { - const { udid, options } = resolveOptionalObjectDeviceCall(args); - const params = new URLSearchParams(); - if (options?.withBezel ?? options?.bezel) { - params.set("bezel", "true"); - } - const query = params.toString(); - return requestBuffer( - endpoint, - simulatorPath(udid, `/screenshot.png${query ? `?${query}` : ""}`), - ); - }, - record: (...args) => { - const { udid, options } = resolveOptionalObjectDeviceCall(args); - return requestBuffer( - endpoint, - simulatorPath(udid, "/screen-recording"), - "POST", - { - seconds: options?.seconds ?? 5, - }, - ); - }, - close: () => { - if (options.keepDaemon) { - return; - } - if (result.child) { - result.child.kill(); - if (result.isolatedRoot) { - fs.rmSync(result.isolatedRoot, { recursive: true, force: true }); - } - return; - } - if (result.started) { - spawnSync(cliPath, ["daemon", "stop"], { cwd: options.projectRoot }); - } - }, - }; - return session; -} -async function startIsolatedDaemon(cliPath, options) { - const port = options.port ?? (await freePortPair()); - const projectRoot = fs.mkdtempSync( - path.join(os.tmpdir(), "simdeck-test-project-"), - ); - const metadataPath = path.join( - os.tmpdir(), - `simdeck-test-${process.pid}-${Date.now()}-${crypto.randomUUID()}.json`, - ); - const accessToken = crypto.randomBytes(32).toString("hex"); - const child = spawn( - cliPath, - [ - "daemon", - "run", - "--project-root", - projectRoot, - "--metadata-path", - metadataPath, - "--port", - String(port), - "--bind", - "127.0.0.1", - "--access-token", - accessToken, - "--video-codec", - options.videoCodec ?? "software", - ], - { - cwd: options.projectRoot, - stdio: ["ignore", "pipe", "pipe"], - }, - ); - const output = captureChildOutput(child); - const url = `http://127.0.0.1:${port}`; - try { - await waitForHealth(url, child, output); - } catch (error) { - child.kill(); - fs.rmSync(projectRoot, { recursive: true, force: true }); - throw error; - } - return { - ok: true, - projectRoot, - pid: child.pid ?? 0, - url, - started: true, - child, - isolatedRoot: projectRoot, - }; + started: true, + child, + isolatedRoot: projectRoot, + }; } async function waitForHealth(endpoint, child, output) { - const deadline = Date.now() + 60_000; - let lastError; - while (Date.now() < deadline) { - if (child.exitCode !== null) { - throw new Error( - `SimDeck isolated daemon exited with ${child.exitCode}.\n${output()}`, - ); - } - try { - await requestJson(endpoint, "GET", "/api/health"); - return; - } catch (error) { - lastError = error; - await new Promise((resolve) => setTimeout(resolve, 50)); + const deadline = Date.now() + 60_000; + let lastError; + while (Date.now() < deadline) { + if (child.exitCode !== null) { + throw new Error(`SimDeck isolated daemon exited with ${child.exitCode}.\n${output()}`); + } + try { + await requestJson(endpoint, "GET", "/api/health"); + return; + } + catch (error) { + lastError = error; + await new Promise((resolve) => setTimeout(resolve, 50)); + } } - } - throw new Error( - `Timed out waiting for isolated SimDeck daemon: ${lastError instanceof Error ? lastError.message : String(lastError)}\n${output()}`, - ); + throw new Error(`Timed out waiting for isolated SimDeck daemon: ${lastError instanceof Error ? lastError.message : String(lastError)}\n${output()}`); } function captureChildOutput(child) { - const chunks = []; - const append = (source, chunk) => { - chunks.push(`[${source}] ${chunk.toString("utf8")}`); - while (chunks.join("").length > 16_384) { - chunks.shift(); - } - }; - child.stdout?.on("data", (chunk) => append("stdout", chunk)); - child.stderr?.on("data", (chunk) => append("stderr", chunk)); - return () => chunks.join("").trim(); + const chunks = []; + const append = (source, chunk) => { + chunks.push(`[${source}] ${chunk.toString("utf8")}`); + while (chunks.join("").length > 16_384) { + chunks.shift(); + } + }; + child.stdout?.on("data", (chunk) => append("stdout", chunk)); + child.stderr?.on("data", (chunk) => append("stderr", chunk)); + return () => chunks.join("").trim(); } async function freePortPair() { - for (let attempt = 0; attempt < 100; attempt += 1) { - const port = await freePort(); - if (port < 65535 && (await portAvailable(port + 1))) { - return port; + for (let attempt = 0; attempt < 100; attempt += 1) { + const port = await freePort(); + if (port < 65535 && (await portAvailable(port + 1))) { + return port; + } } - } - throw new Error("Unable to allocate adjacent free TCP ports."); + throw new Error("Unable to allocate adjacent free TCP ports."); } function freePort() { - return new Promise((resolve, reject) => { - const server = net.createServer(); - server.listen(0, "127.0.0.1", () => { - const address = server.address(); - if (!address || typeof address === "string") { - server.close(); - reject(new Error("Unable to allocate a free TCP port.")); - return; - } - const port = address.port; - server.close(() => resolve(port)); + return new Promise((resolve, reject) => { + const server = net.createServer(); + server.listen(0, "127.0.0.1", () => { + const address = server.address(); + if (!address || typeof address === "string") { + server.close(); + reject(new Error("Unable to allocate a free TCP port.")); + return; + } + const port = address.port; + server.close(() => resolve(port)); + }); + server.on("error", reject); }); - server.on("error", reject); - }); } function portAvailable(port) { - return new Promise((resolve) => { - const server = net.createServer(); - server.once("error", () => resolve(false)); - server.listen(port, "127.0.0.1", () => { - server.close(() => resolve(true)); + return new Promise((resolve) => { + const server = net.createServer(); + server.once("error", () => resolve(false)); + server.listen(port, "127.0.0.1", () => { + server.close(() => resolve(true)); + }); }); - }); } function runJson(command, args, options = {}) { - const result = spawnSync(command, args, { - cwd: options.cwd, - encoding: "utf8", - maxBuffer: 1024 * 1024, - }); - if (result.status !== 0) { - throw new Error( - result.stderr.trim() || `${command} ${args.join(" ")} failed`, - ); - } - return JSON.parse(result.stdout); + const result = spawnSync(command, args, { + cwd: options.cwd, + encoding: "utf8", + maxBuffer: 1024 * 1024, + }); + if (result.status !== 0) { + throw new Error(result.stderr.trim() || `${command} ${args.join(" ")} failed`); + } + return JSON.parse(result.stdout); } function requestOk(endpoint, pathName, body) { - return requestJson(endpoint, "POST", pathName, body).then(() => undefined); + return requestJson(endpoint, "POST", pathName, body).then(() => undefined); } function requestJson(endpoint, method, pathName, body) { - return requestBuffer(endpoint, pathName, method, body).then((buffer) => - JSON.parse(buffer.toString("utf8")), - ); + return requestBuffer(endpoint, pathName, method, body).then((buffer) => JSON.parse(buffer.toString("utf8"))); } function requestBuffer(endpoint, pathName, method = "GET", body) { - const url = new URL(pathName, endpoint); - const payload = - body === undefined ? undefined : Buffer.from(JSON.stringify(body)); - return new Promise((resolve, reject) => { - const request = http.request( - url, - { - method, - headers: payload - ? { - "content-type": "application/json", - "content-length": String(payload.length), - origin: endpoint, - } - : { origin: endpoint }, - }, - (response) => { - const chunks = []; - response.on("data", (chunk) => chunks.push(chunk)); - response.on("end", () => { - const buffer = Buffer.concat(chunks); - if ( - (response.statusCode ?? 500) < 200 || - (response.statusCode ?? 500) >= 300 - ) { - reject( - new Error( - `${method} ${pathName} returned ${response.statusCode}: ${buffer.toString("utf8") || response.statusMessage || ""}`, - ), - ); - } else { - resolve(buffer); - } + const url = new URL(pathName, endpoint); + const payload = body === undefined ? undefined : Buffer.from(JSON.stringify(body)); + return new Promise((resolve, reject) => { + const request = http.request(url, { + method, + headers: payload + ? { + "content-type": "application/json", + "content-length": String(payload.length), + origin: endpoint, + } + : { origin: endpoint }, + }, (response) => { + const chunks = []; + response.on("data", (chunk) => chunks.push(chunk)); + response.on("end", () => { + const buffer = Buffer.concat(chunks); + if ((response.statusCode ?? 500) < 200 || + (response.statusCode ?? 500) >= 300) { + reject(new Error(`${method} ${pathName} returned ${response.statusCode}: ${buffer.toString("utf8") || response.statusMessage || ""}`)); + } + else { + resolve(buffer); + } + }); }); - }, - ); - request.on("error", reject); - if (payload) { - request.write(payload); - } - request.end(); - }); + request.on("error", reject); + if (payload) { + request.write(payload); + } + request.end(); + }); } function treeQuery(options = {}) { - const params = new URLSearchParams(); - if (options.source) params.set("source", options.source); - if (options.maxDepth !== undefined) - params.set("maxDepth", String(options.maxDepth)); - if (options.includeHidden) params.set("includeHidden", "true"); - return params.toString(); + const params = new URLSearchParams(); + if (options.source) + params.set("source", options.source); + if (options.maxDepth !== undefined) + params.set("maxDepth", String(options.maxDepth)); + if (options.includeHidden) + params.set("includeHidden", "true"); + return params.toString(); } function logsQuery(options = {}) { - const params = new URLSearchParams(); - if (options.backfill !== undefined) - params.set("backfill", String(options.backfill)); - if (options.seconds !== undefined) - params.set("seconds", String(options.seconds)); - if (options.limit !== undefined) params.set("limit", String(options.limit)); - if (options.levels?.length) params.set("levels", options.levels.join(",")); - if (options.processes?.length) - params.set("processes", options.processes.join(",")); - if (options.q) params.set("q", options.q); - return params.toString(); + const params = new URLSearchParams(); + if (options.backfill !== undefined) + params.set("backfill", String(options.backfill)); + if (options.seconds !== undefined) + params.set("seconds", String(options.seconds)); + if (options.limit !== undefined) + params.set("limit", String(options.limit)); + if (options.levels?.length) + params.set("levels", options.levels.join(",")); + if (options.processes?.length) + params.set("processes", options.processes.join(",")); + if (options.q) + params.set("q", options.q); + return params.toString(); } function selectorPayload(selector) { - return { - id: selector.id, - label: selector.label, - value: selector.value, - elementType: selector.type, - }; + return { + text: selector.text, + id: selector.id, + label: selector.label, + value: selector.value, + elementType: selector.type, + index: selector.index, + enabled: selector.enabled, + checked: selector.checked, + focused: selector.focused, + selected: selector.selected, + regex: selector.regex, + }; } diff --git a/packages/simdeck-test/src/index.ts b/packages/simdeck-test/src/index.ts index 71d22c57..f6c90acd 100644 --- a/packages/simdeck-test/src/index.ts +++ b/packages/simdeck-test/src/index.ts @@ -22,6 +22,7 @@ export type QueryOptions = { | "nativescript" | "react-native" | "flutter" + | "swiftui" | "uikit" | "native-ax" | "android-uiautomator"; @@ -30,10 +31,17 @@ export type QueryOptions = { }; export type ElementSelector = { + text?: string; id?: string; label?: string; value?: string; type?: string; + index?: number; + enabled?: boolean; + checked?: boolean; + focused?: boolean; + selected?: boolean; + regex?: boolean; }; export type TapOptions = QueryOptions & { @@ -86,6 +94,8 @@ export type SimDeckSession = { endpoint: string; pid: number; projectRoot: string; + udid?: string; + device(udid: string): SimDeckSession; list(): Promise; boot: DeviceMethod<[], Promise>; shutdown: DeviceMethod<[], Promise>; @@ -143,6 +153,10 @@ export type SimDeckSession = { [selector: ElementSelector, options?: QueryOptions], Promise >; + assertNot: DeviceMethod< + [selector: ElementSelector, options?: QueryOptions], + Promise + >; waitFor: DeviceMethod< [ selector: ElementSelector, @@ -150,6 +164,26 @@ export type SimDeckSession = { ], Promise >; + waitForNot: DeviceMethod< + [ + selector: ElementSelector, + options?: QueryOptions & { timeoutMs?: number; pollMs?: number }, + ], + Promise + >; + scrollUntilVisible: DeviceMethod< + [ + selector: ElementSelector, + options?: QueryOptions & { + timeoutMs?: number; + pollMs?: number; + direction?: "up" | "down" | "left" | "right"; + durationMs?: number; + steps?: number; + }, + ], + Promise + >; batch: DeviceMethod< [steps: unknown[], continueOnError?: boolean], Promise @@ -182,400 +216,482 @@ export async function connect( cwd: options.projectRoot, }); const endpoint = result.url; - const defaultUdid = options.udid; - const simulatorPath = (udid: string, suffix: string) => - `/api/simulators/${encodeURIComponent(udid)}${suffix}`; - const requireUdid = (udid?: string) => { - const resolved = udid ?? defaultUdid; - if (!resolved) { - throw new Error( - "This SimDeck session method requires a UDID. Pass one as the first argument or call connect({ udid }).", - ); - } - return resolved; - }; - const resolveNoArgDeviceCall = (args: unknown[]) => ({ - udid: requireUdid(typeof args[0] === "string" ? args[0] : undefined), - }); - const resolveStringArgDeviceCall = (args: unknown[]) => { - if ( - args.length >= 2 && - typeof args[0] === "string" && - typeof args[1] === "string" - ) { - return { udid: args[0], value: args[1] as string, rest: args.slice(2) }; - } - return { - udid: requireUdid(), - value: args[0] as string, - rest: args.slice(1), + const createSession = (defaultUdid?: string): SimDeckSession => { + const simulatorPath = (udid: string, suffix: string) => + `/api/simulators/${encodeURIComponent(udid)}${suffix}`; + const requireUdid = (udid?: string) => { + const resolved = udid ?? defaultUdid; + if (!resolved) { + throw new Error( + "This SimDeck session method requires a UDID. Pass one as the first argument or call connect({ udid }).", + ); + } + return resolved; }; - }; - const resolveObjectArgDeviceCall = (args: unknown[]) => { - if (typeof args[0] === "string") { - return { udid: args[0], value: args[1] as T, rest: args.slice(2) }; - } - return { udid: requireUdid(), value: args[0] as T, rest: args.slice(1) }; - }; - const resolveOptionalObjectDeviceCall = (args: unknown[]) => { - if (typeof args[0] === "string") { - return { udid: args[0], options: args[1] as T | undefined }; - } - return { udid: requireUdid(), options: args[0] as T | undefined }; - }; - const session: SimDeckSession = { - endpoint, - pid: result.pid, - projectRoot: result.projectRoot, - list: () => requestJson(endpoint, "GET", "/api/simulators"), - boot: (...args: [] | [string]) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestJson(endpoint, "POST", simulatorPath(udid, "/boot"), null); - }, - shutdown: (...args: [] | [string]) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestJson( - endpoint, - "POST", - simulatorPath(udid, "/shutdown"), - null, - ); - }, - erase: (...args: [] | [string]) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestJson(endpoint, "POST", simulatorPath(udid, "/erase"), null); - }, - install: (...args) => { - const { udid, value: appPath } = resolveStringArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/install"), { - appPath, - }); - }, - uninstall: (...args) => { - const { udid, value: bundleId } = resolveStringArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/uninstall"), { - bundleId, - }); - }, - launch: (...args) => { - const { udid, value: bundleId } = resolveStringArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/launch"), { - bundleId, - }); - }, - openUrl: (...args) => { - const { udid, value: url } = resolveStringArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/open-url"), { - url, - }); - }, - tap: (...args) => { - const [udid, x, y] = - typeof args[0] === "string" - ? [args[0], args[1] as number, args[2] as number] - : [requireUdid(), args[0] as number, args[1] as number]; - return requestOk(endpoint, simulatorPath(udid, "/tap"), { - x, - y, - normalized: true, - }); - }, - tapElement: (...args) => { - const { - udid, - value: selector, - rest, - } = resolveObjectArgDeviceCall(args); - const [tapOptions] = rest as [TapOptions?]; - return requestOk(endpoint, simulatorPath(udid, "/tap"), { - selector: selectorPayload(selector), - ...tapOptions, - }); - }, - touch: (...args) => { - const [udid, x, y, phase] = - typeof args[0] === "string" - ? [args[0], args[1] as number, args[2] as number, args[3] as string] - : [ - requireUdid(), - args[0] as number, - args[1] as number, - args[2] as string, - ]; - return requestOk(endpoint, simulatorPath(udid, "/touch"), { - x, - y, - phase, - }); - }, - swipe: (...args) => { - const [udid, startX, startY, endX, endY, swipeOptions = {}] = - typeof args[0] === "string" - ? [ - args[0], - args[1] as number, - args[2] as number, - args[3] as number, - args[4] as number, - args[5] as SwipeOptions | undefined, - ] - : [ - requireUdid(), - args[0] as number, - args[1] as number, - args[2] as number, - args[3] as number, - args[4] as SwipeOptions | undefined, - ]; - return requestJson(endpoint, "POST", simulatorPath(udid, "/batch"), { - steps: [ + const resolveNoArgDeviceCall = (args: unknown[]) => ({ + udid: requireUdid(typeof args[0] === "string" ? args[0] : undefined), + }); + const resolveStringArgDeviceCall = (args: unknown[]) => { + if ( + args.length >= 2 && + typeof args[0] === "string" && + typeof args[1] === "string" + ) { + return { udid: args[0], value: args[1] as string, rest: args.slice(2) }; + } + return { + udid: requireUdid(), + value: args[0] as string, + rest: args.slice(1), + }; + }; + const resolveObjectArgDeviceCall = (args: unknown[]) => { + if (typeof args[0] === "string") { + return { udid: args[0], value: args[1] as T, rest: args.slice(2) }; + } + return { udid: requireUdid(), value: args[0] as T, rest: args.slice(1) }; + }; + const resolveOptionalObjectDeviceCall = (args: unknown[]) => { + if (typeof args[0] === "string") { + return { udid: args[0], options: args[1] as T | undefined }; + } + return { udid: requireUdid(), options: args[0] as T | undefined }; + }; + const session: SimDeckSession = { + endpoint, + pid: result.pid, + projectRoot: result.projectRoot, + udid: defaultUdid, + device: (udid: string) => createSession(udid), + list: () => requestJson(endpoint, "GET", "/api/simulators"), + boot: (...args: [] | [string]) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestJson( + endpoint, + "POST", + simulatorPath(udid, "/boot"), + null, + ); + }, + shutdown: (...args: [] | [string]) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestJson( + endpoint, + "POST", + simulatorPath(udid, "/shutdown"), + null, + ); + }, + erase: (...args: [] | [string]) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestJson( + endpoint, + "POST", + simulatorPath(udid, "/erase"), + null, + ); + }, + install: (...args) => { + const { udid, value: appPath } = resolveStringArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/install"), { + appPath, + }); + }, + uninstall: (...args) => { + const { udid, value: bundleId } = resolveStringArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/uninstall"), { + bundleId, + }); + }, + launch: (...args) => { + const { udid, value: bundleId } = resolveStringArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/launch"), { + bundleId, + }); + }, + openUrl: (...args) => { + const { udid, value: url } = resolveStringArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/open-url"), { + url, + }); + }, + tap: (...args) => { + const [udid, x, y] = + typeof args[0] === "string" + ? [args[0], args[1] as number, args[2] as number] + : [requireUdid(), args[0] as number, args[1] as number]; + return requestOk(endpoint, simulatorPath(udid, "/tap"), { + x, + y, + normalized: true, + }); + }, + tapElement: (...args) => { + const { + udid, + value: selector, + rest, + } = resolveObjectArgDeviceCall(args); + const [tapOptions] = rest as [TapOptions?]; + return requestOk(endpoint, simulatorPath(udid, "/tap"), { + selector: selectorPayload(selector), + ...tapOptions, + }); + }, + touch: (...args) => { + const [udid, x, y, phase] = + typeof args[0] === "string" + ? [args[0], args[1] as number, args[2] as number, args[3] as string] + : [ + requireUdid(), + args[0] as number, + args[1] as number, + args[2] as string, + ]; + return requestOk(endpoint, simulatorPath(udid, "/touch"), { + x, + y, + phase, + }); + }, + swipe: (...args) => { + const [udid, startX, startY, endX, endY, swipeOptions = {}] = + typeof args[0] === "string" + ? [ + args[0], + args[1] as number, + args[2] as number, + args[3] as number, + args[4] as number, + args[5] as SwipeOptions | undefined, + ] + : [ + requireUdid(), + args[0] as number, + args[1] as number, + args[2] as number, + args[3] as number, + args[4] as SwipeOptions | undefined, + ]; + return requestJson(endpoint, "POST", simulatorPath(udid, "/batch"), { + steps: [ + { + action: "swipe", + startX, + startY, + endX, + endY, + ...swipeOptions, + }, + ], + }); + }, + gesture: (...args) => { + const { udid, value: preset, rest } = resolveStringArgDeviceCall(args); + const [gestureOptions = {}] = rest as [GestureOptions?]; + return requestJson(endpoint, "POST", simulatorPath(udid, "/batch"), { + steps: [ + { + action: "gesture", + preset, + ...gestureOptions, + }, + ], + }); + }, + typeText: (...args) => { + const { udid, value: text, rest } = resolveStringArgDeviceCall(args); + const [typeOptions = {}] = rest as [TypeTextOptions?]; + return requestJson(endpoint, "POST", simulatorPath(udid, "/batch"), { + steps: [ + { + action: "type", + text, + ...typeOptions, + }, + ], + }); + }, + key: (...args) => { + const [udid, keyCode, modifiers = 0] = + typeof args[0] === "string" + ? [args[0], args[1] as number, args[2] as number | undefined] + : [requireUdid(), args[0] as number, args[1] as number | undefined]; + return requestOk(endpoint, simulatorPath(udid, "/key"), { + keyCode, + modifiers, + }); + }, + keySequence: (...args) => { + const { + udid, + value: keyCodes, + rest, + } = resolveObjectArgDeviceCall(args); + const [keySequenceOptions = {}] = rest as [KeySequenceOptions?]; + return requestOk(endpoint, simulatorPath(udid, "/key-sequence"), { + keyCodes, + ...keySequenceOptions, + }); + }, + button: (...args) => { + const { udid, value: button, rest } = resolveStringArgDeviceCall(args); + const [durationMs = 0] = rest as [number?]; + return requestOk(endpoint, simulatorPath(udid, "/button"), { + button, + durationMs, + }); + }, + home: (...args: [] | [string]) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/home"), null); + }, + dismissKeyboard: (...args: [] | [string]) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestOk( + endpoint, + simulatorPath(udid, "/dismiss-keyboard"), + null, + ); + }, + appSwitcher: (...args: [] | [string]) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/app-switcher"), null); + }, + rotateLeft: (...args: [] | [string]) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/rotate-left"), null); + }, + rotateRight: (...args: [] | [string]) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/rotate-right"), null); + }, + toggleAppearance: (...args: [] | [string]) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestOk( + endpoint, + simulatorPath(udid, "/toggle-appearance"), + null, + ); + }, + pasteboardSet: (...args) => { + const { udid, value: text } = resolveStringArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/pasteboard"), { + text, + }); + }, + pasteboardGet: async (...args: [] | [string]) => { + const { udid } = resolveNoArgDeviceCall(args); + const result = await requestJson<{ text?: string }>( + endpoint, + "GET", + simulatorPath(udid, "/pasteboard"), + ); + return result.text ?? ""; + }, + chromeProfile: (...args: [] | [string]) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestJson( + endpoint, + "GET", + simulatorPath(udid, "/chrome-profile"), + ); + }, + logs: async (...args) => { + const { udid, value: logsOptions } = + typeof args[0] === "string" + ? { udid: args[0], value: args[1] as LogsOptions | undefined } + : { + udid: requireUdid(), + value: args[0] as LogsOptions | undefined, + }; + const result = await requestJson<{ entries?: unknown[] }>( + endpoint, + "GET", + simulatorPath(udid, `/logs?${logsQuery(logsOptions)}`), + ); + return result.entries ?? []; + }, + tree: (...args) => { + const { udid, value: treeOptions } = + typeof args[0] === "string" + ? { udid: args[0], value: args[1] as QueryOptions | undefined } + : { + udid: requireUdid(), + value: args[0] as QueryOptions | undefined, + }; + return requestJson( + endpoint, + "GET", + simulatorPath(udid, `/accessibility-tree?${treeQuery(treeOptions)}`), + ); + }, + query: async (...args) => { + const { + udid, + value: selector, + rest, + } = resolveObjectArgDeviceCall(args); + const [treeOptions] = rest as [QueryOptions?]; + const result = await requestJson<{ matches: unknown[] }>( + endpoint, + "POST", + simulatorPath(udid, "/query"), { - action: "swipe", - startX, - startY, - endX, - endY, - ...swipeOptions, + selector: selectorPayload(selector), + ...treeOptions, }, - ], - }); - }, - gesture: (...args) => { - const { udid, value: preset, rest } = resolveStringArgDeviceCall(args); - const [gestureOptions = {}] = rest as [GestureOptions?]; - return requestJson(endpoint, "POST", simulatorPath(udid, "/batch"), { - steps: [ + ); + return result.matches; + }, + assert: (...args) => { + const { + udid, + value: selector, + rest, + } = resolveObjectArgDeviceCall(args); + const [assertOptions] = rest as [QueryOptions?]; + return requestJson(endpoint, "POST", simulatorPath(udid, "/assert"), { + selector: selectorPayload(selector), + ...assertOptions, + }); + }, + assertNot: (...args) => { + const { + udid, + value: selector, + rest, + } = resolveObjectArgDeviceCall(args); + const [assertOptions] = rest as [QueryOptions?]; + return requestJson( + endpoint, + "POST", + simulatorPath(udid, "/assert-not"), { - action: "gesture", - preset, - ...gestureOptions, + selector: selectorPayload(selector), + ...assertOptions, }, - ], - }); - }, - typeText: (...args) => { - const { udid, value: text, rest } = resolveStringArgDeviceCall(args); - const [typeOptions = {}] = rest as [TypeTextOptions?]; - return requestJson(endpoint, "POST", simulatorPath(udid, "/batch"), { - steps: [ + ); + }, + waitFor: (...args) => { + const { + udid, + value: selector, + rest, + } = resolveObjectArgDeviceCall(args); + const [waitOptions] = rest as [ + (QueryOptions & { timeoutMs?: number; pollMs?: number })?, + ]; + return requestJson(endpoint, "POST", simulatorPath(udid, "/wait-for"), { + selector: selectorPayload(selector), + ...waitOptions, + }); + }, + waitForNot: (...args) => { + const { + udid, + value: selector, + rest, + } = resolveObjectArgDeviceCall(args); + const [waitOptions] = rest as [ + (QueryOptions & { timeoutMs?: number; pollMs?: number })?, + ]; + return requestJson( + endpoint, + "POST", + simulatorPath(udid, "/wait-for-not"), { - action: "type", - text, - ...typeOptions, + selector: selectorPayload(selector), + ...waitOptions, }, - ], - }); - }, - key: (...args) => { - const [udid, keyCode, modifiers = 0] = - typeof args[0] === "string" - ? [args[0], args[1] as number, args[2] as number | undefined] - : [requireUdid(), args[0] as number, args[1] as number | undefined]; - return requestOk(endpoint, simulatorPath(udid, "/key"), { - keyCode, - modifiers, - }); - }, - keySequence: (...args) => { - const { - udid, - value: keyCodes, - rest, - } = resolveObjectArgDeviceCall(args); - const [keySequenceOptions = {}] = rest as [KeySequenceOptions?]; - return requestOk(endpoint, simulatorPath(udid, "/key-sequence"), { - keyCodes, - ...keySequenceOptions, - }); - }, - button: (...args) => { - const { udid, value: button, rest } = resolveStringArgDeviceCall(args); - const [durationMs = 0] = rest as [number?]; - return requestOk(endpoint, simulatorPath(udid, "/button"), { - button, - durationMs, - }); - }, - home: (...args: [] | [string]) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/home"), null); - }, - dismissKeyboard: (...args: [] | [string]) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestOk( - endpoint, - simulatorPath(udid, "/dismiss-keyboard"), - null, - ); - }, - appSwitcher: (...args: [] | [string]) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/app-switcher"), null); - }, - rotateLeft: (...args: [] | [string]) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/rotate-left"), null); - }, - rotateRight: (...args: [] | [string]) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/rotate-right"), null); - }, - toggleAppearance: (...args: [] | [string]) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestOk( - endpoint, - simulatorPath(udid, "/toggle-appearance"), - null, - ); - }, - pasteboardSet: (...args) => { - const { udid, value: text } = resolveStringArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/pasteboard"), { - text, - }); - }, - pasteboardGet: async (...args: [] | [string]) => { - const { udid } = resolveNoArgDeviceCall(args); - const result = await requestJson<{ text?: string }>( - endpoint, - "GET", - simulatorPath(udid, "/pasteboard"), - ); - return result.text ?? ""; - }, - chromeProfile: (...args: [] | [string]) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestJson( - endpoint, - "GET", - simulatorPath(udid, "/chrome-profile"), - ); - }, - logs: async (...args) => { - const { udid, value: logsOptions } = - typeof args[0] === "string" - ? { udid: args[0], value: args[1] as LogsOptions | undefined } - : { udid: requireUdid(), value: args[0] as LogsOptions | undefined }; - const result = await requestJson<{ entries?: unknown[] }>( - endpoint, - "GET", - simulatorPath(udid, `/logs?${logsQuery(logsOptions)}`), - ); - return result.entries ?? []; - }, - tree: (...args) => { - const { udid, value: treeOptions } = - typeof args[0] === "string" - ? { udid: args[0], value: args[1] as QueryOptions | undefined } - : { udid: requireUdid(), value: args[0] as QueryOptions | undefined }; - return requestJson( - endpoint, - "GET", - simulatorPath(udid, `/accessibility-tree?${treeQuery(treeOptions)}`), - ); - }, - query: async (...args) => { - const { - udid, - value: selector, - rest, - } = resolveObjectArgDeviceCall(args); - const [treeOptions] = rest as [QueryOptions?]; - const result = await requestJson<{ matches: unknown[] }>( - endpoint, - "POST", - simulatorPath(udid, "/query"), - { - selector: selectorPayload(selector), - ...treeOptions, - }, - ); - return result.matches; - }, - assert: (...args) => { - const { - udid, - value: selector, - rest, - } = resolveObjectArgDeviceCall(args); - const [assertOptions] = rest as [QueryOptions?]; - return requestJson(endpoint, "POST", simulatorPath(udid, "/assert"), { - selector: selectorPayload(selector), - ...assertOptions, - }); - }, - waitFor: (...args) => { - const { - udid, - value: selector, - rest, - } = resolveObjectArgDeviceCall(args); - const [waitOptions] = rest as [ - (QueryOptions & { timeoutMs?: number; pollMs?: number })?, - ]; - return requestJson(endpoint, "POST", simulatorPath(udid, "/wait-for"), { - selector: selectorPayload(selector), - ...waitOptions, - }); - }, - batch: (...args) => { - const { - udid, - value: steps, - rest, - } = resolveObjectArgDeviceCall(args); - const [continueOnError = false] = rest as [boolean?]; - return requestJson(endpoint, "POST", simulatorPath(udid, "/batch"), { - steps, - continueOnError, - }); - }, - screenshot: ( - ...args: [string, ScreenshotOptions?] | [ScreenshotOptions?] - ) => { - const { udid, options } = - resolveOptionalObjectDeviceCall(args); - const params = new URLSearchParams(); - if (options?.withBezel ?? options?.bezel) { - params.set("bezel", "true"); - } - const query = params.toString(); - return requestBuffer( - endpoint, - simulatorPath(udid, `/screenshot.png${query ? `?${query}` : ""}`), - ); - }, - record: ( - ...args: [string, ScreenRecordingOptions?] | [ScreenRecordingOptions?] - ) => { - const { udid, options } = - resolveOptionalObjectDeviceCall(args); - return requestBuffer( - endpoint, - simulatorPath(udid, "/screen-recording"), - "POST", - { - seconds: options?.seconds ?? 5, - }, - ); - }, - close: () => { - if (options.keepDaemon) { - return; - } - if (result.child) { - result.child.kill(); - if (result.isolatedRoot) { - fs.rmSync(result.isolatedRoot, { recursive: true, force: true }); + ); + }, + scrollUntilVisible: (...args) => { + const { + udid, + value: selector, + rest, + } = resolveObjectArgDeviceCall(args); + const [scrollOptions] = rest as [ + | (QueryOptions & { + timeoutMs?: number; + pollMs?: number; + direction?: "up" | "down" | "left" | "right"; + durationMs?: number; + steps?: number; + }) + | undefined, + ]; + return requestJson( + endpoint, + "POST", + simulatorPath(udid, "/scroll-until-visible"), + { + selector: selectorPayload(selector), + ...scrollOptions, + }, + ); + }, + batch: (...args) => { + const { + udid, + value: steps, + rest, + } = resolveObjectArgDeviceCall(args); + const [continueOnError = false] = rest as [boolean?]; + return requestJson(endpoint, "POST", simulatorPath(udid, "/batch"), { + steps, + continueOnError, + }); + }, + screenshot: ( + ...args: [string, ScreenshotOptions?] | [ScreenshotOptions?] + ) => { + const { udid, options } = + resolveOptionalObjectDeviceCall(args); + const params = new URLSearchParams(); + if (options?.withBezel ?? options?.bezel) { + params.set("bezel", "true"); } - return; - } - if (result.started) { - spawnSync(cliPath, ["daemon", "stop"], { cwd: options.projectRoot }); - } - }, + const query = params.toString(); + return requestBuffer( + endpoint, + simulatorPath(udid, `/screenshot.png${query ? `?${query}` : ""}`), + ); + }, + record: ( + ...args: [string, ScreenRecordingOptions?] | [ScreenRecordingOptions?] + ) => { + const { udid, options } = + resolveOptionalObjectDeviceCall(args); + return requestBuffer( + endpoint, + simulatorPath(udid, "/screen-recording"), + "POST", + { + seconds: options?.seconds ?? 5, + }, + ); + }, + close: () => { + if (options.keepDaemon) { + return; + } + if (result.child) { + result.child.kill(); + if (result.isolatedRoot) { + fs.rmSync(result.isolatedRoot, { recursive: true, force: true }); + } + return; + } + if (result.started) { + spawnSync(cliPath, ["daemon", "stop"], { cwd: options.projectRoot }); + } + }, + }; + return session; }; - return session; + return createSession(options.udid); } async function startIsolatedDaemon( @@ -826,11 +942,18 @@ function logsQuery(options: LogsOptions = {}): string { function selectorPayload( selector: ElementSelector, -): Record { +): Record { return { + text: selector.text, id: selector.id, label: selector.label, value: selector.value, elementType: selector.type, + index: selector.index, + enabled: selector.enabled, + checked: selector.checked, + focused: selector.focused, + selected: selector.selected, + regex: selector.regex, }; } diff --git a/server/Cargo.lock b/server/Cargo.lock index 341ce51a..706464cc 100644 --- a/server/Cargo.lock +++ b/server/Cargo.lock @@ -2093,6 +2093,19 @@ dependencies = [ "serde", ] +[[package]] +name = "serde_yaml" +version = "0.9.34+deprecated" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6a8b1a1a2ebf674015cc02edccce75287f1a0130d394307b36743c2f5d504b47" +dependencies = [ + "indexmap 2.14.0", + "itoa", + "ryu", + "serde", + "unsafe-libyaml", +] + [[package]] name = "sha1" version = "0.10.6" @@ -2167,9 +2180,11 @@ dependencies = [ "plist", "prost", "qrcode", + "regex", "roxmltree", "serde", "serde_json", + "serde_yaml", "sha2", "thiserror 2.0.18", "tokio", @@ -2704,6 +2719,12 @@ dependencies = [ "subtle", ] +[[package]] +name = "unsafe-libyaml" +version = "0.2.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "673aac59facbab8a9007c7f6108d11f63b603f7cabff99fabf650fea5c32b861" + [[package]] name = "untrusted" version = "0.9.0" diff --git a/server/Cargo.toml b/server/Cargo.toml index 26701016..31099f82 100644 --- a/server/Cargo.toml +++ b/server/Cargo.toml @@ -18,8 +18,10 @@ plist = "1.7" prost = "0.13" qrcode = "0.14" roxmltree = "0.20" +regex = "1.11" serde = { version = "1.0", features = ["derive"] } serde_json = "1.0" +serde_yaml = "0.9" sha2 = "0.10" thiserror = "2.0" tokio = { version = "1.42", features = ["fs", "io-util", "macros", "process", "rt-multi-thread", "signal", "sync", "time"] } diff --git a/server/src/api/routes.rs b/server/src/api/routes.rs index c88e6cd3..17704270 100644 --- a/server/src/api/routes.rs +++ b/server/src/api/routes.rs @@ -30,6 +30,7 @@ use axum::routing::{get, post}; use axum::{Json, Router}; use bytes::{Bytes, BytesMut}; use futures::{SinkExt, StreamExt}; +use regex::Regex; use serde::Deserialize; use serde_json::Map; use serde_json::{json as json_value, Value}; @@ -575,11 +576,18 @@ struct ChromeButtonPngQuery { #[derive(Deserialize, Clone, Default)] #[serde(rename_all = "camelCase")] struct ElementSelectorPayload { + text: Option, id: Option, label: Option, value: Option, #[serde(alias = "type")] element_type: Option, + index: Option, + enabled: Option, + checked: Option, + focused: Option, + selected: Option, + regex: Option, } #[derive(Deserialize, Clone)] @@ -605,6 +613,21 @@ struct WaitForPayload { poll_ms: Option, } +#[derive(Deserialize, Clone)] +#[serde(rename_all = "camelCase")] +struct ScrollUntilVisiblePayload { + #[serde(default)] + selector: ElementSelectorPayload, + source: Option, + max_depth: Option, + include_hidden: Option, + timeout_ms: Option, + poll_ms: Option, + direction: Option, + duration_ms: Option, + steps: Option, +} + #[derive(Deserialize, Clone)] #[serde(rename_all = "camelCase")] struct TapElementPayload { @@ -642,6 +665,8 @@ enum BatchStep { Tap(TapElementPayload), WaitFor(WaitForPayload), Assert(WaitForPayload), + AssertNot(WaitForPayload), + ScrollUntilVisible(ScrollUntilVisiblePayload), Key { key_code: u16, modifiers: Option, @@ -863,6 +888,18 @@ pub fn router(state: AppState) -> Router { .route("/api/simulators/{udid}/query", post(accessibility_query)) .route("/api/simulators/{udid}/wait-for", post(wait_for_element)) .route("/api/simulators/{udid}/assert", post(assert_element)) + .route( + "/api/simulators/{udid}/wait-for-not", + post(wait_for_not_element), + ) + .route( + "/api/simulators/{udid}/assert-not", + post(assert_not_element), + ) + .route( + "/api/simulators/{udid}/scroll-until-visible", + post(scroll_until_visible), + ) .route("/api/simulators/{udid}/batch", post(run_batch)) .route("/api/simulators/{udid}/touch", post(send_touch)) .route("/api/simulators/{udid}/edge-touch", post(send_edge_touch)) @@ -2589,6 +2626,30 @@ async fn assert_element( wait_for_element_payload(state, udid, payload).await } +async fn wait_for_not_element( + State(state): State, + Path(udid): Path, + Json(payload): Json, +) -> Result, AppError> { + wait_for_absent_element_payload(state, udid, payload).await +} + +async fn assert_not_element( + State(state): State, + Path(udid): Path, + Json(payload): Json, +) -> Result, AppError> { + wait_for_absent_element_payload(state, udid, payload).await +} + +async fn scroll_until_visible( + State(state): State, + Path(udid): Path, + Json(payload): Json, +) -> Result, AppError> { + scroll_until_visible_payload(state, udid, payload).await +} + async fn run_batch( State(state): State, Path(udid): Path, @@ -4295,6 +4356,39 @@ async fn wait_for_element_payload( }))) } +async fn wait_for_absent_element_payload( + state: AppState, + udid: String, + payload: WaitForPayload, +) -> Result, AppError> { + let started = Instant::now(); + let timeout_ms = payload.timeout_ms.unwrap_or(5_000); + let poll_ms = payload.poll_ms.unwrap_or(100).max(10); + let deadline = Instant::now() + Duration::from_millis(timeout_ms); + loop { + let snapshot = accessibility_tree_value( + state.clone(), + udid.clone(), + payload.source.as_deref(), + payload.max_depth, + payload.include_hidden.unwrap_or(false), + ) + .await?; + if first_matching_element(&snapshot, &payload.selector).is_none() { + return Ok(json(json_value!({ + "ok": true, + "elapsedMs": started.elapsed().as_millis() as u64, + }))); + } + if timeout_ms == 0 || Instant::now() >= deadline { + return Err(AppError::bad_request( + "Accessibility element still matched the selector.", + )); + } + tokio::time::sleep(Duration::from_millis(poll_ms)).await; + } +} + async fn wait_for_snapshot_match( state: AppState, udid: String, @@ -4322,6 +4416,76 @@ async fn wait_for_snapshot_match( } } +async fn scroll_until_visible_payload( + state: AppState, + udid: String, + payload: ScrollUntilVisiblePayload, +) -> Result, AppError> { + let started = Instant::now(); + let timeout_ms = payload.timeout_ms.unwrap_or(10_000); + let poll_ms = payload.poll_ms.unwrap_or(100).max(10); + let deadline = Instant::now() + Duration::from_millis(timeout_ms); + let mut scroll_count = 0usize; + loop { + let snapshot = accessibility_tree_value( + state.clone(), + udid.clone(), + payload.source.as_deref(), + payload.max_depth, + payload.include_hidden.unwrap_or(false), + ) + .await?; + if let Some(found) = first_matching_element(&snapshot, &payload.selector) { + return Ok(json(json_value!({ + "ok": true, + "elapsedMs": started.elapsed().as_millis() as u64, + "scrollCount": scroll_count, + "match": compact_accessibility_node(&found), + }))); + } + if timeout_ms == 0 || Instant::now() >= deadline { + return Err(AppError::not_found("No accessibility element matched.")); + } + let (start_x, start_y, end_x, end_y) = + normalized_scroll_coordinates(payload.direction.as_deref())?; + let duration_ms = payload.duration_ms.unwrap_or(350); + let steps = payload.steps.unwrap_or(12).max(1); + let action_udid = udid.clone(); + run_bridge_action(state.clone(), move |bridge| { + let input = bridge.create_input_session(&action_udid)?; + let delay = Duration::from_millis(duration_ms / u64::from(steps)); + input.send_touch(start_x, start_y, "began")?; + for step in 1..steps { + let t = f64::from(step) / f64::from(steps); + input.send_touch( + start_x + (end_x - start_x) * t, + start_y + (end_y - start_y) * t, + "moved", + )?; + std::thread::sleep(delay); + } + input.send_touch(end_x, end_y, "ended") + }) + .await?; + scroll_count += 1; + tokio::time::sleep(Duration::from_millis(poll_ms)).await; + } +} + +fn normalized_scroll_coordinates( + direction: Option<&str>, +) -> Result<(f64, f64, f64, f64), AppError> { + match direction.unwrap_or("down").to_ascii_lowercase().as_str() { + "down" => Ok((0.5, 0.78, 0.5, 0.22)), + "up" => Ok((0.5, 0.22, 0.5, 0.78)), + "left" => Ok((0.78, 0.5, 0.22, 0.5)), + "right" => Ok((0.22, 0.5, 0.78, 0.5)), + other => Err(AppError::bad_request(format!( + "Unsupported scroll direction `{other}`." + ))), + } +} + async fn run_batch_step(state: AppState, udid: String, step: BatchStep) -> Result { match step { BatchStep::Sleep { ms, seconds } => { @@ -4346,6 +4510,18 @@ async fn run_batch_step(state: AppState, udid: String, step: BatchStep) -> Resul .ok_or_else(|| AppError::not_found("No accessibility element matched."))?; Ok(json_value!({ "action": "assert", "match": compact_accessibility_node(&found) })) } + BatchStep::AssertNot(payload) => { + let Json(_) = wait_for_absent_element_payload(state, udid, payload).await?; + Ok(json_value!({ "action": "assertNot" })) + } + BatchStep::ScrollUntilVisible(payload) => { + let Json(result) = scroll_until_visible_payload(state, udid, payload).await?; + Ok(json_value!({ + "action": "scrollUntilVisible", + "match": result.get("match").cloned().unwrap_or(Value::Null), + "scrollCount": result.get("scrollCount").cloned().unwrap_or(Value::Null), + })) + } BatchStep::Key { key_code, modifiers, @@ -4799,12 +4975,16 @@ fn query_compact_elements( let mut matches = Vec::new(); if let Some(roots) = snapshot.get("roots").and_then(Value::as_array) { for root in roots { - collect_query_matches(root, selector, limit, &mut matches); - if matches.len() >= limit { + let target_limit = selector.index.map(|index| index + 1).unwrap_or(limit); + collect_query_matches(root, selector, target_limit, &mut matches); + if matches.len() >= target_limit { break; } } } + if let Some(index) = selector.index { + return matches.into_iter().nth(index).into_iter().collect(); + } matches } @@ -4832,6 +5012,16 @@ fn collect_query_matches( fn first_matching_element(snapshot: &Value, selector: &ElementSelectorPayload) -> Option { let roots = snapshot.get("roots")?.as_array()?; + if let Some(index) = selector.index { + let mut matches = Vec::new(); + for root in roots { + collect_query_matches(root, selector, index + 1, &mut matches); + if matches.len() > index { + break; + } + } + return matches.into_iter().nth(index); + } for root in roots { if let Some(found) = first_matching_node(root, selector) { return Some(found.clone()); @@ -4864,48 +5054,122 @@ fn element_matches_selector(node: &Value, selector: &ElementSelectorPayload) -> if selector_is_empty(selector) { return true; } - selector - .element_type - .as_ref() - .is_none_or(|expected| string_fields_match(node, expected, &["type", "role", "className"])) - && selector.id.as_ref().is_none_or(|expected| { - string_fields_match( - node, - expected, - &[ - "AXIdentifier", - "AXUniqueId", - "inspectorId", - "id", - "identifier", - ], - ) - }) - && selector.label.as_ref().is_none_or(|expected| { - string_fields_match( - node, - expected, - &["AXLabel", "label", "title", "text", "name"], - ) - }) - && selector - .value - .as_ref() - .is_none_or(|expected| string_fields_match(node, expected, &["AXValue", "value"])) + let use_regex = selector.regex.unwrap_or(false); + selector.element_type.as_ref().is_none_or(|expected| { + string_fields_match(node, expected, use_regex, &["type", "role", "className"]) + }) && selector.id.as_ref().is_none_or(|expected| { + string_fields_match( + node, + expected, + use_regex, + &[ + "AXIdentifier", + "AXUniqueId", + "inspectorId", + "id", + "identifier", + ], + ) + }) && selector.text.as_ref().is_none_or(|expected| { + string_fields_match( + node, + expected, + use_regex, + &["AXLabel", "label", "title", "text", "name"], + ) + }) && selector.label.as_ref().is_none_or(|expected| { + string_fields_match( + node, + expected, + use_regex, + &["AXLabel", "label", "title", "text", "name"], + ) + }) && selector.value.as_ref().is_none_or(|expected| { + string_fields_match(node, expected, use_regex, &["AXValue", "value"]) + }) && selector.enabled.is_none_or(|expected| { + bool_fields_match( + node, + expected, + &[ + "enabled", + "AXEnabled", + "isEnabled", + "isUserInteractionEnabled", + ], + ) + }) && selector.checked.is_none_or(|expected| { + bool_or_state_fields_match( + node, + expected, + &["checked", "isChecked", "AXChecked"], + &["AXValue", "value"], + &["1", "true", "yes", "on", "checked", "selected"], + ) + }) && selector.focused.is_none_or(|expected| { + bool_fields_match(node, expected, &["focused", "isFocused", "AXFocused"]) + }) && selector.selected.is_none_or(|expected| { + bool_or_state_fields_match( + node, + expected, + &["selected", "isSelected", "AXSelected"], + &["AXValue", "value"], + &["selected", "1", "true", "yes", "on"], + ) + }) } fn selector_is_empty(selector: &ElementSelectorPayload) -> bool { - selector.id.is_none() + selector.text.is_none() + && selector.id.is_none() && selector.label.is_none() && selector.value.is_none() && selector.element_type.is_none() + && selector.enabled.is_none() + && selector.checked.is_none() + && selector.focused.is_none() + && selector.selected.is_none() } -fn string_fields_match(node: &Value, expected: &str, fields: &[&str]) -> bool { +fn string_fields_match(node: &Value, expected: &str, use_regex: bool, fields: &[&str]) -> bool { + let regex = use_regex.then(|| Regex::new(expected).ok()).flatten(); fields .iter() .filter_map(|field| node.get(*field).and_then(Value::as_str)) - .any(|value| value == expected) + .any(|value| { + if let Some(regex) = regex.as_ref() { + regex.is_match(value) + } else { + value == expected + } + }) +} + +fn bool_fields_match(node: &Value, expected: bool, fields: &[&str]) -> bool { + fields + .iter() + .find_map(|field| node.get(*field).and_then(Value::as_bool)) + .is_some_and(|value| value == expected) +} + +fn bool_or_state_fields_match( + node: &Value, + expected: bool, + bool_fields: &[&str], + string_fields: &[&str], + truthy_values: &[&str], +) -> bool { + if bool_fields_match(node, expected, bool_fields) { + return true; + } + string_fields + .iter() + .filter_map(|field| node.get(*field).and_then(Value::as_str)) + .any(|value| { + let truthy = truthy_values + .iter() + .any(|truthy| value.eq_ignore_ascii_case(truthy)); + truthy == expected + }) } fn tap_point_from_snapshot( @@ -4916,6 +5180,7 @@ fn tap_point_from_snapshot( .get("roots") .and_then(Value::as_array) .ok_or_else(|| AppError::not_found("Accessibility snapshot does not contain roots."))?; + let mut seen_matches = 0usize; for root in roots { let root_frame = root .get("frame") @@ -4923,7 +5188,7 @@ fn tap_point_from_snapshot( .ok_or_else(|| AppError::not_found("Accessibility root does not expose a frame."))?; let root_width = number_field(root_frame, "width")?; let root_height = number_field(root_frame, "height")?; - if let Some(node) = first_matching_node(root, selector) { + if let Some(node) = indexed_matching_node(root, selector, &mut seen_matches) { let frame = node .get("frame") .or_else(|| node.get("frameInScreen")) @@ -4939,6 +5204,30 @@ fn tap_point_from_snapshot( Err(AppError::not_found("No accessibility element matched.")) } +fn indexed_matching_node<'a>( + node: &'a Value, + selector: &ElementSelectorPayload, + seen_matches: &mut usize, +) -> Option<&'a Value> { + if element_matches_selector(node, selector) { + if selector.index.unwrap_or(0) == *seen_matches { + return Some(node); + } + *seen_matches += 1; + } + for child in node + .get("children") + .and_then(Value::as_array) + .into_iter() + .flatten() + { + if let Some(found) = indexed_matching_node(child, selector, seen_matches) { + return Some(found); + } + } + None +} + fn normalize_screen_point_from_snapshot( snapshot: &Value, x: f64, @@ -7108,10 +7397,17 @@ mod tests { fn selector() -> ElementSelectorPayload { ElementSelectorPayload { + text: None, id: Some("continue-button".to_owned()), label: Some("Continue".to_owned()), value: None, element_type: Some("Button".to_owned()), + index: None, + enabled: None, + checked: None, + focused: None, + selected: None, + regex: None, } } diff --git a/server/src/main.rs b/server/src/main.rs index 091f4bcd..f203efcb 100644 --- a/server/src/main.rs +++ b/server/src/main.rs @@ -35,6 +35,7 @@ use performance::PerformanceRegistry; use qrcode::{render::unicode, QrCode}; use serde::{Deserialize, Serialize}; use serde_json::Value; +use serde_yaml::Value as YamlValue; use simulators::registry::SessionRegistry; use std::collections::{hash_map::DefaultHasher, HashMap, HashSet}; use std::env; @@ -138,6 +139,10 @@ enum Command { #[command(subcommand)] command: ProviderCommand, }, + Maestro { + #[command(subcommand)] + command: MaestroCommand, + }, #[command(hide = true)] Serve { #[arg(long, default_value_t = 4310)] @@ -621,6 +626,18 @@ enum ProviderCommand { }, } +#[derive(Subcommand)] +enum MaestroCommand { + Test { + udid: String, + flow: PathBuf, + #[arg(long)] + artifacts_dir: Option, + #[arg(long)] + continue_on_error: bool, + }, +} + #[derive(Subcommand)] enum ServiceCommand { On { @@ -1876,6 +1893,7 @@ fn is_known_command(value: &str) -> bool { value, "ui" | "pair" | "daemon" + | "maestro" | "service" | "core-simulator" | "simctl-service" @@ -2743,6 +2761,20 @@ fn main() -> anyhow::Result<()> { }), }, Command::Provider { command } => run_provider_command(command), + Command::Maestro { command } => match command { + MaestroCommand::Test { + udid, + flow, + artifacts_dir, + continue_on_error, + } => { + let service_url = command_service_url(explicit_server_url.clone())?; + let report = + run_maestro_flow(&service_url, &udid, &flow, artifacts_dir, continue_on_error)?; + println_json(&report)?; + Ok(()) + } + }, Command::Serve { port, bind, @@ -5671,6 +5703,413 @@ fn parse_modifier_mask(value: &str) -> Result { Ok(mask) } +fn run_maestro_flow( + server_url: &str, + udid: &str, + flow: &Path, + artifacts_dir: Option, + continue_on_error: bool, +) -> anyhow::Result { + let raw = fs::read_to_string(flow) + .with_context(|| format!("read Maestro flow {}", flow.display()))?; + let yaml = parse_maestro_flow_yaml(&raw) + .with_context(|| format!("parse Maestro flow {}", flow.display()))?; + let commands = maestro_commands_from_flow(&yaml)?; + let artifact_root = artifacts_dir.unwrap_or_else(|| { + PathBuf::from("simdeck-artifacts").join( + flow.file_stem() + .and_then(|value| value.to_str()) + .unwrap_or("maestro-flow"), + ) + }); + fs::create_dir_all(&artifact_root)?; + + let mut steps = Vec::new(); + let mut failures = Vec::new(); + for (index, command) in commands.iter().enumerate() { + let started = Instant::now(); + let result = run_maestro_command(server_url, udid, command, &artifact_root); + match result { + Ok(detail) => steps.push(serde_json::json!({ + "index": index, + "ok": true, + "command": maestro_command_name(command), + "elapsedMs": started.elapsed().as_millis() as u64, + "detail": detail, + })), + Err(error) => { + let message = error.to_string(); + let screenshot = + capture_maestro_failure_screenshot(server_url, udid, &artifact_root, index + 1) + .ok(); + steps.push(serde_json::json!({ + "index": index, + "ok": false, + "command": maestro_command_name(command), + "elapsedMs": started.elapsed().as_millis() as u64, + "error": message, + "screenshot": screenshot, + })); + failures.push(message); + if !continue_on_error { + break; + } + } + } + } + + Ok(serde_json::json!({ + "ok": failures.is_empty(), + "flow": flow, + "udid": udid, + "steps": steps, + "failureCount": failures.len(), + "artifactsDir": artifact_root, + })) +} + +fn parse_maestro_flow_yaml(raw: &str) -> anyhow::Result { + let mut documents = Vec::new(); + for document in serde_yaml::Deserializer::from_str(raw) { + documents.push(YamlValue::deserialize(document)?); + } + match documents.len() { + 0 => Err(anyhow::anyhow!("Maestro flow is empty.")), + 1 => Ok(documents.remove(0)), + _ => { + let app_id = documents + .first() + .and_then(|value| yaml_string_or_field(value, "appId")); + let mut commands = documents + .pop() + .ok_or_else(|| anyhow::anyhow!("Maestro flow is empty."))?; + if let Some(app_id) = app_id { + fill_empty_launch_app_commands(&mut commands, &app_id); + } + Ok(commands) + } + } +} + +fn fill_empty_launch_app_commands(commands: &mut YamlValue, app_id: &str) { + let Some(commands) = commands.as_sequence_mut() else { + return; + }; + for command in commands { + if command.as_str() == Some("launchApp") { + let mut mapping = serde_yaml::Mapping::new(); + mapping.insert( + YamlValue::String("launchApp".to_owned()), + YamlValue::String(app_id.to_owned()), + ); + *command = YamlValue::Mapping(mapping); + continue; + } + let Some(mapping) = command.as_mapping_mut() else { + continue; + }; + let key = YamlValue::String("launchApp".to_owned()); + let Some(value) = mapping.get_mut(&key) else { + continue; + }; + if value.is_null() || value.as_mapping().is_some_and(|mapping| mapping.is_empty()) { + *value = YamlValue::String(app_id.to_owned()); + } + } +} + +fn maestro_commands_from_flow(flow: &YamlValue) -> anyhow::Result> { + match flow { + YamlValue::Sequence(commands) => Ok(commands.clone()), + YamlValue::Mapping(mapping) => mapping + .get(YamlValue::String("commands".to_owned())) + .and_then(YamlValue::as_sequence) + .cloned() + .ok_or_else(|| { + anyhow::anyhow!("Maestro flow must be a command list or contain `commands`.") + }), + _ => Err(anyhow::anyhow!( + "Maestro flow must be a command list or contain `commands`." + )), + } +} + +fn run_maestro_command( + server_url: &str, + udid: &str, + command: &YamlValue, + artifacts_dir: &Path, +) -> anyhow::Result { + let null_value = YamlValue::Null; + let (name, value) = if let Some(name) = command.as_str() { + (name, &null_value) + } else { + let Some(mapping) = command.as_mapping() else { + anyhow::bail!("Maestro command must be a string or mapping."); + }; + if mapping.len() != 1 { + anyhow::bail!("Maestro command must contain exactly one action."); + } + let (name, value) = mapping.iter().next().unwrap(); + ( + name.as_str() + .ok_or_else(|| anyhow::anyhow!("Maestro command name must be a string."))?, + value, + ) + }; + match name { + "launchApp" => { + let bundle_id = maestro_bundle_id(value)?; + service_launch(server_url, udid, &bundle_id)?; + Ok(serde_json::json!({ "bundleId": bundle_id })) + } + "openLink" => { + let url = yaml_string_or_field(value, "link") + .or_else(|| yaml_string_or_field(value, "url")) + .ok_or_else(|| anyhow::anyhow!("openLink requires a URL."))?; + service_open_url(server_url, udid, &url)?; + Ok(serde_json::json!({ "url": url })) + } + "tapOn" => { + let body = maestro_tap_body(value)?; + service_tap_element(server_url, udid, body)?; + Ok(Value::Null) + } + "inputText" => { + let text = yaml_string_or_field(value, "text") + .ok_or_else(|| anyhow::anyhow!("inputText requires text."))?; + service_batch( + server_url, + udid, + vec![serde_json::json!({ "action": "type", "text": text })], + false, + )?; + Ok(Value::Null) + } + "eraseText" => { + let count = yaml_u64_or_field(value, "charactersToErase").unwrap_or(64); + let keys = vec![42u16; count as usize]; + service_key_sequence(server_url, udid, &keys, 5)?; + Ok(serde_json::json!({ "charactersToErase": count })) + } + "pressKey" => { + let key = yaml_string_or_field(value, "key") + .ok_or_else(|| anyhow::anyhow!("pressKey requires a key."))?; + service_key(server_url, udid, parse_hid_key(&key)?, 0)?; + Ok(serde_json::json!({ "key": key })) + } + "assertVisible" => { + let selector = maestro_selector(value)?; + service_wait_for(server_url, udid, "assert", selector, 5_000)?; + Ok(Value::Null) + } + "assertNotVisible" => { + let selector = maestro_selector(value)?; + service_wait_for(server_url, udid, "assert-not", selector, 5_000)?; + Ok(Value::Null) + } + "scrollUntilVisible" => { + let selector_value = yaml_field(value, "element").unwrap_or(value); + let selector = maestro_selector(selector_value)?; + let direction = + yaml_string_or_field(value, "direction").unwrap_or_else(|| "down".to_owned()); + service_post_ok( + server_url, + udid, + "scroll-until-visible", + &serde_json::json!({ + "selector": selector, + "direction": direction, + "timeoutMs": yaml_u64_or_field(value, "timeout").unwrap_or(10_000), + }), + )?; + Ok(Value::Null) + } + "swipe" => { + let direction = + yaml_string_or_field(value, "direction").unwrap_or_else(|| "up".to_owned()); + let preset = match direction.as_str() { + "up" => "scroll-up", + "down" => "scroll-down", + "left" => "scroll-left", + "right" => "scroll-right", + _ => "scroll-up", + }; + service_batch( + server_url, + udid, + vec![serde_json::json!({ "action": "gesture", "preset": preset })], + false, + )?; + Ok(serde_json::json!({ "direction": direction })) + } + "takeScreenshot" => { + let name = yaml_string_or_field(value, "path") + .or_else(|| yaml_string_or_field(value, "name")) + .unwrap_or_else(|| "screenshot".to_owned()); + let path = artifacts_dir.join(format!("{}.png", name.trim_end_matches(".png"))); + let png = service_get_bytes( + server_url, + &format!( + "/api/simulators/{}/screenshot.png", + url_path_component(udid) + ), + )?; + fs::write(&path, png)?; + Ok(serde_json::json!({ "path": path })) + } + "waitForAnimationToEnd" | "waitForAnimationToEnd:" => { + sleep_ms( + yaml_u64_or_field(value, "timeout") + .unwrap_or(1_000) + .min(10_000), + ); + Ok(Value::Null) + } + other => Err(anyhow::anyhow!( + "Unsupported Maestro command `{other}` in this compatibility runner." + )), + } +} + +fn maestro_command_name(command: &YamlValue) -> String { + if let Some(name) = command.as_str() { + return name.to_owned(); + } + command + .as_mapping() + .and_then(|mapping| mapping.keys().next()) + .and_then(YamlValue::as_str) + .unwrap_or("unknown") + .to_owned() +} + +fn maestro_bundle_id(value: &YamlValue) -> anyhow::Result { + yaml_string_or_field(value, "appId") + .or_else(|| yaml_string_or_field(value, "bundleId")) + .ok_or_else(|| anyhow::anyhow!("launchApp requires `appId` or `bundleId`.")) +} + +fn maestro_tap_body(value: &YamlValue) -> anyhow::Result { + if let Some(point) = yaml_string_or_field(value, "point") { + let (x, y) = parse_maestro_point(&point)?; + return Ok(serde_json::json!({ "x": x, "y": y, "normalized": true })); + } + Ok(serde_json::json!({ + "selector": maestro_selector(value)?, + "waitTimeoutMs": yaml_u64_or_field(value, "timeout").unwrap_or(5_000), + })) +} + +fn maestro_selector(value: &YamlValue) -> anyhow::Result { + if let Some(text) = value.as_str() { + return Ok(serde_json::json!({ "text": text, "regex": true })); + } + let Some(mapping) = value.as_mapping() else { + anyhow::bail!("Selector must be a string or mapping."); + }; + let text = yaml_string_field(mapping, "text"); + let id = yaml_string_field(mapping, "id"); + Ok(serde_json::json!({ + "text": text, + "id": id, + "label": yaml_string_field(mapping, "label"), + "value": yaml_string_field(mapping, "value"), + "elementType": yaml_string_field(mapping, "type"), + "index": yaml_u64_field(mapping, "index"), + "enabled": yaml_bool_field(mapping, "enabled"), + "checked": yaml_bool_field(mapping, "checked"), + "focused": yaml_bool_field(mapping, "focused"), + "selected": yaml_bool_field(mapping, "selected"), + "regex": text.is_some() || id.is_some(), + })) +} + +fn service_wait_for( + server_url: &str, + udid: &str, + action: &str, + selector: Value, + timeout_ms: u64, +) -> anyhow::Result<()> { + service_post_ok( + server_url, + udid, + action, + &serde_json::json!({ "selector": selector, "timeoutMs": timeout_ms }), + ) +} + +fn parse_maestro_point(point: &str) -> anyhow::Result<(f64, f64)> { + let (x, y) = point + .split_once(',') + .ok_or_else(|| anyhow::anyhow!("point must be `x,y`."))?; + let parse = |value: &str| -> anyhow::Result { + let value = value.trim(); + if let Some(percent) = value.strip_suffix('%') { + Ok(percent.parse::()? / 100.0) + } else { + Ok(value.parse::()?) + } + }; + Ok((parse(x)?, parse(y)?)) +} + +fn capture_maestro_failure_screenshot( + server_url: &str, + udid: &str, + artifacts_dir: &Path, + step: usize, +) -> anyhow::Result { + let path = artifacts_dir.join(format!("failure-step-{step}.png")); + let png = service_get_bytes( + server_url, + &format!( + "/api/simulators/{}/screenshot.png", + url_path_component(udid) + ), + )?; + fs::write(&path, png)?; + Ok(path) +} + +fn yaml_field<'a>(value: &'a YamlValue, field: &str) -> Option<&'a YamlValue> { + value.as_mapping()?.get(YamlValue::String(field.to_owned())) +} + +fn yaml_string_or_field(value: &YamlValue, field: &str) -> Option { + value.as_str().map(str::to_owned).or_else(|| { + yaml_field(value, field) + .and_then(YamlValue::as_str) + .map(str::to_owned) + }) +} + +fn yaml_u64_or_field(value: &YamlValue, field: &str) -> Option { + value + .as_u64() + .or_else(|| yaml_field(value, field).and_then(YamlValue::as_u64)) +} + +fn yaml_string_field(mapping: &serde_yaml::Mapping, field: &str) -> Option { + mapping + .get(YamlValue::String(field.to_owned())) + .and_then(YamlValue::as_str) + .map(str::to_owned) +} + +fn yaml_u64_field(mapping: &serde_yaml::Mapping, field: &str) -> Option { + mapping + .get(YamlValue::String(field.to_owned())) + .and_then(YamlValue::as_u64) +} + +fn yaml_bool_field(mapping: &serde_yaml::Mapping, field: &str) -> Option { + mapping + .get(YamlValue::String(field.to_owned())) + .and_then(YamlValue::as_bool) +} + fn run_batch( bridge: &NativeBridge, udid: &str, @@ -5786,6 +6225,25 @@ fn batch_line_to_json_step(line: &str) -> anyhow::Result { "timeoutMs": args.value("timeout-ms").or_else(|| args.value("wait-timeout-ms")).and_then(|value| value.parse::().ok()).unwrap_or(5_000), "pollMs": args.value("poll-interval-ms").and_then(|value| value.parse::().ok()).unwrap_or(100), }), + "assert-not" | "assertNot" | "wait-for-not" | "waitForNot" => serde_json::json!({ + "action": "assertNot", + "selector": batch_selector_json(&args), + "source": args.value("source"), + "maxDepth": args.value("max-depth").and_then(|value| value.parse::().ok()), + "includeHidden": args.flag("include-hidden"), + "timeoutMs": args.value("timeout-ms").or_else(|| args.value("wait-timeout-ms")).and_then(|value| value.parse::().ok()).unwrap_or(5_000), + "pollMs": args.value("poll-interval-ms").and_then(|value| value.parse::().ok()).unwrap_or(100), + }), + "scroll-until-visible" | "scrollUntilVisible" => serde_json::json!({ + "action": "scrollUntilVisible", + "selector": batch_selector_json(&args), + "source": args.value("source"), + "maxDepth": args.value("max-depth").and_then(|value| value.parse::().ok()), + "includeHidden": args.flag("include-hidden"), + "timeoutMs": args.value("timeout-ms").or_else(|| args.value("wait-timeout-ms")).and_then(|value| value.parse::().ok()).unwrap_or(10_000), + "pollMs": args.value("poll-interval-ms").and_then(|value| value.parse::().ok()).unwrap_or(100), + "direction": args.value("direction").unwrap_or("down"), + }), "key" => serde_json::json!({ "action": "key", "keyCode": parse_hid_key(tokens.get(1).map(String::as_str).unwrap_or(""))?, @@ -6233,10 +6691,17 @@ fn required_f64(args: &StepOptions, key: &str) -> Result Value { serde_json::json!({ + "text": args.value("text"), "id": args.value("id"), "label": args.value("label"), "value": args.value("value"), "elementType": args.value("element-type"), + "index": args.value("index").and_then(|value| value.parse::().ok()), + "enabled": args.value("enabled").and_then(parse_bool_value), + "checked": args.value("checked").and_then(parse_bool_value), + "focused": args.value("focused").and_then(parse_bool_value), + "selected": args.value("selected").and_then(parse_bool_value), + "regex": args.flag("regex"), }) } @@ -6249,6 +6714,14 @@ fn batch_selector_from_args(args: &StepOptions) -> ElementSelector { } } +fn parse_bool_value(value: &str) -> Option { + match value.to_ascii_lowercase().as_str() { + "true" | "1" | "yes" | "on" => Some(true), + "false" | "0" | "no" | "off" => Some(false), + _ => None, + } +} + fn wait_for_batch_selector( bridge: &NativeBridge, udid: &str, @@ -6876,12 +7349,13 @@ fn default_client_root() -> anyhow::Result { mod tests { use super::{ batch_line_to_json_step, daemon_matches_launch_options, http_url_for_host, is_tailscale_ip, - normalize_accessibility_point_for_display, parse_workspace_daemon_process_line, + maestro_commands_from_flow, maestro_selector, normalize_accessibility_point_for_display, + parse_maestro_flow_yaml, parse_maestro_point, parse_workspace_daemon_process_line, render_qr_code, server_health_watchdog_should_restart, service_post_error_is_retryable, simdeck_pair_url, studio_daemon_restart_args, workspace_daemon_process_is_current, Cli, Command, DaemonCommand, DaemonLaunchOptions, DaemonMetadata, PairingAddress, ServiceCommand, StreamQualityProfileArg, StudioExposeOptions, VideoCodecMode, - WorkspaceDaemonProcess, DEFAULT_LOCAL_STREAM_QUALITY_PROFILE, + WorkspaceDaemonProcess, YamlValue, DEFAULT_LOCAL_STREAM_QUALITY_PROFILE, SERVER_HEALTH_WATCHDOG_FAILURE_THRESHOLD, SERVER_HEALTH_WATCHDOG_HTTP_FAILURE_THRESHOLD, }; use clap::Parser; @@ -7328,6 +7802,59 @@ mod tests { assert_eq!(step["timeoutMs"], 5000); } + #[test] + fn batch_assert_not_and_scroll_map_to_daemon_actions() { + let assert_not = batch_line_to_json_step("assert-not --text Loading --regex").unwrap(); + assert_eq!(assert_not["action"], "assertNot"); + assert_eq!(assert_not["selector"]["text"], "Loading"); + assert_eq!(assert_not["selector"]["regex"], true); + + let scroll = + batch_line_to_json_step("scroll-until-visible --text Settings --direction down") + .unwrap(); + assert_eq!(scroll["action"], "scrollUntilVisible"); + assert_eq!(scroll["selector"]["text"], "Settings"); + assert_eq!(scroll["direction"], "down"); + } + + #[test] + fn maestro_flow_accepts_config_with_commands() { + let yaml = parse_maestro_flow_yaml( + r#" +appId: com.example.App +--- +- launchApp +- tapOn: Continue +"#, + ) + .unwrap(); + let commands = maestro_commands_from_flow(&yaml).unwrap(); + assert_eq!(commands.len(), 2); + assert_eq!(commands[0]["launchApp"].as_str(), Some("com.example.App")); + } + + #[test] + fn maestro_selector_maps_text_and_state() { + let yaml: YamlValue = serde_yaml::from_str( + r#" +text: Continue.* +enabled: true +index: 1 +"#, + ) + .unwrap(); + let selector = maestro_selector(&yaml).unwrap(); + assert_eq!(selector["text"], "Continue.*"); + assert_eq!(selector["enabled"], true); + assert_eq!(selector["index"], 1); + assert_eq!(selector["regex"], true); + } + + #[test] + fn maestro_percent_points_become_normalized_coordinates() { + assert_eq!(parse_maestro_point("50%,75%").unwrap(), (0.5, 0.75)); + } + #[test] fn server_health_watchdog_restarts_when_http_listener_is_unhealthy() { assert!(server_health_watchdog_should_restart( diff --git a/skills/simdeck/SKILL.md b/skills/simdeck/SKILL.md index 7b677138..45cf76a4 100644 --- a/skills/simdeck/SKILL.md +++ b/skills/simdeck/SKILL.md @@ -202,7 +202,7 @@ Batch rules: one source (`--step`, `--file`, or `--stdin`); keep `` at bat For JS tests, batch can combine action and verification without extra CLI process startup: ```ts -await simdeck.batch(udid, [ +await simdeck.batch([ { action: "tap", selector: { label: "Continue" }, waitTimeoutMs: 5000 }, { action: "waitFor", @@ -213,6 +213,12 @@ await simdeck.batch(udid, [ ]); ``` +For app-style flows, SimDeck can run a practical subset of Maestro YAML: + +```bash +simdeck maestro test flow.yaml --artifacts-dir artifacts/maestro +``` + ## Evidence ```bash From 673cd858a73c749cb5ce490d78320c7286b204a8 Mon Sep 17 00:00:00 2001 From: DjDeveloperr Date: Thu, 7 May 2026 11:45:53 -0400 Subject: [PATCH 2/6] Add Maestro-compatible YAML runner and testing API --- server/src/main.rs | 21 ++++++++++++++++++--- 1 file changed, 18 insertions(+), 3 deletions(-) diff --git a/server/src/main.rs b/server/src/main.rs index f203efcb..6a91daa2 100644 --- a/server/src/main.rs +++ b/server/src/main.rs @@ -2768,11 +2768,16 @@ fn main() -> anyhow::Result<()> { artifacts_dir, continue_on_error, } => { - let service_url = command_service_url(explicit_server_url.clone())?; + let service_url = command_service_url(explicit_server_url.as_deref())?; let report = run_maestro_flow(&service_url, &udid, &flow, artifacts_dir, continue_on_error)?; + let ok = report.get("ok").and_then(Value::as_bool).unwrap_or(false); println_json(&report)?; - Ok(()) + if ok { + Ok(()) + } else { + anyhow::bail!("Maestro-compatible flow failed.") + } } }, Command::Serve { @@ -5991,7 +5996,10 @@ fn maestro_bundle_id(value: &YamlValue) -> anyhow::Result { } fn maestro_tap_body(value: &YamlValue) -> anyhow::Result { - if let Some(point) = yaml_string_or_field(value, "point") { + if let Some(point) = yaml_field(value, "point") + .and_then(YamlValue::as_str) + .map(str::to_owned) + { let (x, y) = parse_maestro_point(&point)?; return Ok(serde_json::json!({ "x": x, "y": y, "normalized": true })); } @@ -7855,6 +7863,13 @@ index: 1 assert_eq!(parse_maestro_point("50%,75%").unwrap(), (0.5, 0.75)); } + #[test] + fn maestro_tap_on_string_maps_to_text_selector() { + let yaml: YamlValue = serde_yaml::from_str("Continue").unwrap(); + let body = super::maestro_tap_body(&yaml).unwrap(); + assert_eq!(body["selector"]["text"], "Continue"); + } + #[test] fn server_health_watchdog_restarts_when_http_listener_is_unhealthy() { assert!(server_health_watchdog_should_restart( From e90839a3b0b4a3219b76526b63274670f3fbbf50 Mon Sep 17 00:00:00 2001 From: DjDeveloperr Date: Fri, 22 May 2026 20:47:52 -0400 Subject: [PATCH 3/6] fix: tighten test runner edge cases --- server/src/api/routes.rs | 154 ++++++++++++++++++++++++++++++++------- server/src/main.rs | 78 +++++++++++++++++--- 2 files changed, 194 insertions(+), 38 deletions(-) diff --git a/server/src/api/routes.rs b/server/src/api/routes.rs index 17704270..8a006c99 100644 --- a/server/src/api/routes.rs +++ b/server/src/api/routes.rs @@ -4446,32 +4446,108 @@ async fn scroll_until_visible_payload( if timeout_ms == 0 || Instant::now() >= deadline { return Err(AppError::not_found("No accessibility element matched.")); } - let (start_x, start_y, end_x, end_y) = - normalized_scroll_coordinates(payload.direction.as_deref())?; - let duration_ms = payload.duration_ms.unwrap_or(350); - let steps = payload.steps.unwrap_or(12).max(1); - let action_udid = udid.clone(); - run_bridge_action(state.clone(), move |bridge| { - let input = bridge.create_input_session(&action_udid)?; - let delay = Duration::from_millis(duration_ms / u64::from(steps)); - input.send_touch(start_x, start_y, "began")?; - for step in 1..steps { - let t = f64::from(step) / f64::from(steps); - input.send_touch( - start_x + (end_x - start_x) * t, - start_y + (end_y - start_y) * t, - "moved", - )?; - std::thread::sleep(delay); - } - input.send_touch(end_x, end_y, "ended") - }) - .await?; + let scroll_plan = scroll_input_plan_for_udid(&udid, &payload)?; + perform_scroll_input(state.clone(), udid.clone(), scroll_plan).await?; scroll_count += 1; tokio::time::sleep(Duration::from_millis(poll_ms)).await; } } +#[derive(Clone, Copy, Debug, PartialEq)] +struct NormalizedSwipe { + start_x: f64, + start_y: f64, + end_x: f64, + end_y: f64, + duration_ms: u64, + steps: u32, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum ScrollInputBackend { + Android, + Native, +} + +#[derive(Clone, Copy, Debug, PartialEq)] +struct ScrollInputPlan { + backend: ScrollInputBackend, + swipe: NormalizedSwipe, +} + +fn scroll_input_plan_for_udid( + udid: &str, + payload: &ScrollUntilVisiblePayload, +) -> Result { + let (start_x, start_y, end_x, end_y) = + normalized_scroll_coordinates(payload.direction.as_deref())?; + Ok(ScrollInputPlan { + backend: if android::is_android_id(udid) { + ScrollInputBackend::Android + } else { + ScrollInputBackend::Native + }, + swipe: NormalizedSwipe { + start_x, + start_y, + end_x, + end_y, + duration_ms: payload.duration_ms.unwrap_or(350), + steps: payload.steps.unwrap_or(12).max(1), + }, + }) +} + +async fn perform_scroll_input( + state: AppState, + udid: String, + plan: ScrollInputPlan, +) -> Result<(), AppError> { + let swipe = plan.swipe; + match plan.backend { + ScrollInputBackend::Android => { + run_android_action(state, move |android| { + android.send_swipe( + &udid, + swipe.start_x, + swipe.start_y, + swipe.end_x, + swipe.end_y, + swipe.duration_ms, + ) + }) + .await + } + ScrollInputBackend::Native => { + run_bridge_action(state, move |bridge| { + if bridge_simulator_is_tvos(&bridge, &udid) { + let key_code = tvos_remote_key_for_touch_motion( + swipe.start_x, + swipe.start_y, + swipe.end_x, + swipe.end_y, + ); + return press_tvos_remote_key(&bridge, &udid, key_code); + } + let input = bridge.create_input_session(&udid)?; + let delay = Duration::from_millis(swipe.duration_ms / u64::from(swipe.steps)); + input.send_touch(swipe.start_x, swipe.start_y, "began")?; + for step in 1..swipe.steps { + let t = f64::from(step) / f64::from(swipe.steps); + input.send_touch( + swipe.start_x + (swipe.end_x - swipe.start_x) * t, + swipe.start_y + (swipe.end_y - swipe.start_y) * t, + "moved", + )?; + std::thread::sleep(delay); + } + input.send_touch(swipe.end_x, swipe.end_y, "ended") + }) + .await + } + } +} + fn normalized_scroll_coordinates( direction: Option<&str>, ) -> Result<(f64, f64, f64, f64), AppError> { @@ -7382,12 +7458,13 @@ mod tests { normalize_screen_point_from_snapshot, normalized_gesture_coordinates, parse_lsof_tcp_listener, parse_ui_application_service_line, process_identifier_from_accessibility_snapshot, resolved_stream_quality_limits, - split_filter_values, stream_quality_profile, suppress_native_ax_translation_error, - tap_point_from_snapshot, trim_tree_depth, ui_application_foreground_score, - AccessibilityHierarchySource, ElementSelectorPayload, InspectorSession, - InspectorSessionTransport, StreamClientForegroundRegistry, StreamQualityLimits, - StreamQualityPayload, UIKitApplicationServiceDetails, SOURCE_FLUTTER, SOURCE_NATIVE_AX, - SOURCE_NATIVE_SCRIPT, SOURCE_REACT_NATIVE, SOURCE_SWIFTUI, SOURCE_UIKIT, + scroll_input_plan_for_udid, split_filter_values, stream_quality_profile, + suppress_native_ax_translation_error, tap_point_from_snapshot, trim_tree_depth, + ui_application_foreground_score, AccessibilityHierarchySource, ElementSelectorPayload, + InspectorSession, InspectorSessionTransport, ScrollInputBackend, ScrollUntilVisiblePayload, + StreamClientForegroundRegistry, StreamQualityLimits, StreamQualityPayload, + UIKitApplicationServiceDetails, SOURCE_FLUTTER, SOURCE_NATIVE_AX, SOURCE_NATIVE_SCRIPT, + SOURCE_REACT_NATIVE, SOURCE_SWIFTUI, SOURCE_UIKIT, }; use crate::inspector::PublishedInspector; use crate::metrics::counters::ClientStreamStats; @@ -7612,6 +7689,29 @@ mod tests { assert!(normalized_gesture_coordinates("orbit", None).is_err()); } + #[test] + fn scroll_until_visible_plans_android_swipe_for_android_ids() { + let payload = ScrollUntilVisiblePayload { + selector: ElementSelectorPayload::default(), + source: Some("android-uiautomator".to_owned()), + max_depth: None, + include_hidden: None, + timeout_ms: None, + poll_ms: None, + direction: Some("down".to_owned()), + duration_ms: Some(225), + steps: Some(7), + }; + + let plan = scroll_input_plan_for_udid("android:Pixel_8", &payload).unwrap(); + + assert_eq!(plan.backend, ScrollInputBackend::Android); + assert_eq!(plan.swipe.start_y, 0.78); + assert_eq!(plan.swipe.end_y, 0.22); + assert_eq!(plan.swipe.duration_ms, 225); + assert_eq!(plan.swipe.steps, 7); + } + #[test] fn compact_accessibility_snapshot_removes_nested_noise_but_keeps_identity() { let compact = compact_accessibility_snapshot(&accessibility_snapshot()); diff --git a/server/src/main.rs b/server/src/main.rs index 6a91daa2..592b2055 100644 --- a/server/src/main.rs +++ b/server/src/main.rs @@ -5933,12 +5933,12 @@ fn run_maestro_command( "swipe" => { let direction = yaml_string_or_field(value, "direction").unwrap_or_else(|| "up".to_owned()); - let preset = match direction.as_str() { + let preset = match direction.to_ascii_lowercase().as_str() { "up" => "scroll-up", "down" => "scroll-down", "left" => "scroll-left", "right" => "scroll-right", - _ => "scroll-up", + _ => anyhow::bail!("Unsupported Maestro swipe direction `{direction}`."), }; service_batch( server_url, @@ -6017,7 +6017,15 @@ fn maestro_selector(value: &YamlValue) -> anyhow::Result { anyhow::bail!("Selector must be a string or mapping."); }; let text = yaml_string_field(mapping, "text"); - let id = yaml_string_field(mapping, "id"); + let explicit_regex = yaml_bool_field(mapping, "regex"); + let use_regex = explicit_regex.unwrap_or_else(|| text.is_some()); + let id = yaml_string_field(mapping, "id").map(|id| { + if use_regex && explicit_regex != Some(true) { + anchored_regex_literal(&id) + } else { + id + } + }); Ok(serde_json::json!({ "text": text, "id": id, @@ -6029,10 +6037,14 @@ fn maestro_selector(value: &YamlValue) -> anyhow::Result { "checked": yaml_bool_field(mapping, "checked"), "focused": yaml_bool_field(mapping, "focused"), "selected": yaml_bool_field(mapping, "selected"), - "regex": text.is_some() || id.is_some(), + "regex": use_regex, })) } +fn anchored_regex_literal(value: &str) -> String { + format!("^{}$", regex::escape(value)) +} + fn service_wait_for( server_url: &str, udid: &str, @@ -7359,17 +7371,18 @@ mod tests { batch_line_to_json_step, daemon_matches_launch_options, http_url_for_host, is_tailscale_ip, maestro_commands_from_flow, maestro_selector, normalize_accessibility_point_for_display, parse_maestro_flow_yaml, parse_maestro_point, parse_workspace_daemon_process_line, - render_qr_code, server_health_watchdog_should_restart, service_post_error_is_retryable, - simdeck_pair_url, studio_daemon_restart_args, workspace_daemon_process_is_current, Cli, - Command, DaemonCommand, DaemonLaunchOptions, DaemonMetadata, PairingAddress, - ServiceCommand, StreamQualityProfileArg, StudioExposeOptions, VideoCodecMode, - WorkspaceDaemonProcess, YamlValue, DEFAULT_LOCAL_STREAM_QUALITY_PROFILE, - SERVER_HEALTH_WATCHDOG_FAILURE_THRESHOLD, SERVER_HEALTH_WATCHDOG_HTTP_FAILURE_THRESHOLD, + render_qr_code, run_maestro_command, server_health_watchdog_should_restart, + service_post_error_is_retryable, simdeck_pair_url, studio_daemon_restart_args, + workspace_daemon_process_is_current, Cli, Command, DaemonCommand, DaemonLaunchOptions, + DaemonMetadata, PairingAddress, ServiceCommand, StreamQualityProfileArg, + StudioExposeOptions, VideoCodecMode, WorkspaceDaemonProcess, YamlValue, + DEFAULT_LOCAL_STREAM_QUALITY_PROFILE, SERVER_HEALTH_WATCHDOG_FAILURE_THRESHOLD, + SERVER_HEALTH_WATCHDOG_HTTP_FAILURE_THRESHOLD, }; use clap::Parser; use std::collections::HashMap; use std::net::{IpAddr, Ipv4Addr}; - use std::path::PathBuf; + use std::path::{Path, PathBuf}; fn daemon_metadata_for_test( port: u16, @@ -7858,6 +7871,49 @@ index: 1 assert_eq!(selector["regex"], true); } + #[test] + fn maestro_selector_keeps_id_literals_exact_by_default() { + let yaml: YamlValue = serde_yaml::from_str("id: login.button").unwrap(); + let selector = maestro_selector(&yaml).unwrap(); + + assert_eq!(selector["id"], "login.button"); + assert_eq!(selector["regex"], false); + } + + #[test] + fn maestro_selector_escapes_literal_ids_when_text_requires_regex() { + let yaml: YamlValue = serde_yaml::from_str( + r#" +text: Continue.* +id: login.button +"#, + ) + .unwrap(); + let selector = maestro_selector(&yaml).unwrap(); + + assert_eq!(selector["text"], "Continue.*"); + assert_eq!(selector["id"], "^login\\.button$"); + assert_eq!(selector["regex"], true); + } + + #[test] + fn maestro_swipe_rejects_unknown_directions() { + let command: YamlValue = serde_yaml::from_str( + r#" +swipe: + direction: rigth +"#, + ) + .unwrap(); + let error = + run_maestro_command("http://127.0.0.1:9", "test-udid", &command, Path::new(".")) + .unwrap_err(); + + assert!(error + .to_string() + .contains("Unsupported Maestro swipe direction `rigth`")); + } + #[test] fn maestro_percent_points_become_normalized_coordinates() { assert_eq!(parse_maestro_point("50%,75%").unwrap(), (0.5, 0.75)); From 078969927a66fd39857e332ae2e751810093d4ad Mon Sep 17 00:00:00 2001 From: DjDeveloperr Date: Fri, 22 May 2026 21:44:31 -0400 Subject: [PATCH 4/6] feat: add agent-friendly simulator CLI shortcuts --- README.md | 12 +- docs/api/rest.md | 11 +- docs/cli/commands.md | 6 +- docs/cli/flags.md | 28 +- docs/cli/index.md | 5 +- docs/guide/index.md | 3 +- docs/guide/quick-start.md | 3 +- docs/inspector/index.md | 3 +- packages/simdeck-test/dist/index.d.ts | 1 + packages/simdeck-test/dist/index.js | 2 + packages/simdeck-test/src/index.ts | 2 + scripts/integration/cli.mjs | 51 +++- server/src/accessibility.rs | 212 +++++++++++++++ server/src/api/routes.rs | 108 ++++++-- server/src/main.rs | 365 ++++++++++++++++++++++++-- skills/simdeck/SKILL.md | 5 +- 16 files changed, 742 insertions(+), 75 deletions(-) create mode 100644 server/src/accessibility.rs diff --git a/README.md b/README.md index 81b07452..2af1f404 100644 --- a/README.md +++ b/README.md @@ -95,6 +95,8 @@ CLI commands automatically use the same warm daemon: simdeck list simdeck tap 0.5 0.5 --normalized simdeck describe --format agent --max-depth 2 +SIMDECK_DEVICE= simdeck tap "Continue" +simdeck --device describe --format agent --max-depth 2 --interactive ``` ## Daemon @@ -155,11 +157,13 @@ simdeck record --seconds 5 --output screen-recording.mp4 simdeck stream --frames 120 > stream.h264 simdeck describe simdeck describe --format agent --max-depth 4 +simdeck describe --format agent --max-depth 4 --interactive simdeck describe --point 120,240 simdeck wait-for --label "Welcome" --timeout-ms 5000 simdeck assert --id login.button --source auto --max-depth 8 simdeck tap 120 240 simdeck tap --label "Continue" --wait-timeout-ms 5000 +simdeck tap "Continue" simdeck swipe 200 700 200 200 simdeck gesture scroll-down simdeck pinch --start-distance 160 --end-distance 80 @@ -214,9 +218,11 @@ external tools such as `ffplay`. Flutter, or UIKit in-app inspectors, then falls back to the built-in private CoreSimulator accessibility bridge. Use `--format agent` or `--format compact-json` for -lower-token hierarchy dumps. Coordinate commands accept screen coordinates from -the accessibility tree by default; pass `--normalized` to send `0.0..1.0` -coordinates directly. +lower-token hierarchy dumps, and add `--interactive`/`-i` when an agent only +needs actionable elements plus their ancestors. `describe` and `tap` can infer a +target from `--device`, `SIMDECK_DEVICE`, `SIMDECK_UDID`, or the only booted +simulator. Coordinate commands accept screen coordinates from the accessibility +tree by default; pass `--normalized` to send `0.0..1.0` coordinates directly. ## JS/TS Tests diff --git a/docs/api/rest.md b/docs/api/rest.md index c4ebe7c8..c3347432 100644 --- a/docs/api/rest.md +++ b/docs/api/rest.md @@ -207,11 +207,12 @@ Touch, edge-touch, and multi-touch coordinates are normalized from `0.0` to `1.0 Tree query parameters: -| Parameter | Values | -| --------------- | --------------------------------------------------------------------------------------------------------- | -| `source` | `auto`, `nativescript`, `react-native`, `flutter`, `swiftui`, `uikit`, `native-ax`, `android-uiautomator` | -| `maxDepth` | Integer depth limit | -| `includeHidden` | `true` or `false` | +| Parameter | Values | +| ----------------- | --------------------------------------------------------------------------------------------------------- | +| `source` | `auto`, `nativescript`, `react-native`, `flutter`, `swiftui`, `uikit`, `native-ax`, `android-uiautomator` | +| `maxDepth` | Integer depth limit | +| `includeHidden` | `true` or `false` | +| `interactiveOnly` | `true` keeps actionable elements plus their ancestors | Point query parameters: diff --git a/docs/cli/commands.md b/docs/cli/commands.md index 1c7871e7..1ed8a13e 100644 --- a/docs/cli/commands.md +++ b/docs/cli/commands.md @@ -69,6 +69,7 @@ simdeck toggle-appearance ```sh simdeck describe simdeck describe --format agent --max-depth 4 +simdeck describe --format agent --max-depth 4 --interactive simdeck describe --format compact-json simdeck describe --source nativescript simdeck describe --source react-native @@ -80,7 +81,7 @@ simdeck wait-for --label "Welcome" --timeout-ms 5000 simdeck assert --id login.button --source auto --max-depth 8 ``` -Default source selection prefers a connected framework inspector, then the Swift in-app agent, then native accessibility. +Default source selection prefers a connected framework inspector, then the Swift in-app agent, then native accessibility. Use `--interactive` or `-i` to keep actionable elements and the ancestor context needed to find them. For quick agent loops, `describe` can infer the device from `--device`, `SIMDECK_DEVICE`, `SIMDECK_UDID`, or the only booted simulator. ## Performance @@ -97,12 +98,13 @@ Performance data is simulator-only and uses host-process telemetry for matching ## Input -Coordinates are screen points unless `--normalized` is present. +Coordinates are screen points unless `--normalized` is present. `tap "Continue"` is shorthand for a label tap on the inferred device. Use `--device ` or `SIMDECK_DEVICE=` when more than one simulator is booted. ```sh simdeck tap 120 240 simdeck tap 0.5 0.5 --normalized simdeck tap --label "Continue" --wait-timeout-ms 5000 +simdeck tap "Continue" simdeck swipe 200 700 200 200 simdeck gesture scroll-down simdeck pinch --start-distance 160 --end-distance 80 diff --git a/docs/cli/flags.md b/docs/cli/flags.md index 12edc2e5..e59c3a87 100644 --- a/docs/cli/flags.md +++ b/docs/cli/flags.md @@ -10,21 +10,22 @@ simdeck daemon start --help ## Global -| Flag | Env | Purpose | -| -------------------- | -------------------- | -------------------------------- | -| `--server-url ` | `SIMDECK_SERVER_URL` | Target a specific running daemon | +| Flag | Env | Purpose | +| --------------------- | -------------------- | ---------------------------------------------- | +| `--server-url ` | `SIMDECK_SERVER_URL` | Target a specific running daemon | +| `--device ` | `SIMDECK_DEVICE` | Default simulator for inferred-device commands | ## Server Options Used by `simdeck ui`, `daemon start`, `daemon restart`, `service on`, and `service restart`. | Flag | Default | Notes | -| ---------------------------- | -------------- | ----------------------------------------------------------------------------------- | ------ | ------------ | +| ---------------------------- | -------------- | ----------------------------------------------------------------------------------- | | `--port ` | `4310` | HTTP port. LaunchAgent service commands probe up to 4320 when this port is occupied | | `--bind ` | `127.0.0.1` | Use `0.0.0.0` or `::` for LAN access | | `--advertise-host ` | detected | Host printed for remote browsers | | `--client-root ` | bundled client | Static client directory | -| `--video-codec auto | hardware | software` | `auto` | Encoder mode | +| `--video-codec ` | `auto` | `auto`, `hardware`, or `software` | | `--stream-quality ` | `full` | `full`, `balanced`, `economy`, `low`, `tiny`, `ci-software`, and related profiles | | `--local-stream-fps ` | `60` | Local stream frame target | | `--low-latency` | off | Conservative software H.264 profile | @@ -32,14 +33,15 @@ Used by `simdeck ui`, `daemon start`, `daemon restart`, `service on`, and `servi ## `describe` -| Flag | Purpose | -| ------------------ | ------------------------------------------------- | ------------ | ------------------- | ----- | --------- | -------------------- | --------------------- | -| `--format json | compact-json | agent` | Choose output shape | -| `--source auto | nativescript | react-native | flutter | uikit | native-ax | android-uiautomator` | Pick inspector source | -| `--max-depth ` | Trim hierarchy depth | -| `--include-hidden` | Include hidden nodes when supported | -| `--point ,` | Describe the element at a screen point | -| `--direct` | Skip daemon and use native accessibility directly | +| Flag | Purpose | +| --------------------- | ------------------------------------------------------------------------------------------------- | +| `--format ` | `json`, `compact-json`, or `agent` | +| `--source ` | `auto`, `nativescript`, `react-native`, `flutter`, `uikit`, `native-ax`, or `android-uiautomator` | +| `--max-depth ` | Trim hierarchy depth | +| `--include-hidden` | Include hidden nodes when supported | +| `-i`, `--interactive` | Keep only actionable elements plus ancestors | +| `--point ,` | Describe the element at a screen point | +| `--direct` | Skip daemon and use native accessibility directly | ## Input diff --git a/docs/cli/index.md b/docs/cli/index.md index d75fa14e..ff53a86e 100644 --- a/docs/cli/index.md +++ b/docs/cli/index.md @@ -37,7 +37,8 @@ simdeck install /path/to/App.ipa simdeck launch com.example.App simdeck open-url https://example.com simdeck tap --label "Continue" --wait-timeout-ms 5000 -simdeck describe --format agent --max-depth 3 +simdeck tap "Continue" +simdeck describe --format agent --max-depth 3 --interactive simdeck screenshot --output screen.png simdeck screenshot --with-bezel --output screen-bezel.png simdeck record --seconds 5 --output screen-recording.mp4 @@ -46,6 +47,8 @@ simdeck stats simdeck sample --seconds 3 ``` +`tap` and `describe` can infer their simulator from `--device`, `SIMDECK_DEVICE`, `SIMDECK_UDID`, or the only booted simulator. + Most successful commands print JSON so they can be piped into tools such as `jq`. ## Help diff --git a/docs/guide/index.md b/docs/guide/index.md index 62eb19c2..1669fdf0 100644 --- a/docs/guide/index.md +++ b/docs/guide/index.md @@ -30,7 +30,8 @@ simdeck install /path/to/App.app simdeck install /path/to/App.ipa simdeck launch com.example.App simdeck tap --label "Continue" --wait-timeout-ms 5000 -simdeck describe --format agent --max-depth 3 +simdeck tap "Continue" +simdeck describe --format agent --max-depth 3 --interactive ``` Use `simdeck -d` for a detached background daemon, `simdeck -k` to stop it, and `simdeck -r` to restart it. diff --git a/docs/guide/quick-start.md b/docs/guide/quick-start.md index 51ef7f9e..547fb80d 100644 --- a/docs/guide/quick-start.md +++ b/docs/guide/quick-start.md @@ -66,7 +66,8 @@ Use selectors when you want automation to wait for UI state: ```sh simdeck tap --label "Continue" --wait-timeout-ms 5000 -simdeck describe --format agent --max-depth 3 +simdeck tap "Continue" +simdeck describe --format agent --max-depth 3 --interactive ``` ## 5. Keep It Running In The Background diff --git a/docs/inspector/index.md b/docs/inspector/index.md index ad62d75b..69925524 100644 --- a/docs/inspector/index.md +++ b/docs/inspector/index.md @@ -20,11 +20,12 @@ Use the built-in accessibility fallback for any app. Add an in-app inspector whe ```sh simdeck describe simdeck describe --format agent --max-depth 3 +simdeck describe --format agent --max-depth 3 --interactive simdeck describe --source native-ax simdeck describe --source react-native ``` -`auto` source selection uses the best available source and falls back to accessibility. +`auto` source selection uses the best available source and falls back to accessibility. Add `--interactive` or `-i` for a smaller agent-oriented tree of actionable elements plus ancestors. ## Use From The Browser diff --git a/packages/simdeck-test/dist/index.d.ts b/packages/simdeck-test/dist/index.d.ts index f7182b04..bc76a141 100644 --- a/packages/simdeck-test/dist/index.d.ts +++ b/packages/simdeck-test/dist/index.d.ts @@ -11,6 +11,7 @@ export type QueryOptions = { source?: "auto" | "nativescript" | "react-native" | "flutter" | "swiftui" | "uikit" | "native-ax" | "android-uiautomator"; maxDepth?: number; includeHidden?: boolean; + interactiveOnly?: boolean; }; export type ElementSelector = { text?: string; diff --git a/packages/simdeck-test/dist/index.js b/packages/simdeck-test/dist/index.js index 8b03c9d6..425c641d 100644 --- a/packages/simdeck-test/dist/index.js +++ b/packages/simdeck-test/dist/index.js @@ -524,6 +524,8 @@ function treeQuery(options = {}) { params.set("maxDepth", String(options.maxDepth)); if (options.includeHidden) params.set("includeHidden", "true"); + if (options.interactiveOnly) + params.set("interactiveOnly", "true"); return params.toString(); } function logsQuery(options = {}) { diff --git a/packages/simdeck-test/src/index.ts b/packages/simdeck-test/src/index.ts index f6c90acd..24bbe28a 100644 --- a/packages/simdeck-test/src/index.ts +++ b/packages/simdeck-test/src/index.ts @@ -28,6 +28,7 @@ export type QueryOptions = { | "android-uiautomator"; maxDepth?: number; includeHidden?: boolean; + interactiveOnly?: boolean; }; export type ElementSelector = { @@ -923,6 +924,7 @@ function treeQuery(options: QueryOptions = {}): string { if (options.maxDepth !== undefined) params.set("maxDepth", String(options.maxDepth)); if (options.includeHidden) params.set("includeHidden", "true"); + if (options.interactiveOnly) params.set("interactiveOnly", "true"); return params.toString(); } diff --git a/scripts/integration/cli.mjs b/scripts/integration/cli.mjs index bb100a27..fc116fae 100644 --- a/scripts/integration/cli.mjs +++ b/scripts/integration/cli.mjs @@ -189,6 +189,29 @@ async function main() { if (!agentTree.includes("source:") || !agentTree.includes("- ")) { throw new Error("agent describe output did not look like a hierarchy"); } + const interactiveTree = await measuredStep( + "server describe agent interactive", + () => + simdeckText([ + "describe", + simulatorUDID, + "--source", + "native-ax", + "--format", + "agent", + "--max-depth", + "8", + "--interactive", + ]), + ); + if ( + !interactiveTree.includes("source:") || + !interactiveTree.includes("Continue") + ) { + throw new Error( + "interactive agent describe did not include fixture controls", + ); + } await runRestControls(); await runCliControls(); @@ -482,6 +505,16 @@ async function runCliControls() { }, { expectFixture: true, expectText: "URL Opened" }, ); + await cliStep( + "CLI tap label shorthand", + ["tap", "Continue", "--wait-timeout-ms", "15000", "--duration-ms", "30"], + { + env: { SIMDECK_DEVICE: simulatorUDID }, + timeoutMs: 180_000, + maxElapsedMs: 60_000, + }, + { expectFixture: true, expectText: "Continue Tapped", attempts: 8 }, + ); await cliStep( "CLI tap fixture text field", [ @@ -497,7 +530,7 @@ async function runCliControls() { { timeoutMs: 180_000, maxElapsedMs: 60_000 }, { expectFixture: true, - expectText: "URL Opened", + expectText: "Continue Tapped", attempts: 6, delayMs: 1_500, }, @@ -623,6 +656,20 @@ async function runRestControls() { }, { phase: phaseSetup }, ); + await measuredStep( + "REST accessibility-tree interactive", + async () => { + const tree = await httpJson( + "GET", + `/api/simulators/${simulatorUDID}/accessibility-tree?source=native-ax&maxDepth=8&interactiveOnly=true`, + ); + assertRoots(tree, "REST accessibility-tree interactive"); + if (tree.interactiveOnly !== true) { + throw new Error("interactive tree did not report interactiveOnly=true"); + } + }, + { phase: phaseSetup }, + ); await measuredStep( "REST chrome-profile", async () => { @@ -1640,6 +1687,7 @@ function simdeckText(args, options = {}) { timeoutMs: options.timeoutMs ?? 120_000, maxElapsedMs: options.maxElapsedMs, input: options.input, + env: options.env, }); } @@ -1654,6 +1702,7 @@ function runText(command, args, options = {}) { cwd: root, encoding: "utf8", input: options.input, + env: options.env ? { ...process.env, ...options.env } : process.env, timeout: options.timeoutMs ?? 120_000, }); if (result.status !== 0) { diff --git a/server/src/accessibility.rs b/server/src/accessibility.rs new file mode 100644 index 00000000..1615a0fe --- /dev/null +++ b/server/src/accessibility.rs @@ -0,0 +1,212 @@ +use serde_json::{Map, Value}; + +pub fn interactive_accessibility_snapshot(snapshot: &Value) -> Value { + let mut output = snapshot.as_object().cloned().unwrap_or_default(); + let roots = snapshot + .get("roots") + .and_then(Value::as_array) + .map(|roots| { + roots + .iter() + .filter_map(interactive_accessibility_node) + .collect::>() + }) + .unwrap_or_default(); + + output.insert("roots".to_owned(), Value::Array(roots)); + output.insert("interactiveOnly".to_owned(), Value::Bool(true)); + Value::Object(output) +} + +fn interactive_accessibility_node(node: &Value) -> Option { + let object = node.as_object()?; + let children = node + .get("children") + .and_then(Value::as_array) + .map(|children| { + children + .iter() + .filter_map(interactive_accessibility_node) + .collect::>() + }) + .unwrap_or_default(); + + if !is_interactive_accessibility_node(node) && children.is_empty() { + return None; + } + + let mut output = object.clone(); + if children.is_empty() { + output.remove("children"); + } else { + output.insert("children".to_owned(), Value::Array(children)); + } + Some(Value::Object(output)) +} + +fn is_interactive_accessibility_node(node: &Value) -> bool { + if bool_field(node, &["hidden", "isHidden"]).unwrap_or(false) { + return false; + } + if numeric_field(node, &["alpha"]).is_some_and(|alpha| alpha <= 0.01) { + return false; + } + + if has_actionable_action(node) { + return true; + } + if bool_field( + node, + &[ + "clickable", + "focusable", + "isUserInteractionEnabled", + "scrollable", + "checked", + "selected", + ], + ) + .unwrap_or(false) + { + return true; + } + + string_field( + node, + &[ + "type", + "role", + "className", + "elementType", + "displayName", + "widgetType", + ], + ) + .is_some_and(|role| role_looks_interactive(&role)) +} + +fn has_actionable_action(node: &Value) -> bool { + for actions in [ + node.get("actions"), + node.get("custom_actions"), + node.get("control") + .and_then(|control| control.get("actions")), + ] + .into_iter() + .flatten() + { + if actions + .as_array() + .into_iter() + .flatten() + .filter_map(Value::as_str) + .any(action_looks_interactive) + { + return true; + } + } + false +} + +fn action_looks_interactive(action: &str) -> bool { + let action = action.trim().to_ascii_lowercase(); + !action.is_empty() + && !matches!( + action.as_str(), + "describe" | "getproperties" | "get_properties" | "highlight" + ) +} + +fn role_looks_interactive(role: &str) -> bool { + let role = role.to_ascii_lowercase(); + [ + "button", + "cell", + "checkbox", + "collection", + "combobox", + "control", + "edittext", + "link", + "menu", + "picker", + "radio", + "scroll", + "search", + "segmented", + "select", + "slider", + "stepper", + "switch", + "tab", + "table", + "textfield", + "text field", + "textinput", + "text input", + "toggle", + "webview", + ] + .iter() + .any(|needle| role.contains(needle)) +} + +fn bool_field(node: &Value, fields: &[&str]) -> Option { + fields.iter().find_map(|field| nested_bool(node, field)) +} + +fn numeric_field(node: &Value, fields: &[&str]) -> Option { + fields.iter().find_map(|field| nested_number(node, field)) +} + +fn string_field(node: &Value, fields: &[&str]) -> Option { + fields.iter().find_map(|field| nested_string(node, field)) +} + +fn nested_bool(node: &Value, field: &str) -> Option { + node.get(field) + .and_then(Value::as_bool) + .or_else(|| { + nested_object(node, "accessibility").and_then(|object| bool_from_map(object, field)) + }) + .or_else(|| nested_object(node, "control").and_then(|object| bool_from_map(object, field))) +} + +fn nested_number(node: &Value, field: &str) -> Option { + node.get(field) + .and_then(Value::as_f64) + .or_else(|| { + nested_object(node, "accessibility").and_then(|object| number_from_map(object, field)) + }) + .or_else(|| { + nested_object(node, "control").and_then(|object| number_from_map(object, field)) + }) +} + +fn nested_string(node: &Value, field: &str) -> Option { + node.get(field) + .and_then(Value::as_str) + .map(str::to_owned) + .or_else(|| { + nested_object(node, "accessibility").and_then(|object| string_from_map(object, field)) + }) + .or_else(|| { + nested_object(node, "control").and_then(|object| string_from_map(object, field)) + }) +} + +fn nested_object<'a>(node: &'a Value, field: &str) -> Option<&'a Map> { + node.get(field).and_then(Value::as_object) +} + +fn bool_from_map(object: &Map, field: &str) -> Option { + object.get(field).and_then(Value::as_bool) +} + +fn number_from_map(object: &Map, field: &str) -> Option { + object.get(field).and_then(Value::as_f64) +} + +fn string_from_map(object: &Map, field: &str) -> Option { + object.get(field).and_then(Value::as_str).map(str::to_owned) +} diff --git a/server/src/api/routes.rs b/server/src/api/routes.rs index 8a006c99..64527451 100644 --- a/server/src/api/routes.rs +++ b/server/src/api/routes.rs @@ -1,3 +1,4 @@ +use crate::accessibility::interactive_accessibility_snapshot; use crate::android::{self, AndroidBridge, AndroidEmulatorSpec}; use crate::api::json::json; use crate::auth; @@ -744,6 +745,7 @@ struct AccessibilityTreeQuery { source: Option, max_depth: Option, include_hidden: Option, + interactive_only: Option, } #[derive(Deserialize)] @@ -2595,6 +2597,7 @@ async fn accessibility_query( payload.source.as_deref(), payload.max_depth, payload.include_hidden.unwrap_or(false), + false, ) .await?; let matches = query_compact_elements( @@ -4075,6 +4078,7 @@ async fn accessibility_tree( query.source.as_deref(), query.max_depth, query.include_hidden.unwrap_or(false), + query.interactive_only.unwrap_or(false), ) .await?, )) @@ -4086,6 +4090,7 @@ async fn accessibility_tree_value( source: Option<&str>, max_depth: Option, include_hidden: bool, + interactive_only: bool, ) -> Result { if android::is_android_id(&udid) { let requested_source = source @@ -4099,6 +4104,9 @@ async fn accessibility_tree_value( if let Some(source) = requested_source { tree["requestedSource"] = Value::String(source); } + if interactive_only { + tree = interactive_accessibility_snapshot(&tree); + } Ok(tree) }) .await; @@ -4113,11 +4121,16 @@ async fn accessibility_tree_value( match accessibility_snapshot(state.clone(), udid.clone(), None, max_depth).await { Ok(snapshot) => snapshot, Err(error) => { - return Ok(empty_accessibility_tree( + let snapshot = empty_accessibility_tree( SOURCE_NATIVE_AX, &available_sources, suppress_native_ax_translation_error(&error.to_string()), - )); + ); + return Ok(if interactive_only { + interactive_accessibility_snapshot(&snapshot) + } else { + snapshot + }); } }; merge_connected_sources_for_pid( @@ -4128,6 +4141,11 @@ async fn accessibility_tree_value( ) .await; let snapshot = attach_available_sources(native_snapshot, &available_sources); + let snapshot = if interactive_only { + interactive_accessibility_snapshot(&snapshot) + } else { + snapshot + }; return Ok(snapshot); } @@ -4148,6 +4166,7 @@ async fn accessibility_tree_value( hierarchy_source, max_depth, include_hidden, + interactive_only, ) .await { @@ -4176,11 +4195,13 @@ async fn accessibility_tree_value( } else { None }; - Ok(attach_tree_metadata( - snapshot, - &available_sources, - fallback_reason, - )) + let snapshot = + attach_tree_metadata(snapshot, &available_sources, fallback_reason); + Ok(if interactive_only { + interactive_accessibility_snapshot(&snapshot) + } else { + snapshot + }) } Err(_inspector_error) => { let mut available_sources = available_sources_with_native_ax(Some(&session)); @@ -4191,6 +4212,7 @@ async fn accessibility_tree_value( InAppHierarchySource::Automatic, Some(0), include_hidden, + false, ) .await { @@ -4200,15 +4222,29 @@ async fn accessibility_tree_value( } match accessibility_snapshot(state.clone(), udid.clone(), None, max_depth).await { - Ok(native_snapshot) => Ok(attach_available_sources( - trim_tree_depth(native_snapshot, max_depth), - &available_sources, - )), - Err(native_ax_error) => Ok(empty_accessibility_tree( - SOURCE_NATIVE_AX, - &available_sources, - suppress_native_ax_translation_error(&native_ax_error.to_string()), - )), + Ok(native_snapshot) => { + let snapshot = attach_available_sources( + trim_tree_depth(native_snapshot, max_depth), + &available_sources, + ); + Ok(if interactive_only { + interactive_accessibility_snapshot(&snapshot) + } else { + snapshot + }) + } + Err(native_ax_error) => { + let snapshot = empty_accessibility_tree( + SOURCE_NATIVE_AX, + &available_sources, + suppress_native_ax_translation_error(&native_ax_error.to_string()), + ); + Ok(if interactive_only { + interactive_accessibility_snapshot(&snapshot) + } else { + snapshot + }) + } } } } @@ -4216,15 +4252,29 @@ async fn accessibility_tree_value( Err(_inspector_error) => { let available_sources = available_sources_with_native_ax(None); match accessibility_snapshot(state.clone(), udid.clone(), None, max_depth).await { - Ok(native_snapshot) => Ok(attach_available_sources( - trim_tree_depth(native_snapshot, max_depth), - &available_sources, - )), - Err(native_ax_error) => Ok(empty_accessibility_tree( - SOURCE_NATIVE_AX, - &available_sources, - suppress_native_ax_translation_error(&native_ax_error.to_string()), - )), + Ok(native_snapshot) => { + let snapshot = attach_available_sources( + trim_tree_depth(native_snapshot, max_depth), + &available_sources, + ); + Ok(if interactive_only { + interactive_accessibility_snapshot(&snapshot) + } else { + snapshot + }) + } + Err(native_ax_error) => { + let snapshot = empty_accessibility_tree( + SOURCE_NATIVE_AX, + &available_sources, + suppress_native_ax_translation_error(&native_ax_error.to_string()), + ); + Ok(if interactive_only { + interactive_accessibility_snapshot(&snapshot) + } else { + snapshot + }) + } } } } @@ -4298,6 +4348,7 @@ async fn perform_tap_payload( payload.source.as_deref(), payload.max_depth, payload.include_hidden.unwrap_or(false), + false, ) .await?; normalize_screen_point_from_snapshot(&snapshot, x, y)? @@ -4372,6 +4423,7 @@ async fn wait_for_absent_element_payload( payload.source.as_deref(), payload.max_depth, payload.include_hidden.unwrap_or(false), + false, ) .await?; if first_matching_element(&snapshot, &payload.selector).is_none() { @@ -4404,6 +4456,7 @@ async fn wait_for_snapshot_match( payload.source.as_deref(), payload.max_depth, payload.include_hidden.unwrap_or(false), + false, ) .await?; if first_matching_element(&snapshot, &payload.selector).is_some() { @@ -4433,6 +4486,7 @@ async fn scroll_until_visible_payload( payload.source.as_deref(), payload.max_depth, payload.include_hidden.unwrap_or(false), + false, ) .await?; if let Some(found) = first_matching_element(&snapshot, &payload.selector) { @@ -5033,6 +5087,7 @@ async fn run_batch_step(state: AppState, udid: String, step: BatchStep) -> Resul source.as_deref(), max_depth, include_hidden.unwrap_or(false), + false, ) .await?; Ok(json_value!({ @@ -6481,16 +6536,19 @@ async fn run_in_app_inspector_hierarchy( source: InAppHierarchySource, max_depth: Option, include_hidden: bool, + interactive_only: bool, ) -> Result { let max_depth = max_depth.unwrap_or(80); let params = match source { InAppHierarchySource::Automatic => json_value!({ "includeHidden": include_hidden, "maxDepth": max_depth, + "interactiveOnly": interactive_only, }), InAppHierarchySource::UIKit => json_value!({ "includeHidden": include_hidden, "maxDepth": max_depth, + "interactiveOnly": interactive_only, "source": "uikit", }), }; diff --git a/server/src/main.rs b/server/src/main.rs index 592b2055..ddbf7c06 100644 --- a/server/src/main.rs +++ b/server/src/main.rs @@ -1,3 +1,4 @@ +mod accessibility; mod android; mod api; mod auth; @@ -17,6 +18,7 @@ mod static_files; mod transport; mod webkit; +use accessibility::interactive_accessibility_snapshot; use anyhow::Context; use api::routes::{router, AppState}; use axum::Router; @@ -78,6 +80,13 @@ const SERVER_HEALTH_WATCHDOG_HTTP_FAILURE_THRESHOLD: usize = 3; struct Cli { #[arg(long, global = true, hide = true)] server_url: Option, + #[arg( + long, + global = true, + value_name = "SIMULATOR_NAME_OR_UDID", + help = "Default simulator for commands that can infer their target" + )] + device: Option, #[command(subcommand)] command: Command, } @@ -264,7 +273,7 @@ enum Command { }, #[command(name = "describe")] DescribeUi { - udid: String, + udid: Option, #[arg(long, value_parser = parse_point)] point: Option<(f64, f64)>, #[arg(long, value_enum, default_value_t = DescribeUiFormat::Json)] @@ -275,6 +284,8 @@ enum Command { max_depth: Option, #[arg(long)] include_hidden: bool, + #[arg(short = 'i', long = "interactive", visible_alias = "interactive-only")] + interactive_only: bool, #[arg(long)] direct: bool, }, @@ -294,9 +305,8 @@ enum Command { delay_ms: u64, }, Tap { - udid: String, - x: Option, - y: Option, + #[arg(value_name = "UDID_OR_TARGET", num_args = 0..)] + args: Vec, #[arg(long)] id: Option, #[arg(long)] @@ -2413,6 +2423,152 @@ fn list_studio_simulators(server_url: &str) -> anyhow::Result, + global_selector: Option<&str>, + explicit_server_url: Option<&str>, +) -> anyhow::Result { + if let Some(udid) = positional.map(str::trim).filter(|value| !value.is_empty()) { + return Ok(udid.to_owned()); + } + + let selector = global_selector + .map(str::to_owned) + .or_else(|| env::var("SIMDECK_DEVICE").ok()) + .or_else(|| env::var("SIMDECK_UDID").ok()) + .map(|value| value.trim().to_owned()) + .filter(|value| !value.is_empty()); + + if let Some(selector) = selector { + if android::is_android_id(&selector) || looks_like_device_selector(&selector) { + return Ok(selector); + } + let server_url = command_service_url(explicit_server_url)?; + if let Some(simulator) = select_studio_simulator(&server_url, &selector)? { + return Ok(simulator.udid); + } + return Ok(selector); + } + + let server_url = command_service_url(explicit_server_url)?; + if let Some(simulator) = infer_default_cli_simulator(&server_url)? { + return Ok(simulator.udid); + } + + let simulators = list_studio_simulators(&server_url)?; + let booted = simulators + .iter() + .filter(|simulator| simulator.is_booted) + .collect::>(); + if booted.len() > 1 { + anyhow::bail!( + "Multiple booted simulators are available. Pass a UDID, use --device, or set SIMDECK_DEVICE." + ); + } + if simulators.is_empty() { + anyhow::bail!("No simulators are available. Boot one or pass a UDID explicitly."); + } + anyhow::bail!( + "No default simulator could be inferred. Pass a UDID, use --device, or set SIMDECK_DEVICE." + ) +} + +fn infer_default_cli_simulator( + server_url: &str, +) -> anyhow::Result> { + let simulators = list_studio_simulators(server_url)?; + let booted = simulators + .iter() + .filter(|simulator| simulator.is_booted) + .cloned() + .collect::>(); + if booted.len() == 1 { + return Ok(booted.into_iter().next()); + } + if booted.is_empty() && simulators.len() == 1 { + return Ok(simulators.into_iter().next()); + } + Ok(None) +} + +fn parse_tap_command_args( + args: Vec, + id: Option, + label: Option, + value: Option, + element_type: Option, +) -> anyhow::Result { + let mut target = TapCommandTarget { + selector: ElementSelector { + id, + label, + value, + element_type, + }, + ..Default::default() + }; + + let args = args + .into_iter() + .map(|arg| arg.trim().to_owned()) + .filter(|arg| !arg.is_empty()) + .collect::>(); + + if !target.selector.is_empty() { + match args.as_slice() { + [] => return Ok(target), + [udid] => { + target.udid = Some(udid.clone()); + return Ok(target); + } + _ => anyhow::bail!( + "tap accepts at most one positional UDID when selector flags are used." + ), + } + } + + if args.is_empty() { + return Ok(target); + } + + let (udid, target_args) = if args.len() >= 2 && looks_like_device_selector(&args[0]) { + (Some(args[0].clone()), &args[1..]) + } else { + (None, args.as_slice()) + }; + target.udid = udid; + + if target_args.len() == 2 { + if let (Some(x), Some(y)) = ( + parse_f64_arg(&target_args[0]), + parse_f64_arg(&target_args[1]), + ) { + target.x = Some(x); + target.y = Some(y); + return Ok(target); + } + } + + if target_args.len() == 1 && parse_f64_arg(&target_args[0]).is_some() { + anyhow::bail!("tap requires both x and y coordinates."); + } + if target_args.iter().any(|arg| parse_f64_arg(arg).is_some()) { + anyhow::bail!("tap coordinates must be provided as exactly two numeric values."); + } + + target.selector.label = Some(target_args.join(" ")); + Ok(target) +} + +fn parse_f64_arg(value: &str) -> Option { + value.parse::().ok().filter(|value| value.is_finite()) +} + +fn looks_like_device_selector(value: &str) -> bool { + android::is_android_id(value) + || (value.len() == 36 && value.chars().all(|ch| ch.is_ascii_hexdigit() || ch == '-')) +} + fn studio_provider_bridge_script() -> anyhow::Result { let mut candidates = Vec::new(); if let Ok(root) = project_root() { @@ -2568,6 +2724,7 @@ fn main() -> anyhow::Result<()> { let cli = Cli::parse(); let explicit_server_url = cli.server_url.clone(); + let device_selector = cli.device.clone(); let service_url = explicit_server_url .clone() .or_else(|| env::var("SIMDECK_SERVER_URL").ok()) @@ -3165,9 +3322,19 @@ fn main() -> anyhow::Result<()> { source, max_depth, include_hidden, + interactive_only, direct, } => { - let service_url = command_service_url(explicit_server_url.as_deref())?; + let udid = resolve_cli_device_udid( + udid.as_deref(), + device_selector.as_deref(), + explicit_server_url.as_deref(), + )?; + let service_url = if direct { + String::new() + } else { + command_service_url(explicit_server_url.as_deref())? + }; let snapshot = describe_ui_snapshot( &bridge, &udid, @@ -3175,6 +3342,7 @@ fn main() -> anyhow::Result<()> { source, max_depth, include_hidden, + interactive_only, direct, &service_url, )?; @@ -3242,9 +3410,7 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::Tap { - udid, - x, - y, + args, id, label, value, @@ -3256,8 +3422,29 @@ fn main() -> anyhow::Result<()> { pre_delay_ms, post_delay_ms, } => { + let target = parse_tap_command_args(args, id, label, value, element_type)?; + let uses_inferred_device = target.udid.is_none(); + let uses_selector = !target.selector.is_empty(); + let udid = resolve_cli_device_udid( + target.udid.as_deref(), + device_selector.as_deref(), + explicit_server_url.as_deref(), + )?; + let x = target.x; + let y = target.y; + let ElementSelector { + id, + label, + value, + element_type, + } = target.selector; + let preferred_service_url = if uses_inferred_device || uses_selector { + Some(command_service_url(explicit_server_url.as_deref())?) + } else { + service_url.clone() + }; let command_server_url = - command_service_url_for_udid(&udid, &explicit_server_url, &service_url)?; + command_service_url_for_udid(&udid, &explicit_server_url, &preferred_service_url)?; if let (Some(server_url), Some(x), Some(y), true, None, None, None, None) = ( command_server_url.as_deref(), x, @@ -4001,7 +4188,7 @@ fn now_secs() -> u64 { .as_secs() } -#[derive(Clone, Debug, Default)] +#[derive(Clone, Debug, Default, PartialEq)] struct ElementSelector { id: Option, label: Option, @@ -4009,6 +4196,23 @@ struct ElementSelector { element_type: Option, } +impl ElementSelector { + fn is_empty(&self) -> bool { + self.id.is_none() + && self.label.is_none() + && self.value.is_none() + && self.element_type.is_none() + } +} + +#[derive(Clone, Debug, Default, PartialEq)] +struct TapCommandTarget { + udid: Option, + x: Option, + y: Option, + selector: ElementSelector, +} + #[derive(Clone, Copy, Debug)] struct GestureCoordinates { start_x: f64, @@ -4295,6 +4499,7 @@ fn describe_ui_snapshot( source: DescribeUiSource, max_depth: Option, include_hidden: bool, + interactive_only: bool, direct: bool, server_url: &str, ) -> anyhow::Result { @@ -4318,6 +4523,7 @@ fn describe_ui_snapshot( source, max_depth, include_hidden, + interactive_only, server_url, ) { Ok(snapshot) => return Ok(snapshot), @@ -4334,7 +4540,12 @@ fn describe_ui_snapshot( ); } - Ok(bridge.accessibility_snapshot_with_max_depth(udid, point, max_depth)?) + let snapshot = bridge.accessibility_snapshot_with_max_depth(udid, point, max_depth)?; + Ok(if interactive_only && point.is_none() { + interactive_accessibility_snapshot(&snapshot) + } else { + snapshot + }) } fn fetch_service_accessibility_tree( @@ -4342,6 +4553,7 @@ fn fetch_service_accessibility_tree( source: DescribeUiSource, max_depth: Option, include_hidden: bool, + interactive_only: bool, server_url: &str, ) -> anyhow::Result { let mut query = vec![format!("source={}", source.as_query_value())]; @@ -4351,6 +4563,9 @@ fn fetch_service_accessibility_tree( if include_hidden { query.push("includeHidden=true".to_owned()); } + if interactive_only { + query.push("interactiveOnly=true".to_owned()); + } let path = format!( "/api/simulators/{}/accessibility-tree?{}", url_path_component(udid), @@ -7368,16 +7583,17 @@ fn default_client_root() -> anyhow::Result { #[cfg(test)] mod tests { use super::{ - batch_line_to_json_step, daemon_matches_launch_options, http_url_for_host, is_tailscale_ip, - maestro_commands_from_flow, maestro_selector, normalize_accessibility_point_for_display, - parse_maestro_flow_yaml, parse_maestro_point, parse_workspace_daemon_process_line, - render_qr_code, run_maestro_command, server_health_watchdog_should_restart, - service_post_error_is_retryable, simdeck_pair_url, studio_daemon_restart_args, - workspace_daemon_process_is_current, Cli, Command, DaemonCommand, DaemonLaunchOptions, - DaemonMetadata, PairingAddress, ServiceCommand, StreamQualityProfileArg, - StudioExposeOptions, VideoCodecMode, WorkspaceDaemonProcess, YamlValue, - DEFAULT_LOCAL_STREAM_QUALITY_PROFILE, SERVER_HEALTH_WATCHDOG_FAILURE_THRESHOLD, - SERVER_HEALTH_WATCHDOG_HTTP_FAILURE_THRESHOLD, + batch_line_to_json_step, daemon_matches_launch_options, http_url_for_host, + interactive_accessibility_snapshot, is_tailscale_ip, maestro_commands_from_flow, + maestro_selector, normalize_accessibility_point_for_display, parse_maestro_flow_yaml, + parse_maestro_point, parse_tap_command_args, parse_workspace_daemon_process_line, + render_agent_accessibility_tree, render_qr_code, run_maestro_command, + server_health_watchdog_should_restart, service_post_error_is_retryable, simdeck_pair_url, + studio_daemon_restart_args, workspace_daemon_process_is_current, Cli, Command, + DaemonCommand, DaemonLaunchOptions, DaemonMetadata, ElementSelector, PairingAddress, + ServiceCommand, StreamQualityProfileArg, StudioExposeOptions, TapCommandTarget, + VideoCodecMode, WorkspaceDaemonProcess, YamlValue, DEFAULT_LOCAL_STREAM_QUALITY_PROFILE, + SERVER_HEALTH_WATCHDOG_FAILURE_THRESHOLD, SERVER_HEALTH_WATCHDOG_HTTP_FAILURE_THRESHOLD, }; use clap::Parser; use std::collections::HashMap; @@ -7772,6 +7988,113 @@ mod tests { )); } + #[test] + fn describe_interactive_flag_prunes_agent_tree_but_keeps_context() { + let parsed = + Cli::try_parse_from(["simdeck", "describe", "sim-1", "--format", "agent", "-i"]) + .unwrap(); + let Command::DescribeUi { + interactive_only, .. + } = parsed.command + else { + panic!("expected describe command"); + }; + assert!(interactive_only); + + let snapshot = serde_json::json!({ + "source": "native-ax", + "roots": [{ + "type": "Window", + "children": [{ + "type": "View", + "AXLabel": "Static wrapper", + "children": [{ + "type": "Button", + "AXLabel": "Continue", + "enabled": true, + "children": [] + }, { + "type": "Label", + "AXLabel": "Read only", + "children": [] + }] + }] + }] + }); + + let pruned = interactive_accessibility_snapshot(&snapshot); + let output = render_agent_accessibility_tree(&pruned); + + assert!(output.contains("- Window")); + assert!(output.contains("- View: Static wrapper")); + assert!(output.contains("- Button: Continue")); + assert!(!output.contains("Read only")); + } + + #[test] + fn tap_single_positional_arg_is_label_shorthand() { + let parsed = Cli::try_parse_from(["simdeck", "tap", "Continue"]).unwrap(); + let Command::Tap { args, .. } = parsed.command else { + panic!("expected tap command"); + }; + let target = parse_tap_command_args(args, None, None, None, None).unwrap(); + + assert_eq!( + target, + TapCommandTarget { + udid: None, + x: None, + y: None, + selector: ElementSelector { + label: Some("Continue".to_owned()), + ..Default::default() + } + } + ); + } + + #[test] + fn tap_legacy_udid_coordinates_still_parse() { + let udid = "00000000-0000-0000-0000-000000000001"; + let target = parse_tap_command_args( + vec![udid.to_owned(), "120".to_owned(), "240".to_owned()], + None, + None, + None, + None, + ) + .unwrap(); + + assert_eq!(target.udid.as_deref(), Some(udid)); + assert_eq!(target.x, Some(120.0)); + assert_eq!(target.y, Some(240.0)); + assert!(target.selector.is_empty()); + } + + #[test] + fn tap_legacy_udid_label_shorthand_still_parse() { + let udid = "00000000-0000-0000-0000-000000000001"; + let target = parse_tap_command_args( + vec![udid.to_owned(), "Continue".to_owned()], + None, + None, + None, + None, + ) + .unwrap(); + + assert_eq!(target.udid.as_deref(), Some(udid)); + assert_eq!(target.selector.label.as_deref(), Some("Continue")); + } + + #[test] + fn global_device_flag_is_available_for_agent_shortcuts() { + let parsed = + Cli::try_parse_from(["simdeck", "--device", "iPhone 16", "tap", "Continue"]).unwrap(); + + assert_eq!(parsed.device.as_deref(), Some("iPhone 16")); + } + #[test] fn batch_sleep_positional_duration_defaults_to_milliseconds() { let step = batch_line_to_json_step("sleep 500").unwrap(); diff --git a/skills/simdeck/SKILL.md b/skills/simdeck/SKILL.md index 45cf76a4..b0d59b70 100644 --- a/skills/simdeck/SKILL.md +++ b/skills/simdeck/SKILL.md @@ -47,7 +47,7 @@ If Browser Use is not available, only then use `simdeck ui --open` - it would op ## Device And App -Device commands take `` immediately after the command. +Most device commands take `` immediately after the command. For fast agent loops, `describe` and `tap` can infer the device from `--device`, `SIMDECK_DEVICE`, `SIMDECK_UDID`, or the only booted simulator. ```bash simdeck list @@ -82,6 +82,7 @@ Use targeted checks for test loops. `describe` is a diagnostic snapshot of the w ```bash simdeck describe simdeck describe --format agent --max-depth 4 +simdeck describe --format agent --max-depth 4 --interactive simdeck describe --format compact-json simdeck describe --point 120,240 simdeck describe --source auto @@ -99,6 +100,7 @@ simdeck assert --id login.button --source auto --max-depth 8 Use `--source auto` with the project daemon. Use `--direct` or `--source native-ax` for the private CoreSimulator accessibility bridge. Use `--source android-uiautomator` for Android emulator UIAutomator hierarchies. NativeScript, React Native, and Flutter inspector runtimes can add richer hierarchy data. For Android IDs, `describe` uses `uiautomator dump`; use `--format agent` or `--format compact-json` the same way as iOS. +Use `--interactive` or `-i` when an agent only needs controls and actionable framework nodes; SimDeck keeps ancestor context so the output is still navigable. Prefer selectors, coordinates only when needed. Selector taps go through the daemon and wait for the element server-side. @@ -106,6 +108,7 @@ Prefer selectors, coordinates only when needed. Selector taps go through the dae simdeck tap --id LoginButton --wait-timeout-ms 5000 simdeck tap --label "Continue" --element-type Button simdeck tap 120 240 +simdeck tap "Continue" ``` For persistent app integration tests, use `simdeck/test` instead of shelling out repeatedly: From cd2cfdf70163a56903d90b8ff26b55d409c3a56f Mon Sep 17 00:00:00 2001 From: DjDeveloperr Date: Fri, 22 May 2026 22:39:21 -0400 Subject: [PATCH 5/6] perf: speed up shallow accessibility describe --- AGENTS.md | 2 +- cli/XCWAccessibilityBridge.m | 5 ++++- 2 files changed, 5 insertions(+), 2 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index aab69475..4ca68e94 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -79,7 +79,7 @@ Private simulator behavior is implemented locally in: The current repo uses the private boot path, private display bridge, and private accessibility translation bridge directly. The browser streams frames from that bridge, injects touch and keyboard events through the same native session layer, inspects accessibility through `AccessibilityPlatformTranslation`, and renders device chrome from `cli/XCWChromeRenderer.*`. CoreSimulator service contexts resolve the active developer directory from `DEVELOPER_DIR`, then `xcode-select -p`, then `/Applications/Xcode.app/Contents/Developer`. The display bridge prefers direct CoreSimulator screen IOSurface callbacks and activates the SimulatorKit offscreen renderable view only if direct callbacks are unavailable. -Accessibility recovery may use simulator launchctl UIKit application state plus hit-tested translations to recover candidate foreground pids; the returned tree must still be rooted at tokenized `AXPTranslator` application objects, because `translationApplicationObjectForPid:` can omit the bridge delegate token after private display lifecycle changes. Full-tree snapshots merge those recovered roots with the private frontmost application translation. When multiple candidate application roots are discovered, serialize all of them in preferred order: non-extension app roots first, then largest translated roots, with `.appex`/PlugIns processes de-prioritized so SpringBoard and Safari app roots stay primary while widgets and WebContent roots remain debuggable. Widget renderer extension roots may report local frames; normalize those roots and children against matching SpringBoard widget placeholder frames before returning the snapshot. +Accessibility recovery may use simulator launchctl UIKit application state plus hit-tested translations to recover candidate foreground pids; the returned tree must still be rooted at tokenized `AXPTranslator` application objects, because `translationApplicationObjectForPid:` can omit the bridge delegate token after private display lifecycle changes. Full-tree snapshots merge those recovered roots with the private frontmost application translation. Shallow snapshots with `maxDepth <= 2` use the tokenized frontmost application translation directly when it is available, and only run the expensive recovery sweep if frontmost lookup fails, so agent-oriented describe loops avoid launchctl and hit-test recovery overhead. When multiple candidate application roots are discovered, serialize all of them in preferred order: non-extension app roots first, then largest translated roots, with `.appex`/PlugIns processes de-prioritized so SpringBoard and Safari app roots stay primary while widgets and WebContent roots remain debuggable. Widget renderer extension roots may report local frames; normalize those roots and children against matching SpringBoard widget placeholder frames before returning the snapshot. Physical chrome button support uses DeviceKit `chrome.json` input geometry for browser hit targets. Volume, action, mute, Apple Watch digital crown, Watch side button, and Watch left-side button dispatch through `IndigoHIDMessageForHIDArbitrary` with consumer/telephony/vendor HID usage pairs from the device chrome metadata; home, lock, and app-switcher remain on the existing SimulatorKit button paths. Apple Watch Digital Crown rotation dispatches through `IndigoHIDMessageForDigitalCrownEvent` when SimulatorKit exposes it, with `IndigoHIDMessageForScrollEvent(..., target=0x34)` as the fallback. tvOS simulators do not support direct screen touch; browser/API tap maps to Enter, swipe maps to arrow keys, and the native bridge rejects tvOS touch packets before they reach guest `SimulatorHID`. watchOS/tvOS skip dynamic pointer/mouse service warm-up because those guest runtimes abort on unsupported virtual services. Apple TV and Apple Watch simulators are fixed-orientation devices, so client and server rotation paths must not expose or dispatch device rotation for those families. Two-point multi-touch dispatch prefers the current SimulatorKit/Indigo packet constructor and falls back to SimDeck's manual Indigo packet adapter. On Xcode 26 SimulatorKit, the constructor expects pixel-space points and stable two-finger movement requires sending `LeftMouseDown` for both `began` and `moved`, then `LeftMouseUp` for `ended`/`cancelled`; using `LeftMouseDragged` for multi-touch moves only advances one contact in UIKit. Do not coalesce multi-touch move packets in the WebSocket or WebRTC control paths, because gesture recognizers need the intermediate two-contact samples. WebKit inspection uses the simulator `webinspectord` Unix socket named `com.apple.webinspectord_sim.socket` and WebKit's binary-plist Remote Inspector selectors. It lists only WebKit content that the runtime exposes as inspectable. For app-owned `WKWebView` on iOS 16.4 and newer, the app must set `isInspectable = true`. diff --git a/cli/XCWAccessibilityBridge.m b/cli/XCWAccessibilityBridge.m index 53a8b55a..96c4d5c6 100644 --- a/cli/XCWAccessibilityBridge.m +++ b/cli/XCWAccessibilityBridge.m @@ -1419,7 +1419,10 @@ + (nullable NSDictionary *)accessibilitySnapshotForSimulatorUDID:(NSString *)udi break; } } - if (pointValue == nil) { + // Shallow snapshots power fast agent describe loops. Keep the expensive + // multi-root recovery pass for full trees, or when frontmost lookup fails. + BOOL shouldRecoverRoots = pointValue == nil && (translation == nil || maxDepth > 2); + if (shouldRecoverRoots) { NSMutableDictionary *candidatesByKey = [NSMutableDictionary dictionary]; if (translation != nil) { NSMutableDictionary *candidate = XCWAXRootRecoveryCandidateFromTranslation(translator, translation, resolvedDisplayID ?: @0, token); From fcede525304c6b973aa40110cf0dabc33f3825cf Mon Sep 17 00:00:00 2001 From: DjDeveloperr Date: Fri, 22 May 2026 23:05:50 -0400 Subject: [PATCH 6/6] feat: add project default simulator selection --- README.md | 127 ++++---- docs/cli/commands.md | 129 ++++---- docs/cli/flags.md | 12 +- docs/cli/index.md | 32 +- docs/guide/index.md | 11 +- docs/guide/quick-start.md | 23 +- docs/guide/testing.md | 3 +- docs/guide/troubleshooting.md | 8 +- scripts/integration/cli.mjs | 52 ++- server/src/main.rs | 584 +++++++++++++++++++++++++++------- skills/simdeck/SKILL.md | 177 ++++++----- 11 files changed, 778 insertions(+), 380 deletions(-) diff --git a/README.md b/README.md index 2af1f404..d417021e 100644 --- a/README.md +++ b/README.md @@ -93,10 +93,11 @@ CLI commands automatically use the same warm daemon: ```sh simdeck list -simdeck tap 0.5 0.5 --normalized -simdeck describe --format agent --max-depth 2 -SIMDECK_DEVICE= simdeck tap "Continue" -simdeck --device describe --format agent --max-depth 2 --interactive +simdeck use +simdeck tap 0.5 0.5 --normalized +simdeck tap "Continue" +simdeck describe --format agent --max-depth 2 --interactive +simdeck --device describe --format agent --max-depth 2 ``` ## Daemon @@ -139,65 +140,72 @@ simdeck core-simulator shutdown ```sh simdeck list +simdeck use simdeck boot -simdeck shutdown -simdeck erase -simdeck install /path/to/App.app -simdeck install /path/to/App.ipa +simdeck shutdown +simdeck erase +simdeck install /path/to/App.app +simdeck install /path/to/App.ipa simdeck install android: /path/to/app.apk -simdeck uninstall com.example.App -simdeck open-url https://example.com -simdeck launch com.apple.Preferences -simdeck toggle-appearance -simdeck pasteboard set "hello" -simdeck pasteboard get -simdeck screenshot --output screen.png -simdeck screenshot --with-bezel --output screen-bezel.png -simdeck record --seconds 5 --output screen-recording.mp4 -simdeck stream --frames 120 > stream.h264 -simdeck describe -simdeck describe --format agent --max-depth 4 -simdeck describe --format agent --max-depth 4 --interactive -simdeck describe --point 120,240 -simdeck wait-for --label "Welcome" --timeout-ms 5000 -simdeck assert --id login.button --source auto --max-depth 8 -simdeck tap 120 240 -simdeck tap --label "Continue" --wait-timeout-ms 5000 +simdeck uninstall com.example.App +simdeck open-url https://example.com +simdeck launch com.apple.Preferences +simdeck toggle-appearance +simdeck pasteboard set "hello" +simdeck pasteboard get +simdeck screenshot --output screen.png +simdeck screenshot --with-bezel --output screen-bezel.png +simdeck record --seconds 5 --output screen-recording.mp4 +simdeck stream --frames 120 > stream.h264 +simdeck describe +simdeck describe --format agent --max-depth 4 +simdeck describe --format agent --max-depth 4 --interactive +simdeck describe --point 120,240 +simdeck wait-for --label "Welcome" --timeout-ms 5000 +simdeck assert --id login.button --source auto --max-depth 8 +simdeck tap 120 240 +simdeck tap --label "Continue" --wait-timeout-ms 5000 simdeck tap "Continue" -simdeck swipe 200 700 200 200 -simdeck gesture scroll-down -simdeck pinch --start-distance 160 --end-distance 80 -simdeck rotate-gesture --radius 100 --degrees 90 -simdeck touch 0.5 0.5 --phase began --normalized -simdeck touch 120 240 --down --up --delay-ms 800 -simdeck key enter -simdeck key-sequence --keycodes h,e,l,l,o -simdeck key-combo --modifiers cmd --key a -simdeck type "hello" -simdeck type --file message.txt -simdeck button lock --duration-ms 1000 -simdeck button volume-up -simdeck button action --duration-ms 1000 -simdeck button digital-crown -simdeck crown --delta 50 -simdeck button left-side-button -simdeck batch --step "tap --label Continue" --step "type 'hello'" --step "wait-for --label hello" -simdeck dismiss-keyboard -simdeck home -simdeck app-switcher -simdeck rotate-left -simdeck rotate-right -simdeck chrome-profile -simdeck logs --seconds 30 --limit 200 -simdeck processes -simdeck stats --watch -simdeck sample --seconds 3 +simdeck swipe 200 700 200 200 +simdeck gesture scroll-down +simdeck pinch --start-distance 160 --end-distance 80 +simdeck rotate-gesture --radius 100 --degrees 90 +simdeck touch 0.5 0.5 --phase began --normalized +simdeck touch 120 240 --down --up --delay-ms 800 +simdeck key enter +simdeck key-sequence --keycodes h,e,l,l,o +simdeck key-combo --modifiers cmd --key a +simdeck type "hello" +simdeck type --file message.txt +simdeck button lock --duration-ms 1000 +simdeck button volume-up +simdeck button action --duration-ms 1000 +simdeck button digital-crown +simdeck crown --delta 50 +simdeck button left-side-button +simdeck batch --step "tap --label Continue" --step "type 'hello'" --step "wait-for --label hello" +simdeck dismiss-keyboard +simdeck home +simdeck app-switcher +simdeck rotate-left +simdeck rotate-right +simdeck chrome-profile +simdeck logs --seconds 30 --limit 200 +simdeck processes +simdeck stats --watch +simdeck sample --seconds 3 ``` `simdeck list` defaults to compact JSON for agent-friendly device selection. Use `simdeck list --format json` for the full inventory with paths and display metadata. +`simdeck use ` stores a default simulator for the current project +directory. Most device commands accept `[]`; when it is omitted, SimDeck +uses `--device`, `SIMDECK_DEVICE`, `SIMDECK_UDID`, the saved project default, +or the only booted simulator, in that order. The old explicit-UDID form still +works for every command. + `boot` uses SimDeck's private CoreSimulator boot path so it can start devices without launching Simulator.app. If that private path is unavailable, the command returns the CoreSimulator error instead of falling back to @@ -219,10 +227,11 @@ Flutter, or UIKit in-app inspectors, then falls back to the built-in private CoreSimulator accessibility bridge. Use `--format agent` or `--format compact-json` for lower-token hierarchy dumps, and add `--interactive`/`-i` when an agent only -needs actionable elements plus their ancestors. `describe` and `tap` can infer a -target from `--device`, `SIMDECK_DEVICE`, `SIMDECK_UDID`, or the only booted -simulator. Coordinate commands accept screen coordinates from the accessibility -tree by default; pass `--normalized` to send `0.0..1.0` coordinates directly. +needs actionable elements plus their ancestors. Set a project default with +`simdeck use ` so agent commands can use short forms like +`simdeck tap "Continue"` and `simdeck describe --format agent --max-depth 2`. +Coordinate commands accept screen coordinates from the accessibility tree by +default; pass `--normalized` to send `0.0..1.0` coordinates directly. ## JS/TS Tests @@ -249,7 +258,7 @@ explicit UDID as the first argument when needed. Run common Maestro YAML flows against the same daemon-backed simulator API: ```sh -simdeck maestro test flow.yaml --artifacts-dir artifacts/maestro +simdeck maestro test flow.yaml --artifacts-dir artifacts/maestro ``` ## NativeScript Inspector diff --git a/docs/cli/commands.md b/docs/cli/commands.md index 1ed8a13e..cd51a173 100644 --- a/docs/cli/commands.md +++ b/docs/cli/commands.md @@ -43,98 +43,103 @@ rotate the token and pairing code, then restart the LaunchAgent. ```sh simdeck list simdeck list --format json +simdeck use simdeck boot -simdeck shutdown -simdeck erase +simdeck shutdown +simdeck erase ``` Android emulators appear as IDs such as `android:Pixel_8_API_36`. `list` defaults to compact JSON. Use `--format json` for the full simulator inventory, including paths and display metadata. +`simdeck use ` saves a default simulator for the current project +directory. After that, most device commands can omit ``; explicit UDIDs +still override the default. + ## Apps And URLs ```sh -simdeck install /path/to/App.app -simdeck install /path/to/App.ipa +simdeck install /path/to/App.app +simdeck install /path/to/App.ipa simdeck install android: /path/to/app.apk -simdeck uninstall com.example.App -simdeck launch com.example.App -simdeck open-url https://example.com -simdeck toggle-appearance +simdeck uninstall com.example.App +simdeck launch com.example.App +simdeck open-url https://example.com +simdeck toggle-appearance ``` ## Inspect UI ```sh -simdeck describe -simdeck describe --format agent --max-depth 4 -simdeck describe --format agent --max-depth 4 --interactive -simdeck describe --format compact-json -simdeck describe --source nativescript -simdeck describe --source react-native -simdeck describe --source flutter -simdeck describe --source uikit -simdeck describe --source native-ax -simdeck describe --point 120,240 -simdeck wait-for --label "Welcome" --timeout-ms 5000 -simdeck assert --id login.button --source auto --max-depth 8 +simdeck describe +simdeck describe --format agent --max-depth 4 +simdeck describe --format agent --max-depth 4 --interactive +simdeck describe --format compact-json +simdeck describe --source nativescript +simdeck describe --source react-native +simdeck describe --source flutter +simdeck describe --source uikit +simdeck describe --source native-ax +simdeck describe --point 120,240 +simdeck wait-for --label "Welcome" --timeout-ms 5000 +simdeck assert --id login.button --source auto --max-depth 8 ``` -Default source selection prefers a connected framework inspector, then the Swift in-app agent, then native accessibility. Use `--interactive` or `-i` to keep actionable elements and the ancestor context needed to find them. For quick agent loops, `describe` can infer the device from `--device`, `SIMDECK_DEVICE`, `SIMDECK_UDID`, or the only booted simulator. +Default source selection prefers a connected framework inspector, then the Swift in-app agent, then native accessibility. Use `--interactive` or `-i` to keep actionable elements and the ancestor context needed to find them. For quick agent loops, set the project default once and keep `describe` shallow. ## Performance ```sh -simdeck processes -simdeck stats -simdeck stats --pid 12345 -simdeck stats --watch -simdeck sample -simdeck sample --pid 12345 --seconds 3 +simdeck processes +simdeck stats +simdeck stats --pid 12345 +simdeck stats --watch +simdeck sample +simdeck sample --pid 12345 --seconds 3 ``` Performance data is simulator-only and uses host-process telemetry for matching app, extension, helper, and web-content PIDs. `stats` reports CPU, memory, disk write rate, network receive/send rates, connection count, hang state, and recent crash or termination signals. `sample` captures a short macOS `sample` report for the selected or foreground app process. ## Input -Coordinates are screen points unless `--normalized` is present. `tap "Continue"` is shorthand for a label tap on the inferred device. Use `--device ` or `SIMDECK_DEVICE=` when more than one simulator is booted. +Coordinates are screen points unless `--normalized` is present. `tap "Continue"` is shorthand for a label tap on the selected device. Use `--device ` or `SIMDECK_DEVICE=` for one-off overrides. ```sh -simdeck tap 120 240 -simdeck tap 0.5 0.5 --normalized -simdeck tap --label "Continue" --wait-timeout-ms 5000 +simdeck tap 120 240 +simdeck tap 0.5 0.5 --normalized +simdeck tap --label "Continue" --wait-timeout-ms 5000 simdeck tap "Continue" -simdeck swipe 200 700 200 200 -simdeck gesture scroll-down -simdeck pinch --start-distance 160 --end-distance 80 -simdeck rotate-gesture --radius 100 --degrees 90 -simdeck type "hello" -simdeck type --file message.txt -simdeck key enter -simdeck key-sequence --keycodes h,e,l,l,o -simdeck key-combo --modifiers cmd --key a +simdeck swipe 200 700 200 200 +simdeck gesture scroll-down +simdeck pinch --start-distance 160 --end-distance 80 +simdeck rotate-gesture --radius 100 --degrees 90 +simdeck type "hello" +simdeck type --file message.txt +simdeck key enter +simdeck key-sequence --keycodes h,e,l,l,o +simdeck key-combo --modifiers cmd --key a ``` System controls: ```sh -simdeck button lock --duration-ms 1000 -simdeck button volume-up -simdeck button action -simdeck button digital-crown -simdeck crown --delta 50 -simdeck dismiss-keyboard -simdeck home -simdeck app-switcher -simdeck rotate-left -simdeck rotate-right +simdeck button lock --duration-ms 1000 +simdeck button volume-up +simdeck button action +simdeck button digital-crown +simdeck crown --delta 50 +simdeck dismiss-keyboard +simdeck home +simdeck app-switcher +simdeck rotate-left +simdeck rotate-right ``` ## Batch ```sh -simdeck batch \ +simdeck batch \ --step "tap --label Continue --wait-timeout-ms 5000" \ --step "type 'hello world'" \ --step "wait-for --label 'hello world' --timeout-ms 5000" @@ -147,7 +152,7 @@ Use `wait-for` or `assert` steps instead of fixed sleeps when possible. Run common Maestro flows through SimDeck's daemon-backed iOS Simulator API: ```sh -simdeck maestro test flow.yaml --artifacts-dir artifacts/maestro +simdeck maestro test flow.yaml --artifacts-dir artifacts/maestro ``` The compatibility runner supports the core local commands: `launchApp`, `openLink`, `tapOn`, `inputText`, `eraseText`, `pressKey`, `assertVisible`, `assertNotVisible`, `scrollUntilVisible`, `swipe`, `takeScreenshot`, and `waitForAnimationToEnd`. @@ -155,21 +160,21 @@ The compatibility runner supports the core local commands: `launchApp`, `openLin ## Evidence ```sh -simdeck screenshot --output screen.png -simdeck screenshot --with-bezel --output screen-bezel.png -simdeck screenshot --stdout > screen.png -simdeck record --seconds 5 --output screen-recording.mp4 -simdeck record --seconds 5 --stdout > screen-recording.mp4 -simdeck pasteboard set "hello" -simdeck pasteboard get -simdeck logs --seconds 30 --limit 200 -simdeck chrome-profile +simdeck screenshot --output screen.png +simdeck screenshot --with-bezel --output screen-bezel.png +simdeck screenshot --stdout > screen.png +simdeck record --seconds 5 --output screen-recording.mp4 +simdeck record --seconds 5 --stdout > screen-recording.mp4 +simdeck pasteboard set "hello" +simdeck pasteboard get +simdeck logs --seconds 30 --limit 200 +simdeck chrome-profile ``` Diagnostic iOS H.264 stream: ```sh -simdeck stream --frames 120 > stream.h264 +simdeck stream --frames 120 > stream.h264 ``` ## Studio And Providers diff --git a/docs/cli/flags.md b/docs/cli/flags.md index e59c3a87..2adc1d0f 100644 --- a/docs/cli/flags.md +++ b/docs/cli/flags.md @@ -10,10 +10,14 @@ simdeck daemon start --help ## Global -| Flag | Env | Purpose | -| --------------------- | -------------------- | ---------------------------------------------- | -| `--server-url ` | `SIMDECK_SERVER_URL` | Target a specific running daemon | -| `--device ` | `SIMDECK_DEVICE` | Default simulator for inferred-device commands | +| Flag | Env | Purpose | +| --------------------- | -------------------- | -------------------------------- | +| `--server-url ` | `SIMDECK_SERVER_URL` | Target a specific running daemon | +| `--device ` | `SIMDECK_DEVICE` | One-off simulator override | + +`SIMDECK_UDID` is also accepted for compatibility. Device commands resolve in +this order: positional UDID, `--device`, `SIMDECK_DEVICE`, `SIMDECK_UDID`, the +project default from `simdeck use `, then auto-inference from the daemon. ## Server Options diff --git a/docs/cli/index.md b/docs/cli/index.md index ff53a86e..9d0e32a5 100644 --- a/docs/cli/index.md +++ b/docs/cli/index.md @@ -21,6 +21,11 @@ simdeck [SIMULATOR_NAME_OR_UDID] simdeck [--server-url ] [options] ``` +Use `simdeck use ` once per project directory to make that simulator the +default for later device commands. Most commands accept `[]`; `--device`, +`SIMDECK_DEVICE`, and `SIMDECK_UDID` override the saved project default when a +one-off target is needed. + Use `--server-url` or `SIMDECK_SERVER_URL` when a script should target a specific daemon: ```sh @@ -31,23 +36,24 @@ SIMDECK_SERVER_URL=http://127.0.0.1:4310 simdeck list ```sh simdeck list +simdeck use simdeck boot -simdeck install /path/to/App.app -simdeck install /path/to/App.ipa -simdeck launch com.example.App -simdeck open-url https://example.com -simdeck tap --label "Continue" --wait-timeout-ms 5000 +simdeck install /path/to/App.app +simdeck install /path/to/App.ipa +simdeck launch com.example.App +simdeck open-url https://example.com +simdeck tap --label "Continue" --wait-timeout-ms 5000 simdeck tap "Continue" -simdeck describe --format agent --max-depth 3 --interactive -simdeck screenshot --output screen.png -simdeck screenshot --with-bezel --output screen-bezel.png -simdeck record --seconds 5 --output screen-recording.mp4 -simdeck logs --seconds 30 --limit 200 -simdeck stats -simdeck sample --seconds 3 +simdeck describe --format agent --max-depth 3 --interactive +simdeck screenshot --output screen.png +simdeck screenshot --with-bezel --output screen-bezel.png +simdeck record --seconds 5 --output screen-recording.mp4 +simdeck logs --seconds 30 --limit 200 +simdeck stats +simdeck sample --seconds 3 ``` -`tap` and `describe` can infer their simulator from `--device`, `SIMDECK_DEVICE`, `SIMDECK_UDID`, or the only booted simulator. +The explicit form still works, for example `simdeck launch com.example.App`. Most successful commands print JSON so they can be piped into tools such as `jq`. diff --git a/docs/guide/index.md b/docs/guide/index.md index 1669fdf0..b63bd291 100644 --- a/docs/guide/index.md +++ b/docs/guide/index.md @@ -25,13 +25,14 @@ Open the local URL, pick a device, and use the toolbar or CLI commands: ```sh simdeck list +simdeck use simdeck boot -simdeck install /path/to/App.app -simdeck install /path/to/App.ipa -simdeck launch com.example.App -simdeck tap --label "Continue" --wait-timeout-ms 5000 +simdeck install /path/to/App.app +simdeck install /path/to/App.ipa +simdeck launch com.example.App +simdeck tap --label "Continue" --wait-timeout-ms 5000 simdeck tap "Continue" -simdeck describe --format agent --max-depth 3 --interactive +simdeck describe --format agent --max-depth 3 --interactive ``` Use `simdeck -d` for a detached background daemon, `simdeck -k` to stop it, and `simdeck -r` to restart it. diff --git a/docs/guide/quick-start.md b/docs/guide/quick-start.md index 547fb80d..a4a2a995 100644 --- a/docs/guide/quick-start.md +++ b/docs/guide/quick-start.md @@ -31,18 +31,21 @@ The UI lists available iOS Simulators and Android emulators. You can also use th ```sh simdeck list +simdeck use simdeck boot ``` -Android emulator IDs are prefixed with `android:`. +`simdeck use ` saves the simulator default for this project directory so +later device commands can omit the UDID. Android emulator IDs are prefixed with +`android:`. ## 3. Install And Launch An App ```sh -simdeck install /path/to/App.app -simdeck install /path/to/App.ipa -simdeck launch com.example.App -simdeck open-url myapp://debug +simdeck install /path/to/App.app +simdeck install /path/to/App.ipa +simdeck launch com.example.App +simdeck open-url myapp://debug ``` For Android: @@ -57,17 +60,17 @@ simdeck launch android: com.example.app Use coordinates when you know them: ```sh -simdeck tap 120 240 -simdeck swipe 200 700 200 200 -simdeck type "hello" +simdeck tap 120 240 +simdeck swipe 200 700 200 200 +simdeck type "hello" ``` Use selectors when you want automation to wait for UI state: ```sh -simdeck tap --label "Continue" --wait-timeout-ms 5000 +simdeck tap --label "Continue" --wait-timeout-ms 5000 simdeck tap "Continue" -simdeck describe --format agent --max-depth 3 --interactive +simdeck describe --format agent --max-depth 3 --interactive ``` ## 5. Keep It Running In The Background diff --git a/docs/guide/testing.md b/docs/guide/testing.md index 930353df..92959d28 100644 --- a/docs/guide/testing.md +++ b/docs/guide/testing.md @@ -51,7 +51,8 @@ Selectors can match `text`, `id`, `label`, `value`, `type`, `index`, `enabled`, The CLI includes a compatibility runner for common Maestro YAML flows: ```sh -simdeck maestro test flow.yaml --artifacts-dir artifacts/maestro +simdeck use +simdeck maestro test flow.yaml --artifacts-dir artifacts/maestro ``` Supported commands include `launchApp`, `openLink`, `tapOn`, `inputText`, `eraseText`, `pressKey`, `assertVisible`, `assertNotVisible`, `scrollUntilVisible`, `swipe`, `takeScreenshot`, and `waitForAnimationToEnd`. Unsupported Maestro commands fail clearly so the flow can be adjusted or the compatibility layer can be expanded. diff --git a/docs/guide/troubleshooting.md b/docs/guide/troubleshooting.md index 3a32fc6f..503b8c19 100644 --- a/docs/guide/troubleshooting.md +++ b/docs/guide/troubleshooting.md @@ -161,10 +161,10 @@ The fallback is expected when no in-app inspector is available. Check: Use a forced source to see the failure reason: ```sh -simdeck describe --source nativescript -simdeck describe --source react-native -simdeck describe --source flutter -simdeck describe --source uikit +simdeck describe --source nativescript +simdeck describe --source react-native +simdeck describe --source flutter +simdeck describe --source uikit ``` ### NativeScript inspector does not connect diff --git a/scripts/integration/cli.mjs b/scripts/integration/cli.mjs index fc116fae..1f888fb5 100644 --- a/scripts/integration/cli.mjs +++ b/scripts/integration/cli.mjs @@ -137,20 +137,26 @@ async function main() { await measuredStep("CLI list", () => assertSimulatorListed(simulatorUDID), { phase: phaseSetup, }); + await measuredStep( + "CLI use default simulator", + () => { + const selection = simdeckJson(["use", simulatorUDID]); + if (selection.udid !== simulatorUDID) { + throw new Error(`simdeck use selected ${JSON.stringify(selection)}`); + } + }, + { phase: phaseSetup }, + ); await measuredStep( "CLI chrome-profile", - () => - assertJson( - simdeckJson(["chrome-profile", simulatorUDID]), - "chrome-profile", - ), + () => assertJson(simdeckJson(["chrome-profile"]), "chrome-profile"), { phase: phaseSetup }, ); await measuredStep( "CLI logs", () => assertJson( - simdeckJson(["logs", simulatorUDID, "--seconds", "1", "--limit", "1"]), + simdeckJson(["logs", "--seconds", "1", "--limit", "1"]), "logs", ), { phase: phaseSetup }, @@ -159,7 +165,7 @@ async function main() { await measuredStep( "CLI install fixture", async () => { - simdeckJson(["install", simulatorUDID, fixture.appPath]); + simdeckJson(["install", fixture.appPath]); preapproveFixtureUrlScheme(); }, { phase: phaseSetup }, @@ -177,7 +183,6 @@ async function main() { const agentTree = await measuredStep("server describe agent", () => simdeckText([ "describe", - simulatorUDID, "--source", "native-ax", "--format", @@ -194,7 +199,6 @@ async function main() { () => simdeckText([ "describe", - simulatorUDID, "--source", "native-ax", "--format", @@ -219,7 +223,7 @@ async function main() { await measuredStep( "CLI screenshot file", async () => { - simdeckJson(["screenshot", simulatorUDID, "--output", screenshotPath]); + simdeckJson(["screenshot", "--output", screenshotPath]); assertPng(screenshotPath); }, { phase: phaseCommandSmoke }, @@ -232,10 +236,11 @@ async function main() { stdoutPng, runBuffer( simdeck, - ["--server-url", serverUrl, "screenshot", simulatorUDID, "--stdout"], + ["--server-url", serverUrl, "screenshot", "--stdout"], { timeoutMs: 300_000, maxBuffer: 64 * 1024 * 1024, + env: { HOME: tempRoot }, }, ), ); @@ -249,7 +254,6 @@ async function main() { async () => { simdeckJson([ "screenshot", - simulatorUDID, "--with-bezel", "--output", bezeledScreenshotPath, @@ -262,14 +266,7 @@ async function main() { await measuredStep( "CLI screen recording", async () => { - simdeckJson([ - "record", - simulatorUDID, - "--seconds", - "1", - "--output", - recordingPath, - ]); + simdeckJson(["record", "--seconds", "1", "--output", recordingPath]); assertMp4(recordingPath); }, { phase: phaseCommandSmoke }, @@ -278,14 +275,14 @@ async function main() { await measuredStep( "CLI pasteboard set", async () => { - simdeckJson(["pasteboard", "set", simulatorUDID, "simdeck integration"]); + simdeckJson(["pasteboard", "set", "simdeck integration"]); }, { phase: phaseCommandSmoke }, ); await measuredStep( "CLI pasteboard get", async () => { - const pasteboard = simdeckJson(["pasteboard", "get", simulatorUDID]); + const pasteboard = simdeckJson(["pasteboard", "get"]); if (pasteboard.text !== "simdeck integration") { throw new Error( `pasteboard round-trip failed: ${JSON.stringify(pasteboard)}`, @@ -300,14 +297,14 @@ async function main() { await measuredStep( "CLI type file", async () => { - simdeckJson(["type", simulatorUDID, "--file", fileInput]); + simdeckJson(["type", "--file", fileInput]); }, { phase: phaseCommandSmoke }, ); await measuredStep( "CLI type stdin", async () => { - simdeckJson(["type", simulatorUDID, "--stdin"], { + simdeckJson(["type", "--stdin"], { input: "stdin input", }); }, @@ -319,7 +316,6 @@ async function main() { async () => { const batch = simdeckJson([ "batch", - simulatorUDID, "--step", "button home", "--step", @@ -339,7 +335,7 @@ async function main() { await measuredStep( "CLI uninstall fixture", - () => simdeckJson(["uninstall", simulatorUDID, fixtureBundleId]), + () => simdeckJson(["uninstall", fixtureBundleId]), { phase: phaseSimulatorLifecycle }, ); await measuredStep( @@ -509,7 +505,6 @@ async function runCliControls() { "CLI tap label shorthand", ["tap", "Continue", "--wait-timeout-ms", "15000", "--duration-ms", "30"], { - env: { SIMDECK_DEVICE: simulatorUDID }, timeoutMs: 180_000, maxElapsedMs: 60_000, }, @@ -1687,7 +1682,7 @@ function simdeckText(args, options = {}) { timeoutMs: options.timeoutMs ?? 120_000, maxElapsedMs: options.maxElapsedMs, input: options.input, - env: options.env, + env: { HOME: tempRoot, ...(options.env ?? {}) }, }); } @@ -1728,6 +1723,7 @@ function runBuffer(command, args, options = {}) { cwd: root, encoding: "buffer", maxBuffer: options.maxBuffer ?? 16 * 1024 * 1024, + env: options.env ? { ...process.env, ...options.env } : process.env, timeout: options.timeoutMs ?? 120_000, }); if (result.status !== 0) { diff --git a/server/src/main.rs b/server/src/main.rs index ddbf7c06..d6635633 100644 --- a/server/src/main.rs +++ b/server/src/main.rs @@ -84,7 +84,7 @@ struct Cli { long, global = true, value_name = "SIMULATOR_NAME_OR_UDID", - help = "Default simulator for commands that can infer their target" + help = "Override the simulator target for this command" )] device: Option, #[command(subcommand)] @@ -190,50 +190,54 @@ enum Command { #[arg(long, value_enum, default_value_t = ListFormat::CompactJson)] format: ListFormat, }, - Boot { + Use { + #[arg(value_name = "UDID")] udid: String, }, + Boot { + udid: Option, + }, Shutdown { - udid: String, + udid: Option, }, OpenUrl { - udid: String, - url: String, + #[arg(value_name = "UDID_OR_URL", num_args = 1..=2)] + args: Vec, }, Launch { - udid: String, - bundle_id: String, + #[arg(value_name = "UDID_OR_BUNDLE_ID", num_args = 1..=2)] + args: Vec, }, ToggleAppearance { - udid: String, + udid: Option, }, Erase { - udid: String, + udid: Option, }, Install { - udid: String, - app_path: String, + #[arg(value_name = "UDID_OR_APP_PATH", num_args = 1..=2)] + args: Vec, }, Uninstall { - udid: String, - bundle_id: String, + #[arg(value_name = "UDID_OR_BUNDLE_ID", num_args = 1..=2)] + args: Vec, }, Pasteboard { #[command(subcommand)] command: PasteboardCommand, }, Logs { - udid: String, + udid: Option, #[arg(long, default_value_t = 30.0)] seconds: f64, #[arg(long, default_value_t = 200)] limit: usize, }, Processes { - udid: String, + udid: Option, }, Stats { - udid: String, + udid: Option, #[arg(long)] pid: Option, #[arg(long)] @@ -242,14 +246,14 @@ enum Command { interval: f64, }, Sample { - udid: String, + udid: Option, #[arg(long)] pid: Option, #[arg(long, default_value_t = 3)] seconds: u64, }, Screenshot { - udid: String, + udid: Option, #[arg(short, long)] output: Option, #[arg(long)] @@ -258,7 +262,7 @@ enum Command { with_bezel: bool, }, Record { - udid: String, + udid: Option, #[arg(short, long)] output: Option, #[arg(long)] @@ -267,7 +271,7 @@ enum Command { seconds: f64, }, Stream { - udid: String, + udid: Option, #[arg(long, default_value_t = 0)] frames: u64, }, @@ -290,9 +294,8 @@ enum Command { direct: bool, }, Touch { - udid: String, - x: f64, - y: f64, + #[arg(value_name = "UDID_OR_POINT", num_args = 2..=3)] + args: Vec, #[arg(long, default_value = "began")] phase: String, #[arg(long)] @@ -329,7 +332,7 @@ enum Command { post_delay_ms: u64, }, WaitFor { - udid: String, + udid: Option, #[command(flatten)] selector: SelectorArgs, #[arg(long, value_enum, default_value_t = DescribeUiSource::Auto)] @@ -344,7 +347,7 @@ enum Command { poll_interval_ms: u64, }, Assert { - udid: String, + udid: Option, #[command(flatten)] selector: SelectorArgs, #[arg(long, value_enum, default_value_t = DescribeUiSource::Auto)] @@ -359,11 +362,8 @@ enum Command { poll_interval_ms: u64, }, Swipe { - udid: String, - start_x: f64, - start_y: f64, - end_x: f64, - end_y: f64, + #[arg(value_name = "UDID_OR_POINTS", num_args = 4..=5)] + args: Vec, #[arg(long)] normalized: bool, #[arg(long, default_value_t = 350)] @@ -376,8 +376,8 @@ enum Command { post_delay_ms: u64, }, Gesture { - udid: String, - preset: String, + #[arg(value_name = "UDID_OR_PRESET", num_args = 1..=2)] + args: Vec, #[arg(long)] screen_width: Option, #[arg(long)] @@ -394,9 +394,8 @@ enum Command { post_delay_ms: u64, }, Pinch { - udid: String, - center_x: Option, - center_y: Option, + #[arg(value_name = "UDID_OR_CENTER", num_args = 0..=3)] + args: Vec, #[arg(long, default_value_t = 160.0)] start_distance: f64, #[arg(long, default_value_t = 80.0)] @@ -411,9 +410,8 @@ enum Command { steps: u32, }, RotateGesture { - udid: String, - center_x: Option, - center_y: Option, + #[arg(value_name = "UDID_OR_CENTER", num_args = 0..=3)] + args: Vec, #[arg(long, default_value_t = 100.0)] radius: f64, #[arg(long, default_value_t = 90.0)] @@ -426,8 +424,8 @@ enum Command { steps: u32, }, Key { - udid: String, - key: String, + #[arg(value_name = "UDID_OR_KEY", num_args = 1..=2)] + args: Vec, #[arg(long, default_value_t = 0)] modifiers: u32, #[arg(long, default_value_t = 0)] @@ -438,22 +436,22 @@ enum Command { post_delay_ms: u64, }, KeySequence { - udid: String, + udid: Option, #[arg(long = "keycodes", alias = "keys")] keycodes: String, #[arg(long, default_value_t = 100)] delay_ms: u64, }, KeyCombo { - udid: String, + udid: Option, #[arg(long)] modifiers: String, #[arg(long)] key: String, }, Type { - udid: String, - text: Option, + #[arg(value_name = "UDID_OR_TEXT", num_args = 0..=2)] + args: Vec, #[arg(long)] stdin: bool, #[arg(long)] @@ -462,18 +460,18 @@ enum Command { delay_ms: u64, }, Button { - udid: String, - button: String, + #[arg(value_name = "UDID_OR_BUTTON", num_args = 1..=2)] + args: Vec, #[arg(long, default_value_t = 0)] duration_ms: u32, }, Crown { - udid: String, + udid: Option, #[arg(long, default_value_t = 50.0)] delta: f64, }, Batch { - udid: String, + udid: Option, #[arg(long = "step")] steps: Vec, #[arg(long)] @@ -484,22 +482,22 @@ enum Command { continue_on_error: bool, }, DismissKeyboard { - udid: String, + udid: Option, }, Home { - udid: String, + udid: Option, }, AppSwitcher { - udid: String, + udid: Option, }, RotateLeft { - udid: String, + udid: Option, }, RotateRight { - udid: String, + udid: Option, }, ChromeProfile { - udid: String, + udid: Option, }, } @@ -639,8 +637,8 @@ enum ProviderCommand { #[derive(Subcommand)] enum MaestroCommand { Test { - udid: String, - flow: PathBuf, + #[arg(value_name = "UDID_OR_FLOW", num_args = 1..=2)] + args: Vec, #[arg(long)] artifacts_dir: Option, #[arg(long)] @@ -723,11 +721,11 @@ enum CoreSimulatorCommand { #[derive(Subcommand)] enum PasteboardCommand { Get { - udid: String, + udid: Option, }, Set { - udid: String, - text: Option, + #[arg(value_name = "UDID_OR_TEXT", num_args = 0..=2)] + args: Vec, #[arg(long)] stdin: bool, #[arg(long)] @@ -878,6 +876,18 @@ struct DaemonMetadata { local_stream_fps: Option, } +#[derive(Clone, Debug, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +struct ProjectDeviceSelection { + project_root: PathBuf, + udid: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + name: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + runtime_name: Option, + selected_at: u64, +} + fn default_daemon_port() -> u16 { 4310 } @@ -1807,6 +1817,47 @@ fn daemon_log_path_for_root(root: &Path) -> anyhow::Result { .join(format!("{:016x}.log", hasher.finish()))) } +fn read_project_device_selection() -> anyhow::Result> { + let root = project_root()?; + let path = project_device_selection_path_for_root(&root)?; + if !path.exists() { + return Ok(None); + } + let data = fs::read_to_string(&path).with_context(|| format!("read {}", path.display()))?; + let selection = serde_json::from_str::(&data) + .with_context(|| format!("parse simulator selection {}", path.display()))?; + if selection.project_root != root { + return Ok(None); + } + Ok(Some(selection)) +} + +fn write_project_device_selection(selection: &ProjectDeviceSelection) -> anyhow::Result { + let path = project_device_selection_path_for_root(&selection.project_root)?; + if let Some(parent) = path.parent() { + fs::create_dir_all(parent)?; + } + fs::write(&path, serde_json::to_vec_pretty(selection)?) + .with_context(|| format!("write {}", path.display()))?; + Ok(path) +} + +fn project_device_selection_path_for_root(root: &Path) -> anyhow::Result { + let mut hasher = DefaultHasher::new(); + root.to_string_lossy().hash(&mut hasher); + Ok(simdeck_user_state_dir() + .join("default-devices") + .join(format!("{:016x}.json", hasher.finish()))) +} + +fn simdeck_user_state_dir() -> PathBuf { + env::var_os("HOME") + .filter(|value| !value.is_empty()) + .map(PathBuf::from) + .map(|home| home.join(".simdeck")) + .unwrap_or_else(|| env::temp_dir().join("simdeck")) +} + fn daemon_metadata_paths() -> anyhow::Result> { let dir = env::temp_dir().join("simdeck"); if !dir.exists() { @@ -1908,6 +1959,7 @@ fn is_known_command(value: &str) -> bool { | "core-simulator" | "simctl-service" | "list" + | "use" | "boot" | "shutdown" | "open-url" @@ -1948,7 +2000,15 @@ fn is_known_command(value: &str) -> bool { fn run_no_command_action(action: NoCommandAction) -> anyhow::Result<()> { match action { - NoCommandAction::Foreground(selector) => run_foreground_ui(selector), + NoCommandAction::Foreground(selector) => { + let selector = selector.or_else(|| { + read_project_device_selection() + .ok() + .flatten() + .map(|selection| selection.udid) + }); + run_foreground_ui(selector) + } NoCommandAction::Detached => start_detached_daemon(DaemonLaunchOptions::default()), NoCommandAction::Kill => stop_project_daemon(), NoCommandAction::Restart => restart_detached_daemon(DaemonLaunchOptions::default()), @@ -2450,6 +2510,13 @@ fn resolve_cli_device_udid( return Ok(selector); } + if let Some(selection) = read_project_device_selection()? { + let udid = selection.udid.trim(); + if !udid.is_empty() { + return Ok(udid.to_owned()); + } + } + let server_url = command_service_url(explicit_server_url)?; if let Some(simulator) = infer_default_cli_simulator(&server_url)? { return Ok(simulator.udid); @@ -2462,14 +2529,14 @@ fn resolve_cli_device_udid( .collect::>(); if booted.len() > 1 { anyhow::bail!( - "Multiple booted simulators are available. Pass a UDID, use --device, or set SIMDECK_DEVICE." + "Multiple booted simulators are available. Pass a UDID, run `simdeck use `, use --device, or set SIMDECK_DEVICE." ); } if simulators.is_empty() { anyhow::bail!("No simulators are available. Boot one or pass a UDID explicitly."); } anyhow::bail!( - "No default simulator could be inferred. Pass a UDID, use --device, or set SIMDECK_DEVICE." + "No default simulator could be inferred. Pass a UDID, run `simdeck use `, use --device, or set SIMDECK_DEVICE." ) } @@ -2560,6 +2627,134 @@ fn parse_tap_command_args( Ok(target) } +fn project_device_selection_for_selector( + selector: &str, + explicit_server_url: Option<&str>, +) -> anyhow::Result { + let selector = selector.trim(); + if selector.is_empty() { + anyhow::bail!("simdeck use requires a simulator UDID or name."); + } + + let project_root = project_root()?; + if android::is_android_id(selector) || looks_like_device_selector(selector) { + return Ok(ProjectDeviceSelection { + project_root, + udid: selector.to_owned(), + name: None, + runtime_name: None, + selected_at: now_secs(), + }); + } + + let server_url = command_service_url(explicit_server_url)?; + let matched = select_studio_simulator(&server_url, selector)?; + if let Some(simulator) = matched { + return Ok(ProjectDeviceSelection { + project_root, + udid: simulator.udid, + name: Some(simulator.name), + runtime_name: simulator.runtime_name, + selected_at: now_secs(), + }); + } + + anyhow::bail!("No simulator matched {selector:?}. Run `simdeck list` to see available UDIDs.") +} + +fn parse_optional_udid_value_args( + command: &str, + args: Vec, + value_name: &str, +) -> anyhow::Result<(Option, String)> { + let args = clean_cli_args(args); + match args.as_slice() { + [value] => Ok((None, value.clone())), + [udid, value] => Ok((Some(udid.clone()), value.clone())), + [] => anyhow::bail!("{command} requires {value_name}."), + _ => anyhow::bail!("{command} accepts either {value_name} or UDID {value_name}."), + } +} + +fn parse_optional_udid_text_args( + command: &str, + args: Vec, + has_non_positional_input: bool, +) -> anyhow::Result<(Option, Option)> { + let args = clean_cli_args(args); + if has_non_positional_input { + return match args.as_slice() { + [] => Ok((None, None)), + [udid] => Ok((Some(udid.clone()), None)), + _ => anyhow::bail!( + "{command} accepts at most one positional UDID with --stdin or --file." + ), + }; + } + match args.as_slice() { + [] => Ok((None, None)), + [text] => Ok((None, Some(text.clone()))), + [udid, text] => Ok((Some(udid.clone()), Some(text.clone()))), + _ => anyhow::bail!("{command} accepts either TEXT or UDID TEXT. Quote multi-word text."), + } +} + +fn parse_optional_udid_f64_args( + command: &str, + args: Vec, + expected_values: usize, +) -> anyhow::Result<(Option, Vec)> { + let args = clean_cli_args(args); + let (udid, values) = match args.len() { + len if len == expected_values => (None, args.as_slice()), + len if len == expected_values + 1 => (Some(args[0].clone()), &args[1..]), + _ => anyhow::bail!( + "{command} accepts either {expected_values} numeric values or UDID plus {expected_values} numeric values." + ), + }; + let mut parsed = Vec::with_capacity(values.len()); + for value in values { + parsed.push(parse_f64_arg(value).ok_or_else(|| { + anyhow::anyhow!("{command} expected a finite number, got {value:?}.") + })?); + } + Ok((udid, parsed)) +} + +fn parse_optional_udid_point_args( + command: &str, + args: Vec, +) -> anyhow::Result<(Option, Option, Option)> { + let args = clean_cli_args(args); + match args.as_slice() { + [] => Ok((None, None, None)), + [udid] => Ok((Some(udid.clone()), None, None)), + [x, y] => Ok(( + None, + Some(parse_required_f64_arg(command, x)?), + Some(parse_required_f64_arg(command, y)?), + )), + [udid, x, y] => Ok(( + Some(udid.clone()), + Some(parse_required_f64_arg(command, x)?), + Some(parse_required_f64_arg(command, y)?), + )), + _ => anyhow::bail!("{command} accepts [UDID] or [UDID] CENTER_X CENTER_Y."), + } +} + +fn parse_required_f64_arg(command: &str, value: &str) -> anyhow::Result { + parse_f64_arg(value) + .ok_or_else(|| anyhow::anyhow!("{command} expected a finite number, got {value:?}.")) +} + +fn clean_cli_args(args: Vec) -> Vec { + args.into_iter() + .map(|arg| arg.trim().to_owned()) + .filter(|arg| !arg.is_empty()) + .collect() +} + fn parse_f64_arg(value: &str) -> Option { value.parse::().ok().filter(|value| value.is_finite()) } @@ -2730,6 +2925,13 @@ fn main() -> anyhow::Result<()> { .or_else(|| env::var("SIMDECK_SERVER_URL").ok()) .filter(|value| !value.trim().is_empty()); let bridge = NativeBridge; + let resolve_device_udid = |udid: Option<&str>| -> anyhow::Result { + resolve_cli_device_udid( + udid, + device_selector.as_deref(), + explicit_server_url.as_deref(), + ) + }; match cli.command { Command::Ui { @@ -2920,11 +3122,13 @@ fn main() -> anyhow::Result<()> { Command::Provider { command } => run_provider_command(command), Command::Maestro { command } => match command { MaestroCommand::Test { - udid, - flow, + args, artifacts_dir, continue_on_error, } => { + let (udid, flow) = parse_optional_udid_value_args("maestro test", args, "FLOW")?; + let udid = resolve_device_udid(udid.as_deref())?; + let flow = PathBuf::from(flow); let service_url = command_service_url(explicit_server_url.as_deref())?; let report = run_maestro_flow(&service_url, &udid, &flow, artifacts_dir, continue_on_error)?; @@ -3063,7 +3267,23 @@ fn main() -> anyhow::Result<()> { print_list_simulators(&simulators, format)?; Ok(()) } + Command::Use { udid } => { + let selection = + project_device_selection_for_selector(&udid, explicit_server_url.as_deref())?; + let path = write_project_device_selection(&selection)?; + println_json(&serde_json::json!({ + "ok": true, + "action": "use", + "udid": selection.udid, + "name": selection.name, + "runtimeName": selection.runtime_name, + "projectRoot": selection.project_root, + "path": path, + }))?; + Ok(()) + } Command::Boot { udid } => { + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; service_post_ok(&service_url, &udid, "boot", &Value::Null)?; println!( @@ -3075,6 +3295,7 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::Shutdown { udid } => { + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; service_post_ok(&service_url, &udid, "shutdown", &Value::Null)?; println!( @@ -3085,7 +3306,9 @@ fn main() -> anyhow::Result<()> { ); Ok(()) } - Command::OpenUrl { udid, url } => { + Command::OpenUrl { args } => { + let (udid, url) = parse_optional_udid_value_args("open-url", args, "URL")?; + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; service_open_url(&service_url, &udid, &url)?; println!( @@ -3096,7 +3319,9 @@ fn main() -> anyhow::Result<()> { ); Ok(()) } - Command::Launch { udid, bundle_id } => { + Command::Launch { args } => { + let (udid, bundle_id) = parse_optional_udid_value_args("launch", args, "BUNDLE_ID")?; + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; service_launch(&service_url, &udid, &bundle_id)?; println!( @@ -3108,6 +3333,7 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::ToggleAppearance { udid } => { + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; service_post_ok(&service_url, &udid, "toggle-appearance", &Value::Null)?; println_json( @@ -3116,12 +3342,15 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::Erase { udid } => { + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; service_post_ok(&service_url, &udid, "erase", &Value::Null)?; println_json(&serde_json::json!({ "ok": true, "udid": udid, "action": "erase" }))?; Ok(()) } - Command::Install { udid, app_path } => { + Command::Install { args } => { + let (udid, app_path) = parse_optional_udid_value_args("install", args, "APP_PATH")?; + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; service_post_ok( &service_url, @@ -3134,7 +3363,9 @@ fn main() -> anyhow::Result<()> { )?; Ok(()) } - Command::Uninstall { udid, bundle_id } => { + Command::Uninstall { args } => { + let (udid, bundle_id) = parse_optional_udid_value_args("uninstall", args, "BUNDLE_ID")?; + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; service_post_ok( &service_url, @@ -3149,6 +3380,7 @@ fn main() -> anyhow::Result<()> { } Command::Pasteboard { command } => match command { PasteboardCommand::Get { udid } => { + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; let text = service_get_json( &service_url, @@ -3161,12 +3393,14 @@ fn main() -> anyhow::Result<()> { println_json(&serde_json::json!({ "udid": udid, "text": text }))?; Ok(()) } - PasteboardCommand::Set { - udid, - text, - stdin, - file, - } => { + PasteboardCommand::Set { args, stdin, file } => { + let has_non_positional_input = stdin || file.is_some(); + let (udid, text) = parse_optional_udid_text_args( + "pasteboard set", + args, + has_non_positional_input, + )?; + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; let text = read_text_input(text, stdin, file)?; service_post_ok( @@ -3186,6 +3420,7 @@ fn main() -> anyhow::Result<()> { seconds, limit, } => { + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; let filters = native::bridge::LogFilters::new(Vec::new(), Vec::new(), String::new()); let _ = filters; @@ -3203,6 +3438,7 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::Processes { udid } => { + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; let processes = service_get_json( &service_url, @@ -3217,6 +3453,7 @@ fn main() -> anyhow::Result<()> { watch, interval, } => { + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; if watch { run_stats_watch(&service_url, &udid, pid, interval)?; @@ -3227,6 +3464,7 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::Sample { udid, pid, seconds } => { + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; let pid = match pid { Some(pid) => pid, @@ -3255,6 +3493,7 @@ fn main() -> anyhow::Result<()> { stdout, with_bezel, } => { + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; let query = if with_bezel { "?bezel=true" } else { "" }; let png = service_get_bytes( @@ -3288,6 +3527,7 @@ fn main() -> anyhow::Result<()> { stdout, seconds, } => { + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; let mp4 = service_post_bytes( &service_url, @@ -3314,7 +3554,10 @@ fn main() -> anyhow::Result<()> { } Ok(()) } - Command::Stream { udid, frames } => run_stream_stdout(&bridge, udid, frames), + Command::Stream { udid, frames } => { + let udid = resolve_device_udid(udid.as_deref())?; + run_stream_stdout(&bridge, udid, frames) + } Command::DescribeUi { udid, point, @@ -3350,15 +3593,17 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::Touch { - udid, - x, - y, + args, phase, normalized, down, up, delay_ms, } => { + let (udid, points) = parse_optional_udid_f64_args("touch", args, 2)?; + let udid = resolve_device_udid(udid.as_deref())?; + let x = points[0]; + let y = points[1]; let android_device = android::is_android_id(&udid); if android_device && !normalized { anyhow::bail!("Android touch coordinates require --normalized."); @@ -3519,6 +3764,7 @@ fn main() -> anyhow::Result<()> { timeout_ms, poll_interval_ms, } => { + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; let result = service_wait_for_selector( &service_url, @@ -3543,6 +3789,7 @@ fn main() -> anyhow::Result<()> { timeout_ms, poll_interval_ms, } => { + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; let result = service_wait_for_selector( &service_url, @@ -3559,17 +3806,19 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::Swipe { - udid, - start_x, - start_y, - end_x, - end_y, + args, normalized, duration_ms, steps, pre_delay_ms, post_delay_ms, } => { + let (udid, points) = parse_optional_udid_f64_args("swipe", args, 4)?; + let udid = resolve_device_udid(udid.as_deref())?; + let start_x = points[0]; + let start_y = points[1]; + let end_x = points[2]; + let end_y = points[3]; let android_device = android::is_android_id(&udid); if android_device && !normalized { anyhow::bail!("Android swipe coordinates require --normalized."); @@ -3629,8 +3878,7 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::Gesture { - udid, - preset, + args, screen_width, screen_height, normalized, @@ -3639,6 +3887,8 @@ fn main() -> anyhow::Result<()> { pre_delay_ms, post_delay_ms, } => { + let (udid, preset) = parse_optional_udid_value_args("gesture", args, "PRESET")?; + let udid = resolve_device_udid(udid.as_deref())?; let android_device = android::is_android_id(&udid); let command_server_url = command_service_url_for_udid(&udid, &explicit_server_url, &service_url)?; @@ -3718,9 +3968,7 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::Pinch { - udid, - center_x, - center_y, + args, start_distance, end_distance, angle_degrees, @@ -3728,6 +3976,8 @@ fn main() -> anyhow::Result<()> { duration_ms, steps, } => { + let (udid, center_x, center_y) = parse_optional_udid_point_args("pinch", args)?; + let udid = resolve_device_udid(udid.as_deref())?; if android::is_android_id(&udid) { anyhow::bail!("Android pinch gestures are not supported by the ADB input bridge."); } @@ -3747,15 +3997,16 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::RotateGesture { - udid, - center_x, - center_y, + args, radius, degrees, normalized, duration_ms, steps, } => { + let (udid, center_x, center_y) = + parse_optional_udid_point_args("rotate-gesture", args)?; + let udid = resolve_device_udid(udid.as_deref())?; if android::is_android_id(&udid) { anyhow::bail!("Android rotate gestures are not supported by the ADB input bridge."); } @@ -3778,13 +4029,14 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::Key { - udid, - key, + args, modifiers, duration_ms, pre_delay_ms, post_delay_ms, } => { + let (udid, key) = parse_optional_udid_value_args("key", args, "KEY")?; + let udid = resolve_device_udid(udid.as_deref())?; let key_code = parse_hid_key(&key)?; sleep_ms(pre_delay_ms); let command_server_url = @@ -3808,6 +4060,7 @@ fn main() -> anyhow::Result<()> { keycodes, delay_ms, } => { + let udid = resolve_device_udid(udid.as_deref())?; let keys = parse_key_list(&keycodes)?; let command_server_url = command_service_url_for_udid(&udid, &explicit_server_url, &service_url)?; @@ -3832,6 +4085,7 @@ fn main() -> anyhow::Result<()> { modifiers, key, } => { + let udid = resolve_device_udid(udid.as_deref())?; let modifier_mask = parse_modifier_mask(&modifiers)?; let key_code = parse_hid_key(&key)?; let command_server_url = @@ -3845,12 +4099,15 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::Type { - udid, - text, + args, stdin, file, delay_ms, } => { + let has_non_positional_input = stdin || file.is_some(); + let (udid, text) = + parse_optional_udid_text_args("type", args, has_non_positional_input)?; + let udid = resolve_device_udid(udid.as_deref())?; let text = read_text_input(text, stdin, file)?; if android::is_android_id(&udid) { let server_url = command_service_url(explicit_server_url.as_deref())?; @@ -3870,11 +4127,9 @@ fn main() -> anyhow::Result<()> { println_json(&serde_json::json!({ "ok": true, "udid": udid, "action": "type" }))?; Ok(()) } - Command::Button { - udid, - button, - duration_ms, - } => { + Command::Button { args, duration_ms } => { + let (udid, button) = parse_optional_udid_value_args("button", args, "BUTTON")?; + let udid = resolve_device_udid(udid.as_deref())?; let command_server_url = command_service_url_for_udid(&udid, &explicit_server_url, &service_url)?; if let Some(server_url) = command_server_url.as_deref() { @@ -3888,6 +4143,7 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::Crown { udid, delta } => { + let udid = resolve_device_udid(udid.as_deref())?; if let Some(server_url) = service_url.as_deref() { service_crown(server_url, &udid, delta)?; } else { @@ -3905,6 +4161,7 @@ fn main() -> anyhow::Result<()> { stdin, continue_on_error, } => { + let udid = resolve_device_udid(udid.as_deref())?; let command_server_url = command_service_url_for_udid(&udid, &explicit_server_url, &service_url)?; let report = if let Some(server_url) = command_server_url.as_deref() { @@ -3922,6 +4179,7 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::DismissKeyboard { udid } => { + let udid = resolve_device_udid(udid.as_deref())?; let command_server_url = command_service_url_for_udid(&udid, &explicit_server_url, &service_url)?; if let Some(server_url) = command_server_url.as_deref() { @@ -3938,6 +4196,7 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::Home { udid } => { + let udid = resolve_device_udid(udid.as_deref())?; let command_server_url = command_service_url_for_udid(&udid, &explicit_server_url, &service_url)?; if let Some(server_url) = command_server_url.as_deref() { @@ -3949,6 +4208,7 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::AppSwitcher { udid } => { + let udid = resolve_device_udid(udid.as_deref())?; let command_server_url = command_service_url_for_udid(&udid, &explicit_server_url, &service_url)?; if let Some(server_url) = command_server_url.as_deref() { @@ -3962,6 +4222,7 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::RotateLeft { udid } => { + let udid = resolve_device_udid(udid.as_deref())?; let command_server_url = command_service_url_for_udid(&udid, &explicit_server_url, &service_url)?; if let Some(server_url) = command_server_url.as_deref() { @@ -3975,6 +4236,7 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::RotateRight { udid } => { + let udid = resolve_device_udid(udid.as_deref())?; let command_server_url = command_service_url_for_udid(&udid, &explicit_server_url, &service_url)?; if let Some(server_url) = command_server_url.as_deref() { @@ -3988,6 +4250,7 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::ChromeProfile { udid } => { + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; let profile = service_get_json( &service_url, @@ -7586,14 +7849,16 @@ mod tests { batch_line_to_json_step, daemon_matches_launch_options, http_url_for_host, interactive_accessibility_snapshot, is_tailscale_ip, maestro_commands_from_flow, maestro_selector, normalize_accessibility_point_for_display, parse_maestro_flow_yaml, - parse_maestro_point, parse_tap_command_args, parse_workspace_daemon_process_line, - render_agent_accessibility_tree, render_qr_code, run_maestro_command, - server_health_watchdog_should_restart, service_post_error_is_retryable, simdeck_pair_url, - studio_daemon_restart_args, workspace_daemon_process_is_current, Cli, Command, - DaemonCommand, DaemonLaunchOptions, DaemonMetadata, ElementSelector, PairingAddress, - ServiceCommand, StreamQualityProfileArg, StudioExposeOptions, TapCommandTarget, - VideoCodecMode, WorkspaceDaemonProcess, YamlValue, DEFAULT_LOCAL_STREAM_QUALITY_PROFILE, - SERVER_HEALTH_WATCHDOG_FAILURE_THRESHOLD, SERVER_HEALTH_WATCHDOG_HTTP_FAILURE_THRESHOLD, + parse_maestro_point, parse_optional_udid_f64_args, parse_optional_udid_text_args, + parse_optional_udid_value_args, parse_tap_command_args, + parse_workspace_daemon_process_line, render_agent_accessibility_tree, render_qr_code, + run_maestro_command, server_health_watchdog_should_restart, + service_post_error_is_retryable, simdeck_pair_url, studio_daemon_restart_args, + workspace_daemon_process_is_current, Cli, Command, DaemonCommand, DaemonLaunchOptions, + DaemonMetadata, ElementSelector, PairingAddress, ServiceCommand, StreamQualityProfileArg, + StudioExposeOptions, TapCommandTarget, VideoCodecMode, WorkspaceDaemonProcess, YamlValue, + DEFAULT_LOCAL_STREAM_QUALITY_PROFILE, SERVER_HEALTH_WATCHDOG_FAILURE_THRESHOLD, + SERVER_HEALTH_WATCHDOG_HTTP_FAILURE_THRESHOLD, }; use clap::Parser; use std::collections::HashMap; @@ -8095,6 +8360,109 @@ mod tests { assert_eq!(parsed.device.as_deref(), Some("iPhone 16")); } + #[test] + fn use_command_accepts_udid_selector() { + let parsed = + Cli::try_parse_from(["simdeck", "use", "00000000-0000-0000-0000-000000000001"]) + .unwrap(); + + let Command::Use { udid } = parsed.command else { + panic!("expected use command"); + }; + assert_eq!(udid, "00000000-0000-0000-0000-000000000001"); + } + + #[test] + fn device_commands_accept_omitted_udid() { + let parsed = Cli::try_parse_from(["simdeck", "boot"]).unwrap(); + let Command::Boot { udid } = parsed.command else { + panic!("expected boot command"); + }; + assert_eq!(udid, None); + + let parsed = Cli::try_parse_from(["simdeck", "home"]).unwrap(); + let Command::Home { udid } = parsed.command else { + panic!("expected home command"); + }; + assert_eq!(udid, None); + + let parsed = Cli::try_parse_from(["simdeck", "screenshot", "--stdout"]).unwrap(); + let Command::Screenshot { udid, stdout, .. } = parsed.command else { + panic!("expected screenshot command"); + }; + assert_eq!(udid, None); + assert!(stdout); + } + + #[test] + fn payload_commands_keep_legacy_udid_but_allow_default_device() { + let parsed = Cli::try_parse_from(["simdeck", "launch", "com.example.App"]).unwrap(); + let Command::Launch { args } = parsed.command else { + panic!("expected launch command"); + }; + let (udid, bundle_id) = + parse_optional_udid_value_args("launch", args, "BUNDLE_ID").unwrap(); + assert_eq!(udid, None); + assert_eq!(bundle_id, "com.example.App"); + + let parsed = + Cli::try_parse_from(["simdeck", "launch", "SIM-1", "com.example.App"]).unwrap(); + let Command::Launch { args } = parsed.command else { + panic!("expected launch command"); + }; + let (udid, bundle_id) = + parse_optional_udid_value_args("launch", args, "BUNDLE_ID").unwrap(); + assert_eq!(udid.as_deref(), Some("SIM-1")); + assert_eq!(bundle_id, "com.example.App"); + } + + #[test] + fn coordinate_commands_keep_legacy_udid_but_allow_default_device() { + let parsed = Cli::try_parse_from(["simdeck", "touch", "120", "240"]).unwrap(); + let Command::Touch { args, .. } = parsed.command else { + panic!("expected touch command"); + }; + let (udid, points) = parse_optional_udid_f64_args("touch", args, 2).unwrap(); + assert_eq!(udid, None); + assert_eq!(points, vec![120.0, 240.0]); + + let parsed = + Cli::try_parse_from(["simdeck", "swipe", "SIM-1", "10", "20", "30", "40"]).unwrap(); + let Command::Swipe { args, .. } = parsed.command else { + panic!("expected swipe command"); + }; + let (udid, points) = parse_optional_udid_f64_args("swipe", args, 4).unwrap(); + assert_eq!(udid.as_deref(), Some("SIM-1")); + assert_eq!(points, vec![10.0, 20.0, 30.0, 40.0]); + } + + #[test] + fn text_commands_use_positional_text_or_legacy_udid_with_input_flags() { + let parsed = Cli::try_parse_from(["simdeck", "type", "hello"]).unwrap(); + let Command::Type { + args, stdin, file, .. + } = parsed.command + else { + panic!("expected type command"); + }; + let (udid, text) = + parse_optional_udid_text_args("type", args, stdin || file.is_some()).unwrap(); + assert_eq!(udid, None); + assert_eq!(text.as_deref(), Some("hello")); + + let parsed = Cli::try_parse_from(["simdeck", "type", "SIM-1", "--stdin"]).unwrap(); + let Command::Type { + args, stdin, file, .. + } = parsed.command + else { + panic!("expected type command"); + }; + let (udid, text) = + parse_optional_udid_text_args("type", args, stdin || file.is_some()).unwrap(); + assert_eq!(udid.as_deref(), Some("SIM-1")); + assert_eq!(text, None); + } + #[test] fn batch_sleep_positional_duration_defaults_to_milliseconds() { let step = batch_line_to_json_step("sleep 500").unwrap(); diff --git a/skills/simdeck/SKILL.md b/skills/simdeck/SKILL.md index b0d59b70..984c3ee5 100644 --- a/skills/simdeck/SKILL.md +++ b/skills/simdeck/SKILL.md @@ -47,23 +47,28 @@ If Browser Use is not available, only then use `simdeck ui --open` - it would op ## Device And App -Most device commands take `` immediately after the command. For fast agent loops, `describe` and `tap` can infer the device from `--device`, `SIMDECK_DEVICE`, `SIMDECK_UDID`, or the only booted simulator. +Start by choosing a project default device. `simdeck use ` stores the +selection for the current workspace/CWD so later commands can omit the UDID. +Explicit UDIDs, `--device`, `SIMDECK_DEVICE`, and `SIMDECK_UDID` still work for +one-off overrides. Prefer short forms in agent loops, such as +`simdeck tap "Continue"` and `simdeck describe --format agent --max-depth 2`. ```bash simdeck list simdeck list --format json +simdeck use simdeck boot -simdeck shutdown -simdeck erase +simdeck shutdown +simdeck erase simdeck core-simulator restart -simdeck install /path/to/App.app -simdeck install /path/to/App.ipa +simdeck install /path/to/App.app +simdeck install /path/to/App.ipa simdeck install android: /path/to/app.apk -simdeck launch com.example.App -simdeck uninstall com.example.App -simdeck open-url myapp://route -simdeck open-url https://example.com -simdeck toggle-appearance +simdeck launch com.example.App +simdeck uninstall com.example.App +simdeck open-url myapp://route +simdeck open-url https://example.com +simdeck toggle-appearance ``` `simdeck list` defaults to compact JSON for token-efficient agent selection. @@ -80,21 +85,21 @@ AVDs from the Android SDK. Use targeted checks for test loops. `describe` is a diagnostic snapshot of the whole hierarchy. For verification, prefer the daemon APIs exposed by `simdeck/test`: `query`, `waitFor`, `assert`, selector `tap`, and `batch`. ```bash -simdeck describe -simdeck describe --format agent --max-depth 4 -simdeck describe --format agent --max-depth 4 --interactive -simdeck describe --format compact-json -simdeck describe --point 120,240 -simdeck describe --source auto -simdeck describe --source nativescript -simdeck describe --source react-native -simdeck describe --source flutter -simdeck describe --source uikit -simdeck describe --source native-ax -simdeck describe --source android-uiautomator -simdeck describe --direct -simdeck wait-for --label "Welcome" --timeout-ms 5000 -simdeck assert --id login.button --source auto --max-depth 8 +simdeck describe +simdeck describe --format agent --max-depth 4 +simdeck describe --format agent --max-depth 4 --interactive +simdeck describe --format compact-json +simdeck describe --point 120,240 +simdeck describe --source auto +simdeck describe --source nativescript +simdeck describe --source react-native +simdeck describe --source flutter +simdeck describe --source uikit +simdeck describe --source native-ax +simdeck describe --source android-uiautomator +simdeck describe --direct +simdeck wait-for --label "Welcome" --timeout-ms 5000 +simdeck assert --id login.button --source auto --max-depth 8 ``` Use `--source auto` with the project daemon. Use `--direct` or `--source native-ax` for the private CoreSimulator accessibility bridge. Use `--source android-uiautomator` for Android emulator UIAutomator hierarchies. NativeScript, React Native, and Flutter inspector runtimes can add richer hierarchy data. @@ -105,9 +110,9 @@ Use `--interactive` or `-i` when an agent only needs controls and actionable fra Prefer selectors, coordinates only when needed. Selector taps go through the daemon and wait for the element server-side. ```bash -simdeck tap --id LoginButton --wait-timeout-ms 5000 -simdeck tap --label "Continue" --element-type Button -simdeck tap 120 240 +simdeck tap --id LoginButton --wait-timeout-ms 5000 +simdeck tap --label "Continue" --element-type Button +simdeck tap 120 240 simdeck tap "Continue" ``` @@ -136,46 +141,46 @@ Use `tree()`/`describe` only when a test needs to print the whole UI for debuggi ## Interact ```bash -simdeck tap 120 240 -simdeck touch 0.5 0.5 --phase began --normalized -simdeck touch 0.5 0.5 --phase ended --normalized -simdeck touch 120 240 --down --up --delay-ms 800 -simdeck swipe 200 700 200 200 -simdeck swipe 200 700 200 200 --duration-ms 500 --pre-delay-ms 100 --post-delay-ms 250 -simdeck gesture scroll-up -simdeck gesture scroll-down -simdeck gesture swipe-from-left-edge -simdeck gesture swipe-from-right-edge -simdeck pinch --start-distance 160 --end-distance 80 -simdeck pinch --start-distance 0.20 --end-distance 0.35 --normalized --duration-ms 250 --steps 8 -simdeck rotate-gesture --radius 100 --degrees 90 -simdeck rotate-gesture --radius 0.12 --degrees 45 --normalized --duration-ms 250 --steps 8 -simdeck type 'hello' -simdeck type --stdin -simdeck type --file message.txt -simdeck key enter -simdeck key 42 --duration-ms 500 -simdeck key-sequence --keycodes h,e,l,l,o --delay-ms 75 -simdeck key-combo --modifiers cmd,shift --key z -simdeck dismiss-keyboard -simdeck button home -simdeck button lock --duration-ms 1000 -simdeck button side-button -simdeck button volume-up -simdeck button volume-down -simdeck button action --duration-ms 1000 -simdeck button mute -simdeck button digital-crown -simdeck crown --delta 50 -simdeck button left-side-button -simdeck button siri -simdeck button apple-pay -simdeck home -simdeck app-switcher -simdeck rotate-left -simdeck rotate-right -simdeck pasteboard set 'text' -simdeck pasteboard get +simdeck tap 120 240 +simdeck touch 0.5 0.5 --phase began --normalized +simdeck touch 0.5 0.5 --phase ended --normalized +simdeck touch 120 240 --down --up --delay-ms 800 +simdeck swipe 200 700 200 200 +simdeck swipe 200 700 200 200 --duration-ms 500 --pre-delay-ms 100 --post-delay-ms 250 +simdeck gesture scroll-up +simdeck gesture scroll-down +simdeck gesture swipe-from-left-edge +simdeck gesture swipe-from-right-edge +simdeck pinch --start-distance 160 --end-distance 80 +simdeck pinch --start-distance 0.20 --end-distance 0.35 --normalized --duration-ms 250 --steps 8 +simdeck rotate-gesture --radius 100 --degrees 90 +simdeck rotate-gesture --radius 0.12 --degrees 45 --normalized --duration-ms 250 --steps 8 +simdeck type 'hello' +simdeck type --stdin +simdeck type --file message.txt +simdeck key enter +simdeck key 42 --duration-ms 500 +simdeck key-sequence --keycodes h,e,l,l,o --delay-ms 75 +simdeck key-combo --modifiers cmd,shift --key z +simdeck dismiss-keyboard +simdeck button home +simdeck button lock --duration-ms 1000 +simdeck button side-button +simdeck button volume-up +simdeck button volume-down +simdeck button action --duration-ms 1000 +simdeck button mute +simdeck button digital-crown +simdeck crown --delta 50 +simdeck button left-side-button +simdeck button siri +simdeck button apple-pay +simdeck home +simdeck app-switcher +simdeck rotate-left +simdeck rotate-right +simdeck pasteboard set 'text' +simdeck pasteboard get ``` Use `--stdin` or `--file` for text with quotes, newlines, shell variables, or shell-sensitive characters. @@ -183,9 +188,9 @@ Use `--stdin` or `--file` for text with quotes, newlines, shell variables, or sh ## Timing, Batch ```bash -simdeck tap --label "Continue" --wait-timeout-ms 5000 -simdeck swipe 200 700 200 200 --pre-delay-ms 100 --post-delay-ms 250 -simdeck button lock --duration-ms 1000 +simdeck tap --label "Continue" --wait-timeout-ms 5000 +simdeck swipe 200 700 200 200 --pre-delay-ms 100 --post-delay-ms 250 +simdeck button lock --duration-ms 1000 ``` Prefer to use `wait-for` or `assert` in a batch to wait for UI state instead of fixed delays. `sleep 500` in a batch waits 500 ms. Use `sleep 0.5s` or `sleep --seconds 0.5` when you want to write seconds explicitly. @@ -193,14 +198,14 @@ Prefer to use `wait-for` or `assert` in a batch to wait for UI state instead of Use `batch` when steps are known; use discrete commands when a later step depends on parsing previous output. ```bash -simdeck batch \ +simdeck batch \ --step "tap --label Continue --wait-timeout-ms 5000" \ --step "type 'hello world'" \ --step "gesture scroll-down" \ --step "pinch --start-distance 0.20 --end-distance 0.35 --normalized" ``` -Batch rules: one source (`--step`, `--file`, or `--stdin`); keep `` at batch level; ordered steps; fail-fast by default; `--continue-on-error` for best effort. Step commands: `tap`, `wait-for`, `assert`, `swipe`, `gesture`, `pinch`, `rotate-gesture`, `touch`, `type`, `button`, `key`, `key-sequence`, `key-combo`, `sleep`. +Batch rules: one source (`--step`, `--file`, or `--stdin`); set the default with `simdeck use ` or keep `` at batch level; ordered steps; fail-fast by default; `--continue-on-error` for best effort. Step commands: `tap`, `wait-for`, `assert`, `swipe`, `gesture`, `pinch`, `rotate-gesture`, `touch`, `type`, `button`, `key`, `key-sequence`, `key-combo`, `sleep`. For JS tests, batch can combine action and verification without extra CLI process startup: @@ -219,30 +224,30 @@ await simdeck.batch([ For app-style flows, SimDeck can run a practical subset of Maestro YAML: ```bash -simdeck maestro test flow.yaml --artifacts-dir artifacts/maestro +simdeck maestro test flow.yaml --artifacts-dir artifacts/maestro ``` ## Evidence ```bash -simdeck screenshot --output screen.png -simdeck screenshot --with-bezel --output screen-bezel.png -simdeck screenshot --stdout > screen.png -simdeck record --seconds 5 --output screen-recording.mp4 -simdeck record --seconds 5 --stdout > screen-recording.mp4 -simdeck logs --seconds 30 --limit 200 -simdeck chrome-profile -simdeck processes -simdeck stats -simdeck stats --watch -simdeck sample --seconds 3 +simdeck screenshot --output screen.png +simdeck screenshot --with-bezel --output screen-bezel.png +simdeck screenshot --stdout > screen.png +simdeck record --seconds 5 --output screen-recording.mp4 +simdeck record --seconds 5 --stdout > screen-recording.mp4 +simdeck logs --seconds 30 --limit 200 +simdeck chrome-profile +simdeck processes +simdeck stats +simdeck stats --watch +simdeck sample --seconds 3 ``` Use screenshots for still evidence, `--with-bezel` when the device frame matters, and `record` for short MP4 screen recordings. Use `stats` for simulator app CPU, memory, disk write, network receive/send rates, connections, hang, and crash/termination signals. Use `sample` only when a short CPU stack capture is worth the extra pause. Prefer describe for token-efficient state dumps, if they have enough context. ## Default Loop -1. Start UI, list, boot/select ``, open viewer if in-app browser available +1. Start UI, list, `simdeck use `, boot/select the device, open viewer if in-app browser available 2. Build with project tools; install and launch with SimDeck. 3. Use one `describe --format agent --max-depth 4` to understand an unfamiliar screen. 4. Interact with selectors first; use coordinates only when needed.