diff --git a/AGENTS.md b/AGENTS.md index aab69475..4ca68e94 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -79,7 +79,7 @@ Private simulator behavior is implemented locally in: The current repo uses the private boot path, private display bridge, and private accessibility translation bridge directly. The browser streams frames from that bridge, injects touch and keyboard events through the same native session layer, inspects accessibility through `AccessibilityPlatformTranslation`, and renders device chrome from `cli/XCWChromeRenderer.*`. CoreSimulator service contexts resolve the active developer directory from `DEVELOPER_DIR`, then `xcode-select -p`, then `/Applications/Xcode.app/Contents/Developer`. The display bridge prefers direct CoreSimulator screen IOSurface callbacks and activates the SimulatorKit offscreen renderable view only if direct callbacks are unavailable. -Accessibility recovery may use simulator launchctl UIKit application state plus hit-tested translations to recover candidate foreground pids; the returned tree must still be rooted at tokenized `AXPTranslator` application objects, because `translationApplicationObjectForPid:` can omit the bridge delegate token after private display lifecycle changes. Full-tree snapshots merge those recovered roots with the private frontmost application translation. When multiple candidate application roots are discovered, serialize all of them in preferred order: non-extension app roots first, then largest translated roots, with `.appex`/PlugIns processes de-prioritized so SpringBoard and Safari app roots stay primary while widgets and WebContent roots remain debuggable. Widget renderer extension roots may report local frames; normalize those roots and children against matching SpringBoard widget placeholder frames before returning the snapshot. +Accessibility recovery may use simulator launchctl UIKit application state plus hit-tested translations to recover candidate foreground pids; the returned tree must still be rooted at tokenized `AXPTranslator` application objects, because `translationApplicationObjectForPid:` can omit the bridge delegate token after private display lifecycle changes. Full-tree snapshots merge those recovered roots with the private frontmost application translation. Shallow snapshots with `maxDepth <= 2` use the tokenized frontmost application translation directly when it is available, and only run the expensive recovery sweep if frontmost lookup fails, so agent-oriented describe loops avoid launchctl and hit-test recovery overhead. When multiple candidate application roots are discovered, serialize all of them in preferred order: non-extension app roots first, then largest translated roots, with `.appex`/PlugIns processes de-prioritized so SpringBoard and Safari app roots stay primary while widgets and WebContent roots remain debuggable. Widget renderer extension roots may report local frames; normalize those roots and children against matching SpringBoard widget placeholder frames before returning the snapshot. Physical chrome button support uses DeviceKit `chrome.json` input geometry for browser hit targets. Volume, action, mute, Apple Watch digital crown, Watch side button, and Watch left-side button dispatch through `IndigoHIDMessageForHIDArbitrary` with consumer/telephony/vendor HID usage pairs from the device chrome metadata; home, lock, and app-switcher remain on the existing SimulatorKit button paths. Apple Watch Digital Crown rotation dispatches through `IndigoHIDMessageForDigitalCrownEvent` when SimulatorKit exposes it, with `IndigoHIDMessageForScrollEvent(..., target=0x34)` as the fallback. tvOS simulators do not support direct screen touch; browser/API tap maps to Enter, swipe maps to arrow keys, and the native bridge rejects tvOS touch packets before they reach guest `SimulatorHID`. watchOS/tvOS skip dynamic pointer/mouse service warm-up because those guest runtimes abort on unsupported virtual services. Apple TV and Apple Watch simulators are fixed-orientation devices, so client and server rotation paths must not expose or dispatch device rotation for those families. Two-point multi-touch dispatch prefers the current SimulatorKit/Indigo packet constructor and falls back to SimDeck's manual Indigo packet adapter. On Xcode 26 SimulatorKit, the constructor expects pixel-space points and stable two-finger movement requires sending `LeftMouseDown` for both `began` and `moved`, then `LeftMouseUp` for `ended`/`cancelled`; using `LeftMouseDragged` for multi-touch moves only advances one contact in UIKit. Do not coalesce multi-touch move packets in the WebSocket or WebRTC control paths, because gesture recognizers need the intermediate two-contact samples. WebKit inspection uses the simulator `webinspectord` Unix socket named `com.apple.webinspectord_sim.socket` and WebKit's binary-plist Remote Inspector selectors. It lists only WebKit content that the runtime exposes as inspectable. For app-owned `WKWebView` on iOS 16.4 and newer, the app must set `isInspectable = true`. diff --git a/README.md b/README.md index c8c159c0..d417021e 100644 --- a/README.md +++ b/README.md @@ -93,8 +93,11 @@ CLI commands automatically use the same warm daemon: ```sh simdeck list -simdeck tap 0.5 0.5 --normalized -simdeck describe --format agent --max-depth 2 +simdeck use +simdeck tap 0.5 0.5 --normalized +simdeck tap "Continue" +simdeck describe --format agent --max-depth 2 --interactive +simdeck --device describe --format agent --max-depth 2 ``` ## Daemon @@ -137,63 +140,72 @@ simdeck core-simulator shutdown ```sh simdeck list +simdeck use simdeck boot -simdeck shutdown -simdeck erase -simdeck install /path/to/App.app -simdeck install /path/to/App.ipa +simdeck shutdown +simdeck erase +simdeck install /path/to/App.app +simdeck install /path/to/App.ipa simdeck install android: /path/to/app.apk -simdeck uninstall com.example.App -simdeck open-url https://example.com -simdeck launch com.apple.Preferences -simdeck toggle-appearance -simdeck pasteboard set "hello" -simdeck pasteboard get -simdeck screenshot --output screen.png -simdeck screenshot --with-bezel --output screen-bezel.png -simdeck record --seconds 5 --output screen-recording.mp4 -simdeck stream --frames 120 > stream.h264 -simdeck describe -simdeck describe --format agent --max-depth 4 -simdeck describe --point 120,240 -simdeck wait-for --label "Welcome" --timeout-ms 5000 -simdeck assert --id login.button --source auto --max-depth 8 -simdeck tap 120 240 -simdeck tap --label "Continue" --wait-timeout-ms 5000 -simdeck swipe 200 700 200 200 -simdeck gesture scroll-down -simdeck pinch --start-distance 160 --end-distance 80 -simdeck rotate-gesture --radius 100 --degrees 90 -simdeck touch 0.5 0.5 --phase began --normalized -simdeck touch 120 240 --down --up --delay-ms 800 -simdeck key enter -simdeck key-sequence --keycodes h,e,l,l,o -simdeck key-combo --modifiers cmd --key a -simdeck type "hello" -simdeck type --file message.txt -simdeck button lock --duration-ms 1000 -simdeck button volume-up -simdeck button action --duration-ms 1000 -simdeck button digital-crown -simdeck crown --delta 50 -simdeck button left-side-button -simdeck batch --step "tap --label Continue" --step "type 'hello'" --step "wait-for --label hello" -simdeck dismiss-keyboard -simdeck home -simdeck app-switcher -simdeck rotate-left -simdeck rotate-right -simdeck chrome-profile -simdeck logs --seconds 30 --limit 200 -simdeck processes -simdeck stats --watch -simdeck sample --seconds 3 +simdeck uninstall com.example.App +simdeck open-url https://example.com +simdeck launch com.apple.Preferences +simdeck toggle-appearance +simdeck pasteboard set "hello" +simdeck pasteboard get +simdeck screenshot --output screen.png +simdeck screenshot --with-bezel --output screen-bezel.png +simdeck record --seconds 5 --output screen-recording.mp4 +simdeck stream --frames 120 > stream.h264 +simdeck describe +simdeck describe --format agent --max-depth 4 +simdeck describe --format agent --max-depth 4 --interactive +simdeck describe --point 120,240 +simdeck wait-for --label "Welcome" --timeout-ms 5000 +simdeck assert --id login.button --source auto --max-depth 8 +simdeck tap 120 240 +simdeck tap --label "Continue" --wait-timeout-ms 5000 +simdeck tap "Continue" +simdeck swipe 200 700 200 200 +simdeck gesture scroll-down +simdeck pinch --start-distance 160 --end-distance 80 +simdeck rotate-gesture --radius 100 --degrees 90 +simdeck touch 0.5 0.5 --phase began --normalized +simdeck touch 120 240 --down --up --delay-ms 800 +simdeck key enter +simdeck key-sequence --keycodes h,e,l,l,o +simdeck key-combo --modifiers cmd --key a +simdeck type "hello" +simdeck type --file message.txt +simdeck button lock --duration-ms 1000 +simdeck button volume-up +simdeck button action --duration-ms 1000 +simdeck button digital-crown +simdeck crown --delta 50 +simdeck button left-side-button +simdeck batch --step "tap --label Continue" --step "type 'hello'" --step "wait-for --label hello" +simdeck dismiss-keyboard +simdeck home +simdeck app-switcher +simdeck rotate-left +simdeck rotate-right +simdeck chrome-profile +simdeck logs --seconds 30 --limit 200 +simdeck processes +simdeck stats --watch +simdeck sample --seconds 3 ``` `simdeck list` defaults to compact JSON for agent-friendly device selection. Use `simdeck list --format json` for the full inventory with paths and display metadata. +`simdeck use ` stores a default simulator for the current project +directory. Most device commands accept `[]`; when it is omitted, SimDeck +uses `--device`, `SIMDECK_DEVICE`, `SIMDECK_UDID`, the saved project default, +or the only booted simulator, in that order. The old explicit-UDID form still +works for every command. + `boot` uses SimDeck's private CoreSimulator boot path so it can start devices without launching Simulator.app. If that private path is unavailable, the command returns the CoreSimulator error instead of falling back to @@ -214,9 +226,12 @@ external tools such as `ffplay`. Flutter, or UIKit in-app inspectors, then falls back to the built-in private CoreSimulator accessibility bridge. Use `--format agent` or `--format compact-json` for -lower-token hierarchy dumps. Coordinate commands accept screen coordinates from -the accessibility tree by default; pass `--normalized` to send `0.0..1.0` -coordinates directly. +lower-token hierarchy dumps, and add `--interactive`/`-i` when an agent only +needs actionable elements plus their ancestors. Set a project default with +`simdeck use ` so agent commands can use short forms like +`simdeck tap "Continue"` and `simdeck describe --format agent --max-depth 2`. +Coordinate commands accept screen coordinates from the accessibility tree by +default; pass `--normalized` to send `0.0..1.0` coordinates directly. ## JS/TS Tests @@ -240,6 +255,12 @@ healthy, and only stops daemons it started itself. Pass `udid` to `connect()` to make it the default for session methods; each method still accepts an explicit UDID as the first argument when needed. +Run common Maestro YAML flows against the same daemon-backed simulator API: + +```sh +simdeck maestro test flow.yaml --artifacts-dir artifacts/maestro +``` + ## NativeScript Inspector NativeScript apps can connect directly to the running server from JS and expose diff --git a/cli/XCWAccessibilityBridge.m b/cli/XCWAccessibilityBridge.m index 53a8b55a..96c4d5c6 100644 --- a/cli/XCWAccessibilityBridge.m +++ b/cli/XCWAccessibilityBridge.m @@ -1419,7 +1419,10 @@ + (nullable NSDictionary *)accessibilitySnapshotForSimulatorUDID:(NSString *)udi break; } } - if (pointValue == nil) { + // Shallow snapshots power fast agent describe loops. Keep the expensive + // multi-root recovery pass for full trees, or when frontmost lookup fails. + BOOL shouldRecoverRoots = pointValue == nil && (translation == nil || maxDepth > 2); + if (shouldRecoverRoots) { NSMutableDictionary *candidatesByKey = [NSMutableDictionary dictionary]; if (translation != nil) { NSMutableDictionary *candidate = XCWAXRootRecoveryCandidateFromTranslation(translator, translation, resolvedDisplayID ?: @0, token); diff --git a/docs/api/rest.md b/docs/api/rest.md index dcf0c9f8..c3347432 100644 --- a/docs/api/rest.md +++ b/docs/api/rest.md @@ -198,17 +198,21 @@ Touch, edge-touch, and multi-touch coordinates are normalized from `0.0` to `1.0 | `GET` | `/api/simulators/{udid}/accessibility-point?x=120&y=240` | Element at a point | | `POST` | `/api/simulators/{udid}/query` | Query tree by selector | | `POST` | `/api/simulators/{udid}/wait-for` | Wait until selector appears | +| `POST` | `/api/simulators/{udid}/wait-for-not` | Wait until selector disappears | | `POST` | `/api/simulators/{udid}/assert` | Assert selector exists | +| `POST` | `/api/simulators/{udid}/assert-not` | Assert selector is absent | +| `POST` | `/api/simulators/{udid}/scroll-until-visible` | Scroll until selector appears | | `POST` | `/api/simulators/{udid}/batch` | Run multiple control steps | | `POST` | `/api/simulators/{udid}/inspector/request` | Call an in-app inspector method | Tree query parameters: -| Parameter | Values | -| --------------- | --------------------------------------------------------------------------------------------------------- | -| `source` | `auto`, `nativescript`, `react-native`, `flutter`, `swiftui`, `uikit`, `native-ax`, `android-uiautomator` | -| `maxDepth` | Integer depth limit | -| `includeHidden` | `true` or `false` | +| Parameter | Values | +| ----------------- | --------------------------------------------------------------------------------------------------------- | +| `source` | `auto`, `nativescript`, `react-native`, `flutter`, `swiftui`, `uikit`, `native-ax`, `android-uiautomator` | +| `maxDepth` | Integer depth limit | +| `includeHidden` | `true` or `false` | +| `interactiveOnly` | `true` keeps actionable elements plus their ancestors | Point query parameters: @@ -219,6 +223,39 @@ Point query parameters: Every tree response reports the `source` used and may include a `fallbackReason`. +Selector endpoints accept compact accessibility selectors: + +```json +{ + "selector": { + "text": "Continue", + "id": "continue-button", + "elementType": "Button", + "enabled": true, + "regex": false + }, + "source": "auto", + "maxDepth": 8, + "limit": 20 +} +``` + +Selectors can match `text`, `id`, `label`, `value`, `elementType`, `index`, `enabled`, `checked`, `focused`, and `selected`. Set `regex: true` to use regular expression matching for string fields. + +`POST /api/simulators/{udid}/query` returns compact matches. `wait-for` and `assert` use the same body shape for positive checks. `wait-for-not` and `assert-not` perform negative checks. + +`POST /api/simulators/{udid}/scroll-until-visible` scrolls and polls until a selector appears: + +```json +{ + "selector": { "text": "Settings" }, + "direction": "down", + "timeoutMs": 10000 +} +``` + +`direction` accepts `up`, `down`, `left`, and `right`. + ## DevTools And WebKit | Method | Path | Purpose | diff --git a/docs/cli/commands.md b/docs/cli/commands.md index f30a19c4..cd51a173 100644 --- a/docs/cli/commands.md +++ b/docs/cli/commands.md @@ -43,96 +43,103 @@ rotate the token and pairing code, then restart the LaunchAgent. ```sh simdeck list simdeck list --format json +simdeck use simdeck boot -simdeck shutdown -simdeck erase +simdeck shutdown +simdeck erase ``` Android emulators appear as IDs such as `android:Pixel_8_API_36`. `list` defaults to compact JSON. Use `--format json` for the full simulator inventory, including paths and display metadata. +`simdeck use ` saves a default simulator for the current project +directory. After that, most device commands can omit ``; explicit UDIDs +still override the default. + ## Apps And URLs ```sh -simdeck install /path/to/App.app -simdeck install /path/to/App.ipa +simdeck install /path/to/App.app +simdeck install /path/to/App.ipa simdeck install android: /path/to/app.apk -simdeck uninstall com.example.App -simdeck launch com.example.App -simdeck open-url https://example.com -simdeck toggle-appearance +simdeck uninstall com.example.App +simdeck launch com.example.App +simdeck open-url https://example.com +simdeck toggle-appearance ``` ## Inspect UI ```sh -simdeck describe -simdeck describe --format agent --max-depth 4 -simdeck describe --format compact-json -simdeck describe --source nativescript -simdeck describe --source react-native -simdeck describe --source flutter -simdeck describe --source uikit -simdeck describe --source native-ax -simdeck describe --point 120,240 -simdeck wait-for --label "Welcome" --timeout-ms 5000 -simdeck assert --id login.button --source auto --max-depth 8 +simdeck describe +simdeck describe --format agent --max-depth 4 +simdeck describe --format agent --max-depth 4 --interactive +simdeck describe --format compact-json +simdeck describe --source nativescript +simdeck describe --source react-native +simdeck describe --source flutter +simdeck describe --source uikit +simdeck describe --source native-ax +simdeck describe --point 120,240 +simdeck wait-for --label "Welcome" --timeout-ms 5000 +simdeck assert --id login.button --source auto --max-depth 8 ``` -Default source selection prefers a connected framework inspector, then the Swift in-app agent, then native accessibility. +Default source selection prefers a connected framework inspector, then the Swift in-app agent, then native accessibility. Use `--interactive` or `-i` to keep actionable elements and the ancestor context needed to find them. For quick agent loops, set the project default once and keep `describe` shallow. ## Performance ```sh -simdeck processes -simdeck stats -simdeck stats --pid 12345 -simdeck stats --watch -simdeck sample -simdeck sample --pid 12345 --seconds 3 +simdeck processes +simdeck stats +simdeck stats --pid 12345 +simdeck stats --watch +simdeck sample +simdeck sample --pid 12345 --seconds 3 ``` Performance data is simulator-only and uses host-process telemetry for matching app, extension, helper, and web-content PIDs. `stats` reports CPU, memory, disk write rate, network receive/send rates, connection count, hang state, and recent crash or termination signals. `sample` captures a short macOS `sample` report for the selected or foreground app process. ## Input -Coordinates are screen points unless `--normalized` is present. +Coordinates are screen points unless `--normalized` is present. `tap "Continue"` is shorthand for a label tap on the selected device. Use `--device ` or `SIMDECK_DEVICE=` for one-off overrides. ```sh -simdeck tap 120 240 -simdeck tap 0.5 0.5 --normalized -simdeck tap --label "Continue" --wait-timeout-ms 5000 -simdeck swipe 200 700 200 200 -simdeck gesture scroll-down -simdeck pinch --start-distance 160 --end-distance 80 -simdeck rotate-gesture --radius 100 --degrees 90 -simdeck type "hello" -simdeck type --file message.txt -simdeck key enter -simdeck key-sequence --keycodes h,e,l,l,o -simdeck key-combo --modifiers cmd --key a +simdeck tap 120 240 +simdeck tap 0.5 0.5 --normalized +simdeck tap --label "Continue" --wait-timeout-ms 5000 +simdeck tap "Continue" +simdeck swipe 200 700 200 200 +simdeck gesture scroll-down +simdeck pinch --start-distance 160 --end-distance 80 +simdeck rotate-gesture --radius 100 --degrees 90 +simdeck type "hello" +simdeck type --file message.txt +simdeck key enter +simdeck key-sequence --keycodes h,e,l,l,o +simdeck key-combo --modifiers cmd --key a ``` System controls: ```sh -simdeck button lock --duration-ms 1000 -simdeck button volume-up -simdeck button action -simdeck button digital-crown -simdeck crown --delta 50 -simdeck dismiss-keyboard -simdeck home -simdeck app-switcher -simdeck rotate-left -simdeck rotate-right +simdeck button lock --duration-ms 1000 +simdeck button volume-up +simdeck button action +simdeck button digital-crown +simdeck crown --delta 50 +simdeck dismiss-keyboard +simdeck home +simdeck app-switcher +simdeck rotate-left +simdeck rotate-right ``` ## Batch ```sh -simdeck batch \ +simdeck batch \ --step "tap --label Continue --wait-timeout-ms 5000" \ --step "type 'hello world'" \ --step "wait-for --label 'hello world' --timeout-ms 5000" @@ -140,24 +147,34 @@ simdeck batch \ Use `wait-for` or `assert` steps instead of fixed sleeps when possible. +## Maestro YAML + +Run common Maestro flows through SimDeck's daemon-backed iOS Simulator API: + +```sh +simdeck maestro test flow.yaml --artifacts-dir artifacts/maestro +``` + +The compatibility runner supports the core local commands: `launchApp`, `openLink`, `tapOn`, `inputText`, `eraseText`, `pressKey`, `assertVisible`, `assertNotVisible`, `scrollUntilVisible`, `swipe`, `takeScreenshot`, and `waitForAnimationToEnd`. + ## Evidence ```sh -simdeck screenshot --output screen.png -simdeck screenshot --with-bezel --output screen-bezel.png -simdeck screenshot --stdout > screen.png -simdeck record --seconds 5 --output screen-recording.mp4 -simdeck record --seconds 5 --stdout > screen-recording.mp4 -simdeck pasteboard set "hello" -simdeck pasteboard get -simdeck logs --seconds 30 --limit 200 -simdeck chrome-profile +simdeck screenshot --output screen.png +simdeck screenshot --with-bezel --output screen-bezel.png +simdeck screenshot --stdout > screen.png +simdeck record --seconds 5 --output screen-recording.mp4 +simdeck record --seconds 5 --stdout > screen-recording.mp4 +simdeck pasteboard set "hello" +simdeck pasteboard get +simdeck logs --seconds 30 --limit 200 +simdeck chrome-profile ``` Diagnostic iOS H.264 stream: ```sh -simdeck stream --frames 120 > stream.h264 +simdeck stream --frames 120 > stream.h264 ``` ## Studio And Providers diff --git a/docs/cli/flags.md b/docs/cli/flags.md index 12edc2e5..2adc1d0f 100644 --- a/docs/cli/flags.md +++ b/docs/cli/flags.md @@ -10,21 +10,26 @@ simdeck daemon start --help ## Global -| Flag | Env | Purpose | -| -------------------- | -------------------- | -------------------------------- | -| `--server-url ` | `SIMDECK_SERVER_URL` | Target a specific running daemon | +| Flag | Env | Purpose | +| --------------------- | -------------------- | -------------------------------- | +| `--server-url ` | `SIMDECK_SERVER_URL` | Target a specific running daemon | +| `--device ` | `SIMDECK_DEVICE` | One-off simulator override | + +`SIMDECK_UDID` is also accepted for compatibility. Device commands resolve in +this order: positional UDID, `--device`, `SIMDECK_DEVICE`, `SIMDECK_UDID`, the +project default from `simdeck use `, then auto-inference from the daemon. ## Server Options Used by `simdeck ui`, `daemon start`, `daemon restart`, `service on`, and `service restart`. | Flag | Default | Notes | -| ---------------------------- | -------------- | ----------------------------------------------------------------------------------- | ------ | ------------ | +| ---------------------------- | -------------- | ----------------------------------------------------------------------------------- | | `--port ` | `4310` | HTTP port. LaunchAgent service commands probe up to 4320 when this port is occupied | | `--bind ` | `127.0.0.1` | Use `0.0.0.0` or `::` for LAN access | | `--advertise-host ` | detected | Host printed for remote browsers | | `--client-root ` | bundled client | Static client directory | -| `--video-codec auto | hardware | software` | `auto` | Encoder mode | +| `--video-codec ` | `auto` | `auto`, `hardware`, or `software` | | `--stream-quality ` | `full` | `full`, `balanced`, `economy`, `low`, `tiny`, `ci-software`, and related profiles | | `--local-stream-fps ` | `60` | Local stream frame target | | `--low-latency` | off | Conservative software H.264 profile | @@ -32,14 +37,15 @@ Used by `simdeck ui`, `daemon start`, `daemon restart`, `service on`, and `servi ## `describe` -| Flag | Purpose | -| ------------------ | ------------------------------------------------- | ------------ | ------------------- | ----- | --------- | -------------------- | --------------------- | -| `--format json | compact-json | agent` | Choose output shape | -| `--source auto | nativescript | react-native | flutter | uikit | native-ax | android-uiautomator` | Pick inspector source | -| `--max-depth ` | Trim hierarchy depth | -| `--include-hidden` | Include hidden nodes when supported | -| `--point ,` | Describe the element at a screen point | -| `--direct` | Skip daemon and use native accessibility directly | +| Flag | Purpose | +| --------------------- | ------------------------------------------------------------------------------------------------- | +| `--format ` | `json`, `compact-json`, or `agent` | +| `--source ` | `auto`, `nativescript`, `react-native`, `flutter`, `uikit`, `native-ax`, or `android-uiautomator` | +| `--max-depth ` | Trim hierarchy depth | +| `--include-hidden` | Include hidden nodes when supported | +| `-i`, `--interactive` | Keep only actionable elements plus ancestors | +| `--point ,` | Describe the element at a screen point | +| `--direct` | Skip daemon and use native accessibility directly | ## Input diff --git a/docs/cli/index.md b/docs/cli/index.md index d75fa14e..9d0e32a5 100644 --- a/docs/cli/index.md +++ b/docs/cli/index.md @@ -21,6 +21,11 @@ simdeck [SIMULATOR_NAME_OR_UDID] simdeck [--server-url ] [options] ``` +Use `simdeck use ` once per project directory to make that simulator the +default for later device commands. Most commands accept `[]`; `--device`, +`SIMDECK_DEVICE`, and `SIMDECK_UDID` override the saved project default when a +one-off target is needed. + Use `--server-url` or `SIMDECK_SERVER_URL` when a script should target a specific daemon: ```sh @@ -31,21 +36,25 @@ SIMDECK_SERVER_URL=http://127.0.0.1:4310 simdeck list ```sh simdeck list +simdeck use simdeck boot -simdeck install /path/to/App.app -simdeck install /path/to/App.ipa -simdeck launch com.example.App -simdeck open-url https://example.com -simdeck tap --label "Continue" --wait-timeout-ms 5000 -simdeck describe --format agent --max-depth 3 -simdeck screenshot --output screen.png -simdeck screenshot --with-bezel --output screen-bezel.png -simdeck record --seconds 5 --output screen-recording.mp4 -simdeck logs --seconds 30 --limit 200 -simdeck stats -simdeck sample --seconds 3 +simdeck install /path/to/App.app +simdeck install /path/to/App.ipa +simdeck launch com.example.App +simdeck open-url https://example.com +simdeck tap --label "Continue" --wait-timeout-ms 5000 +simdeck tap "Continue" +simdeck describe --format agent --max-depth 3 --interactive +simdeck screenshot --output screen.png +simdeck screenshot --with-bezel --output screen-bezel.png +simdeck record --seconds 5 --output screen-recording.mp4 +simdeck logs --seconds 30 --limit 200 +simdeck stats +simdeck sample --seconds 3 ``` +The explicit form still works, for example `simdeck launch com.example.App`. + Most successful commands print JSON so they can be piped into tools such as `jq`. ## Help diff --git a/docs/guide/index.md b/docs/guide/index.md index 62eb19c2..b63bd291 100644 --- a/docs/guide/index.md +++ b/docs/guide/index.md @@ -25,12 +25,14 @@ Open the local URL, pick a device, and use the toolbar or CLI commands: ```sh simdeck list +simdeck use simdeck boot -simdeck install /path/to/App.app -simdeck install /path/to/App.ipa -simdeck launch com.example.App -simdeck tap --label "Continue" --wait-timeout-ms 5000 -simdeck describe --format agent --max-depth 3 +simdeck install /path/to/App.app +simdeck install /path/to/App.ipa +simdeck launch com.example.App +simdeck tap --label "Continue" --wait-timeout-ms 5000 +simdeck tap "Continue" +simdeck describe --format agent --max-depth 3 --interactive ``` Use `simdeck -d` for a detached background daemon, `simdeck -k` to stop it, and `simdeck -r` to restart it. diff --git a/docs/guide/quick-start.md b/docs/guide/quick-start.md index 51ef7f9e..a4a2a995 100644 --- a/docs/guide/quick-start.md +++ b/docs/guide/quick-start.md @@ -31,18 +31,21 @@ The UI lists available iOS Simulators and Android emulators. You can also use th ```sh simdeck list +simdeck use simdeck boot ``` -Android emulator IDs are prefixed with `android:`. +`simdeck use ` saves the simulator default for this project directory so +later device commands can omit the UDID. Android emulator IDs are prefixed with +`android:`. ## 3. Install And Launch An App ```sh -simdeck install /path/to/App.app -simdeck install /path/to/App.ipa -simdeck launch com.example.App -simdeck open-url myapp://debug +simdeck install /path/to/App.app +simdeck install /path/to/App.ipa +simdeck launch com.example.App +simdeck open-url myapp://debug ``` For Android: @@ -57,16 +60,17 @@ simdeck launch android: com.example.app Use coordinates when you know them: ```sh -simdeck tap 120 240 -simdeck swipe 200 700 200 200 -simdeck type "hello" +simdeck tap 120 240 +simdeck swipe 200 700 200 200 +simdeck type "hello" ``` Use selectors when you want automation to wait for UI state: ```sh -simdeck tap --label "Continue" --wait-timeout-ms 5000 -simdeck describe --format agent --max-depth 3 +simdeck tap --label "Continue" --wait-timeout-ms 5000 +simdeck tap "Continue" +simdeck describe --format agent --max-depth 3 --interactive ``` ## 5. Keep It Running In The Background diff --git a/docs/guide/testing.md b/docs/guide/testing.md index 28c59e8c..92959d28 100644 --- a/docs/guide/testing.md +++ b/docs/guide/testing.md @@ -25,7 +25,7 @@ try { } ``` -`connect()` starts the project daemon if needed, reuses a healthy daemon, and only stops daemons it started itself. Pass `udid` to `connect()` to make it the default for session methods; methods still accept an explicit UDID as their first argument. +`connect()` starts the project daemon if needed, reuses a healthy daemon, and only stops daemons it started itself. Pass `udid` to `connect()` to make it the default for session methods; methods still accept an explicit UDID as their first argument. Use `sim.device("")` to create a session bound to another simulator. ## Useful Test Methods @@ -39,10 +39,23 @@ try { | `typeText()`, `key()`, `keySequence()` | Text and keyboard input | | `button()`, `home()`, `appSwitcher()` | System controls | | `tree()`, `query()`, `waitFor()`, `assert()` | UI state checks | +| `waitForNot()`, `assertNot()` | Negative UI state checks | +| `scrollUntilVisible()` | Scroll until a selector exists | | `screenshot()`, `record()`, `logs()` | Evidence capture | | `batch()` | Multi-step actions | -Selectors can match `id`, `label`, `value`, or `type`. +Selectors can match `text`, `id`, `label`, `value`, `type`, `index`, `enabled`, `checked`, `focused`, or `selected`. Set `regex: true` to treat string selector fields as regular expressions. + +## Maestro-Compatible YAML + +The CLI includes a compatibility runner for common Maestro YAML flows: + +```sh +simdeck use +simdeck maestro test flow.yaml --artifacts-dir artifacts/maestro +``` + +Supported commands include `launchApp`, `openLink`, `tapOn`, `inputText`, `eraseText`, `pressKey`, `assertVisible`, `assertNotVisible`, `scrollUntilVisible`, `swipe`, `takeScreenshot`, and `waitForAnimationToEnd`. Unsupported Maestro commands fail clearly so the flow can be adjusted or the compatibility layer can be expanded. ## Repository Tests diff --git a/docs/guide/troubleshooting.md b/docs/guide/troubleshooting.md index 3a32fc6f..503b8c19 100644 --- a/docs/guide/troubleshooting.md +++ b/docs/guide/troubleshooting.md @@ -161,10 +161,10 @@ The fallback is expected when no in-app inspector is available. Check: Use a forced source to see the failure reason: ```sh -simdeck describe --source nativescript -simdeck describe --source react-native -simdeck describe --source flutter -simdeck describe --source uikit +simdeck describe --source nativescript +simdeck describe --source react-native +simdeck describe --source flutter +simdeck describe --source uikit ``` ### NativeScript inspector does not connect diff --git a/docs/inspector/index.md b/docs/inspector/index.md index ad62d75b..69925524 100644 --- a/docs/inspector/index.md +++ b/docs/inspector/index.md @@ -20,11 +20,12 @@ Use the built-in accessibility fallback for any app. Add an in-app inspector whe ```sh simdeck describe simdeck describe --format agent --max-depth 3 +simdeck describe --format agent --max-depth 3 --interactive simdeck describe --source native-ax simdeck describe --source react-native ``` -`auto` source selection uses the best available source and falls back to accessibility. +`auto` source selection uses the best available source and falls back to accessibility. Add `--interactive` or `-i` for a smaller agent-oriented tree of actionable elements plus ancestors. ## Use From The Browser diff --git a/packages/simdeck-test/dist/index.d.ts b/packages/simdeck-test/dist/index.d.ts index c9326e5f..bc76a141 100644 --- a/packages/simdeck-test/dist/index.d.ts +++ b/packages/simdeck-test/dist/index.d.ts @@ -1,147 +1,163 @@ export type SimDeckLaunchOptions = { - cliPath?: string; - projectRoot?: string; - keepDaemon?: boolean; - isolated?: boolean; - port?: number; - videoCodec?: "auto" | "hardware" | "software" | "h264-software"; - udid?: string; + cliPath?: string; + projectRoot?: string; + keepDaemon?: boolean; + isolated?: boolean; + port?: number; + videoCodec?: "auto" | "hardware" | "software" | "h264-software"; + udid?: string; }; export type QueryOptions = { - source?: - | "auto" - | "nativescript" - | "react-native" - | "flutter" - | "uikit" - | "native-ax" - | "android-uiautomator"; - maxDepth?: number; - includeHidden?: boolean; + source?: "auto" | "nativescript" | "react-native" | "flutter" | "swiftui" | "uikit" | "native-ax" | "android-uiautomator"; + maxDepth?: number; + includeHidden?: boolean; + interactiveOnly?: boolean; }; export type ElementSelector = { - id?: string; - label?: string; - value?: string; - type?: string; + text?: string; + id?: string; + label?: string; + value?: string; + type?: string; + index?: number; + enabled?: boolean; + checked?: boolean; + focused?: boolean; + selected?: boolean; + regex?: boolean; }; export type TapOptions = QueryOptions & { - durationMs?: number; - waitTimeoutMs?: number; - pollMs?: number; + durationMs?: number; + waitTimeoutMs?: number; + pollMs?: number; }; export type SwipeOptions = { - durationMs?: number; - steps?: number; + durationMs?: number; + steps?: number; }; export type GestureOptions = SwipeOptions & { - delta?: number; + delta?: number; }; export type TypeTextOptions = { - delayMs?: number; + delayMs?: number; }; export type KeySequenceOptions = { - delayMs?: number; + delayMs?: number; }; export type LogsOptions = { - backfill?: boolean; - seconds?: number; - limit?: number; - levels?: string[]; - processes?: string[]; - q?: string; + backfill?: boolean; + seconds?: number; + limit?: number; + levels?: string[]; + processes?: string[]; + q?: string; }; export type ScreenshotOptions = { - bezel?: boolean; - withBezel?: boolean; + bezel?: boolean; + withBezel?: boolean; }; export type ScreenRecordingOptions = { - seconds?: number; + seconds?: number; }; type DeviceMethod = { - (udid: string, ...args: TArgs): TResult; - (...args: TArgs): TResult; + (udid: string, ...args: TArgs): TResult; + (...args: TArgs): TResult; }; export type SimDeckSession = { - endpoint: string; - pid: number; - projectRoot: string; - list(): Promise; - boot: DeviceMethod<[], Promise>; - shutdown: DeviceMethod<[], Promise>; - erase: DeviceMethod<[], Promise>; - install: DeviceMethod<[appPath: string], Promise>; - uninstall: DeviceMethod<[bundleId: string], Promise>; - launch: DeviceMethod<[bundleId: string], Promise>; - openUrl: DeviceMethod<[url: string], Promise>; - tap: DeviceMethod<[x: number, y: number], Promise>; - tapElement: DeviceMethod< - [selector: ElementSelector, options?: TapOptions], - Promise - >; - touch: DeviceMethod<[x: number, y: number, phase: string], Promise>; - swipe: DeviceMethod< - [ - startX: number, - startY: number, - endX: number, - endY: number, - options?: SwipeOptions, - ], - Promise - >; - gesture: DeviceMethod< - [preset: string, options?: GestureOptions], - Promise - >; - typeText: DeviceMethod< - [text: string, options?: TypeTextOptions], - Promise - >; - key: DeviceMethod<[keyCode: number, modifiers?: number], Promise>; - keySequence: DeviceMethod< - [keyCodes: number[], options?: KeySequenceOptions], - Promise - >; - button: DeviceMethod<[button: string, durationMs?: number], Promise>; - home: DeviceMethod<[], Promise>; - dismissKeyboard: DeviceMethod<[], Promise>; - appSwitcher: DeviceMethod<[], Promise>; - rotateLeft: DeviceMethod<[], Promise>; - rotateRight: DeviceMethod<[], Promise>; - toggleAppearance: DeviceMethod<[], Promise>; - pasteboardSet: DeviceMethod<[text: string], Promise>; - pasteboardGet: DeviceMethod<[], Promise>; - chromeProfile: DeviceMethod<[], Promise>; - logs: DeviceMethod<[options?: LogsOptions], Promise>; - tree: DeviceMethod<[options?: QueryOptions], Promise>; - query: DeviceMethod< - [selector: ElementSelector, options?: QueryOptions], - Promise - >; - assert: DeviceMethod< - [selector: ElementSelector, options?: QueryOptions], - Promise - >; - waitFor: DeviceMethod< - [ - selector: ElementSelector, - options?: QueryOptions & { - timeoutMs?: number; - pollMs?: number; - }, - ], - Promise - >; - batch: DeviceMethod< - [steps: unknown[], continueOnError?: boolean], - Promise - >; - screenshot: DeviceMethod<[options?: ScreenshotOptions], Promise>; - record: DeviceMethod<[options?: ScreenRecordingOptions], Promise>; - close(): void; + endpoint: string; + pid: number; + projectRoot: string; + udid?: string; + device(udid: string): SimDeckSession; + list(): Promise; + boot: DeviceMethod<[], Promise>; + shutdown: DeviceMethod<[], Promise>; + erase: DeviceMethod<[], Promise>; + install: DeviceMethod<[appPath: string], Promise>; + uninstall: DeviceMethod<[bundleId: string], Promise>; + launch: DeviceMethod<[bundleId: string], Promise>; + openUrl: DeviceMethod<[url: string], Promise>; + tap: DeviceMethod<[x: number, y: number], Promise>; + tapElement: DeviceMethod<[ + selector: ElementSelector, + options?: TapOptions + ], Promise>; + touch: DeviceMethod<[x: number, y: number, phase: string], Promise>; + swipe: DeviceMethod<[ + startX: number, + startY: number, + endX: number, + endY: number, + options?: SwipeOptions + ], Promise>; + gesture: DeviceMethod<[ + preset: string, + options?: GestureOptions + ], Promise>; + typeText: DeviceMethod<[ + text: string, + options?: TypeTextOptions + ], Promise>; + key: DeviceMethod<[keyCode: number, modifiers?: number], Promise>; + keySequence: DeviceMethod<[ + keyCodes: number[], + options?: KeySequenceOptions + ], Promise>; + button: DeviceMethod<[button: string, durationMs?: number], Promise>; + home: DeviceMethod<[], Promise>; + dismissKeyboard: DeviceMethod<[], Promise>; + appSwitcher: DeviceMethod<[], Promise>; + rotateLeft: DeviceMethod<[], Promise>; + rotateRight: DeviceMethod<[], Promise>; + toggleAppearance: DeviceMethod<[], Promise>; + pasteboardSet: DeviceMethod<[text: string], Promise>; + pasteboardGet: DeviceMethod<[], Promise>; + chromeProfile: DeviceMethod<[], Promise>; + logs: DeviceMethod<[options?: LogsOptions], Promise>; + tree: DeviceMethod<[options?: QueryOptions], Promise>; + query: DeviceMethod<[ + selector: ElementSelector, + options?: QueryOptions + ], Promise>; + assert: DeviceMethod<[ + selector: ElementSelector, + options?: QueryOptions + ], Promise>; + assertNot: DeviceMethod<[ + selector: ElementSelector, + options?: QueryOptions + ], Promise>; + waitFor: DeviceMethod<[ + selector: ElementSelector, + options?: QueryOptions & { + timeoutMs?: number; + pollMs?: number; + } + ], Promise>; + waitForNot: DeviceMethod<[ + selector: ElementSelector, + options?: QueryOptions & { + timeoutMs?: number; + pollMs?: number; + } + ], Promise>; + scrollUntilVisible: DeviceMethod<[ + selector: ElementSelector, + options?: QueryOptions & { + timeoutMs?: number; + pollMs?: number; + direction?: "up" | "down" | "left" | "right"; + durationMs?: number; + steps?: number; + } + ], Promise>; + batch: DeviceMethod<[ + steps: unknown[], + continueOnError?: boolean + ], Promise>; + screenshot: DeviceMethod<[options?: ScreenshotOptions], Promise>; + record: DeviceMethod<[options?: ScreenRecordingOptions], Promise>; + close(): void; }; -export declare function connect( - options?: SimDeckLaunchOptions, -): Promise; +export declare function connect(options?: SimDeckLaunchOptions): Promise; export {}; diff --git a/packages/simdeck-test/dist/index.js b/packages/simdeck-test/dist/index.js index 8963c7f7..425c641d 100644 --- a/packages/simdeck-test/dist/index.js +++ b/packages/simdeck-test/dist/index.js @@ -6,567 +6,556 @@ import net from "node:net"; import os from "node:os"; import path from "node:path"; export async function connect(options = {}) { - const cliPath = options.cliPath ?? "simdeck"; - const result = options.isolated - ? await startIsolatedDaemon(cliPath, options) - : runJson(cliPath, ["daemon", "start"], { + const cliPath = options.cliPath ?? "simdeck"; + const result = options.isolated + ? await startIsolatedDaemon(cliPath, options) + : runJson(cliPath, ["daemon", "start"], { + cwd: options.projectRoot, + }); + const endpoint = result.url; + const createSession = (defaultUdid) => { + const simulatorPath = (udid, suffix) => `/api/simulators/${encodeURIComponent(udid)}${suffix}`; + const requireUdid = (udid) => { + const resolved = udid ?? defaultUdid; + if (!resolved) { + throw new Error("This SimDeck session method requires a UDID. Pass one as the first argument or call connect({ udid })."); + } + return resolved; + }; + const resolveNoArgDeviceCall = (args) => ({ + udid: requireUdid(typeof args[0] === "string" ? args[0] : undefined), + }); + const resolveStringArgDeviceCall = (args) => { + if (args.length >= 2 && + typeof args[0] === "string" && + typeof args[1] === "string") { + return { udid: args[0], value: args[1], rest: args.slice(2) }; + } + return { + udid: requireUdid(), + value: args[0], + rest: args.slice(1), + }; + }; + const resolveObjectArgDeviceCall = (args) => { + if (typeof args[0] === "string") { + return { udid: args[0], value: args[1], rest: args.slice(2) }; + } + return { udid: requireUdid(), value: args[0], rest: args.slice(1) }; + }; + const resolveOptionalObjectDeviceCall = (args) => { + if (typeof args[0] === "string") { + return { udid: args[0], options: args[1] }; + } + return { udid: requireUdid(), options: args[0] }; + }; + const session = { + endpoint, + pid: result.pid, + projectRoot: result.projectRoot, + udid: defaultUdid, + device: (udid) => createSession(udid), + list: () => requestJson(endpoint, "GET", "/api/simulators"), + boot: (...args) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestJson(endpoint, "POST", simulatorPath(udid, "/boot"), null); + }, + shutdown: (...args) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestJson(endpoint, "POST", simulatorPath(udid, "/shutdown"), null); + }, + erase: (...args) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestJson(endpoint, "POST", simulatorPath(udid, "/erase"), null); + }, + install: (...args) => { + const { udid, value: appPath } = resolveStringArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/install"), { + appPath, + }); + }, + uninstall: (...args) => { + const { udid, value: bundleId } = resolveStringArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/uninstall"), { + bundleId, + }); + }, + launch: (...args) => { + const { udid, value: bundleId } = resolveStringArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/launch"), { + bundleId, + }); + }, + openUrl: (...args) => { + const { udid, value: url } = resolveStringArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/open-url"), { + url, + }); + }, + tap: (...args) => { + const [udid, x, y] = typeof args[0] === "string" + ? [args[0], args[1], args[2]] + : [requireUdid(), args[0], args[1]]; + return requestOk(endpoint, simulatorPath(udid, "/tap"), { + x, + y, + normalized: true, + }); + }, + tapElement: (...args) => { + const { udid, value: selector, rest, } = resolveObjectArgDeviceCall(args); + const [tapOptions] = rest; + return requestOk(endpoint, simulatorPath(udid, "/tap"), { + selector: selectorPayload(selector), + ...tapOptions, + }); + }, + touch: (...args) => { + const [udid, x, y, phase] = typeof args[0] === "string" + ? [args[0], args[1], args[2], args[3]] + : [ + requireUdid(), + args[0], + args[1], + args[2], + ]; + return requestOk(endpoint, simulatorPath(udid, "/touch"), { + x, + y, + phase, + }); + }, + swipe: (...args) => { + const [udid, startX, startY, endX, endY, swipeOptions = {}] = typeof args[0] === "string" + ? [ + args[0], + args[1], + args[2], + args[3], + args[4], + args[5], + ] + : [ + requireUdid(), + args[0], + args[1], + args[2], + args[3], + args[4], + ]; + return requestJson(endpoint, "POST", simulatorPath(udid, "/batch"), { + steps: [ + { + action: "swipe", + startX, + startY, + endX, + endY, + ...swipeOptions, + }, + ], + }); + }, + gesture: (...args) => { + const { udid, value: preset, rest } = resolveStringArgDeviceCall(args); + const [gestureOptions = {}] = rest; + return requestJson(endpoint, "POST", simulatorPath(udid, "/batch"), { + steps: [ + { + action: "gesture", + preset, + ...gestureOptions, + }, + ], + }); + }, + typeText: (...args) => { + const { udid, value: text, rest } = resolveStringArgDeviceCall(args); + const [typeOptions = {}] = rest; + return requestJson(endpoint, "POST", simulatorPath(udid, "/batch"), { + steps: [ + { + action: "type", + text, + ...typeOptions, + }, + ], + }); + }, + key: (...args) => { + const [udid, keyCode, modifiers = 0] = typeof args[0] === "string" + ? [args[0], args[1], args[2]] + : [requireUdid(), args[0], args[1]]; + return requestOk(endpoint, simulatorPath(udid, "/key"), { + keyCode, + modifiers, + }); + }, + keySequence: (...args) => { + const { udid, value: keyCodes, rest, } = resolveObjectArgDeviceCall(args); + const [keySequenceOptions = {}] = rest; + return requestOk(endpoint, simulatorPath(udid, "/key-sequence"), { + keyCodes, + ...keySequenceOptions, + }); + }, + button: (...args) => { + const { udid, value: button, rest } = resolveStringArgDeviceCall(args); + const [durationMs = 0] = rest; + return requestOk(endpoint, simulatorPath(udid, "/button"), { + button, + durationMs, + }); + }, + home: (...args) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/home"), null); + }, + dismissKeyboard: (...args) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/dismiss-keyboard"), null); + }, + appSwitcher: (...args) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/app-switcher"), null); + }, + rotateLeft: (...args) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/rotate-left"), null); + }, + rotateRight: (...args) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/rotate-right"), null); + }, + toggleAppearance: (...args) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/toggle-appearance"), null); + }, + pasteboardSet: (...args) => { + const { udid, value: text } = resolveStringArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/pasteboard"), { + text, + }); + }, + pasteboardGet: async (...args) => { + const { udid } = resolveNoArgDeviceCall(args); + const result = await requestJson(endpoint, "GET", simulatorPath(udid, "/pasteboard")); + return result.text ?? ""; + }, + chromeProfile: (...args) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestJson(endpoint, "GET", simulatorPath(udid, "/chrome-profile")); + }, + logs: async (...args) => { + const { udid, value: logsOptions } = typeof args[0] === "string" + ? { udid: args[0], value: args[1] } + : { + udid: requireUdid(), + value: args[0], + }; + const result = await requestJson(endpoint, "GET", simulatorPath(udid, `/logs?${logsQuery(logsOptions)}`)); + return result.entries ?? []; + }, + tree: (...args) => { + const { udid, value: treeOptions } = typeof args[0] === "string" + ? { udid: args[0], value: args[1] } + : { + udid: requireUdid(), + value: args[0], + }; + return requestJson(endpoint, "GET", simulatorPath(udid, `/accessibility-tree?${treeQuery(treeOptions)}`)); + }, + query: async (...args) => { + const { udid, value: selector, rest, } = resolveObjectArgDeviceCall(args); + const [treeOptions] = rest; + const result = await requestJson(endpoint, "POST", simulatorPath(udid, "/query"), { + selector: selectorPayload(selector), + ...treeOptions, + }); + return result.matches; + }, + assert: (...args) => { + const { udid, value: selector, rest, } = resolveObjectArgDeviceCall(args); + const [assertOptions] = rest; + return requestJson(endpoint, "POST", simulatorPath(udid, "/assert"), { + selector: selectorPayload(selector), + ...assertOptions, + }); + }, + assertNot: (...args) => { + const { udid, value: selector, rest, } = resolveObjectArgDeviceCall(args); + const [assertOptions] = rest; + return requestJson(endpoint, "POST", simulatorPath(udid, "/assert-not"), { + selector: selectorPayload(selector), + ...assertOptions, + }); + }, + waitFor: (...args) => { + const { udid, value: selector, rest, } = resolveObjectArgDeviceCall(args); + const [waitOptions] = rest; + return requestJson(endpoint, "POST", simulatorPath(udid, "/wait-for"), { + selector: selectorPayload(selector), + ...waitOptions, + }); + }, + waitForNot: (...args) => { + const { udid, value: selector, rest, } = resolveObjectArgDeviceCall(args); + const [waitOptions] = rest; + return requestJson(endpoint, "POST", simulatorPath(udid, "/wait-for-not"), { + selector: selectorPayload(selector), + ...waitOptions, + }); + }, + scrollUntilVisible: (...args) => { + const { udid, value: selector, rest, } = resolveObjectArgDeviceCall(args); + const [scrollOptions] = rest; + return requestJson(endpoint, "POST", simulatorPath(udid, "/scroll-until-visible"), { + selector: selectorPayload(selector), + ...scrollOptions, + }); + }, + batch: (...args) => { + const { udid, value: steps, rest, } = resolveObjectArgDeviceCall(args); + const [continueOnError = false] = rest; + return requestJson(endpoint, "POST", simulatorPath(udid, "/batch"), { + steps, + continueOnError, + }); + }, + screenshot: (...args) => { + const { udid, options } = resolveOptionalObjectDeviceCall(args); + const params = new URLSearchParams(); + if (options?.withBezel ?? options?.bezel) { + params.set("bezel", "true"); + } + const query = params.toString(); + return requestBuffer(endpoint, simulatorPath(udid, `/screenshot.png${query ? `?${query}` : ""}`)); + }, + record: (...args) => { + const { udid, options } = resolveOptionalObjectDeviceCall(args); + return requestBuffer(endpoint, simulatorPath(udid, "/screen-recording"), "POST", { + seconds: options?.seconds ?? 5, + }); + }, + close: () => { + if (options.keepDaemon) { + return; + } + if (result.child) { + result.child.kill(); + if (result.isolatedRoot) { + fs.rmSync(result.isolatedRoot, { recursive: true, force: true }); + } + return; + } + if (result.started) { + spawnSync(cliPath, ["daemon", "stop"], { cwd: options.projectRoot }); + } + }, + }; + return session; + }; + return createSession(options.udid); +} +async function startIsolatedDaemon(cliPath, options) { + const port = options.port ?? (await freePortPair()); + const projectRoot = fs.mkdtempSync(path.join(os.tmpdir(), "simdeck-test-project-")); + const metadataPath = path.join(os.tmpdir(), `simdeck-test-${process.pid}-${Date.now()}-${crypto.randomUUID()}.json`); + const accessToken = crypto.randomBytes(32).toString("hex"); + const child = spawn(cliPath, [ + "daemon", + "run", + "--project-root", + projectRoot, + "--metadata-path", + metadataPath, + "--port", + String(port), + "--bind", + "127.0.0.1", + "--access-token", + accessToken, + "--video-codec", + options.videoCodec ?? "software", + ], { cwd: options.projectRoot, - }); - const endpoint = result.url; - const defaultUdid = options.udid; - const simulatorPath = (udid, suffix) => - `/api/simulators/${encodeURIComponent(udid)}${suffix}`; - const requireUdid = (udid) => { - const resolved = udid ?? defaultUdid; - if (!resolved) { - throw new Error( - "This SimDeck session method requires a UDID. Pass one as the first argument or call connect({ udid }).", - ); + stdio: ["ignore", "pipe", "pipe"], + }); + const output = captureChildOutput(child); + const url = `http://127.0.0.1:${port}`; + try { + await waitForHealth(url, child, output); } - return resolved; - }; - const resolveNoArgDeviceCall = (args) => ({ - udid: requireUdid(typeof args[0] === "string" ? args[0] : undefined), - }); - const resolveStringArgDeviceCall = (args) => { - if ( - args.length >= 2 && - typeof args[0] === "string" && - typeof args[1] === "string" - ) { - return { udid: args[0], value: args[1], rest: args.slice(2) }; + catch (error) { + child.kill(); + fs.rmSync(projectRoot, { recursive: true, force: true }); + throw error; } return { - udid: requireUdid(), - value: args[0], - rest: args.slice(1), - }; - }; - const resolveObjectArgDeviceCall = (args) => { - if (typeof args[0] === "string") { - return { udid: args[0], value: args[1], rest: args.slice(2) }; - } - return { udid: requireUdid(), value: args[0], rest: args.slice(1) }; - }; - const resolveOptionalObjectDeviceCall = (args) => { - if (typeof args[0] === "string") { - return { udid: args[0], options: args[1] }; - } - return { udid: requireUdid(), options: args[0] }; - }; - const session = { - endpoint, - pid: result.pid, - projectRoot: result.projectRoot, - list: () => requestJson(endpoint, "GET", "/api/simulators"), - boot: (...args) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestJson(endpoint, "POST", simulatorPath(udid, "/boot"), null); - }, - shutdown: (...args) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestJson( - endpoint, - "POST", - simulatorPath(udid, "/shutdown"), - null, - ); - }, - erase: (...args) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestJson(endpoint, "POST", simulatorPath(udid, "/erase"), null); - }, - install: (...args) => { - const { udid, value: appPath } = resolveStringArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/install"), { - appPath, - }); - }, - uninstall: (...args) => { - const { udid, value: bundleId } = resolveStringArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/uninstall"), { - bundleId, - }); - }, - launch: (...args) => { - const { udid, value: bundleId } = resolveStringArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/launch"), { - bundleId, - }); - }, - openUrl: (...args) => { - const { udid, value: url } = resolveStringArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/open-url"), { + ok: true, + projectRoot, + pid: child.pid ?? 0, url, - }); - }, - tap: (...args) => { - const [udid, x, y] = - typeof args[0] === "string" - ? [args[0], args[1], args[2]] - : [requireUdid(), args[0], args[1]]; - return requestOk(endpoint, simulatorPath(udid, "/tap"), { - x, - y, - normalized: true, - }); - }, - tapElement: (...args) => { - const { udid, value: selector, rest } = resolveObjectArgDeviceCall(args); - const [tapOptions] = rest; - return requestOk(endpoint, simulatorPath(udid, "/tap"), { - selector: selectorPayload(selector), - ...tapOptions, - }); - }, - touch: (...args) => { - const [udid, x, y, phase] = - typeof args[0] === "string" - ? [args[0], args[1], args[2], args[3]] - : [requireUdid(), args[0], args[1], args[2]]; - return requestOk(endpoint, simulatorPath(udid, "/touch"), { - x, - y, - phase, - }); - }, - swipe: (...args) => { - const [udid, startX, startY, endX, endY, swipeOptions = {}] = - typeof args[0] === "string" - ? [args[0], args[1], args[2], args[3], args[4], args[5]] - : [requireUdid(), args[0], args[1], args[2], args[3], args[4]]; - return requestJson(endpoint, "POST", simulatorPath(udid, "/batch"), { - steps: [ - { - action: "swipe", - startX, - startY, - endX, - endY, - ...swipeOptions, - }, - ], - }); - }, - gesture: (...args) => { - const { udid, value: preset, rest } = resolveStringArgDeviceCall(args); - const [gestureOptions = {}] = rest; - return requestJson(endpoint, "POST", simulatorPath(udid, "/batch"), { - steps: [ - { - action: "gesture", - preset, - ...gestureOptions, - }, - ], - }); - }, - typeText: (...args) => { - const { udid, value: text, rest } = resolveStringArgDeviceCall(args); - const [typeOptions = {}] = rest; - return requestJson(endpoint, "POST", simulatorPath(udid, "/batch"), { - steps: [ - { - action: "type", - text, - ...typeOptions, - }, - ], - }); - }, - key: (...args) => { - const [udid, keyCode, modifiers = 0] = - typeof args[0] === "string" - ? [args[0], args[1], args[2]] - : [requireUdid(), args[0], args[1]]; - return requestOk(endpoint, simulatorPath(udid, "/key"), { - keyCode, - modifiers, - }); - }, - keySequence: (...args) => { - const { udid, value: keyCodes, rest } = resolveObjectArgDeviceCall(args); - const [keySequenceOptions = {}] = rest; - return requestOk(endpoint, simulatorPath(udid, "/key-sequence"), { - keyCodes, - ...keySequenceOptions, - }); - }, - button: (...args) => { - const { udid, value: button, rest } = resolveStringArgDeviceCall(args); - const [durationMs = 0] = rest; - return requestOk(endpoint, simulatorPath(udid, "/button"), { - button, - durationMs, - }); - }, - home: (...args) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/home"), null); - }, - dismissKeyboard: (...args) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestOk( - endpoint, - simulatorPath(udid, "/dismiss-keyboard"), - null, - ); - }, - appSwitcher: (...args) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/app-switcher"), null); - }, - rotateLeft: (...args) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/rotate-left"), null); - }, - rotateRight: (...args) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/rotate-right"), null); - }, - toggleAppearance: (...args) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestOk( - endpoint, - simulatorPath(udid, "/toggle-appearance"), - null, - ); - }, - pasteboardSet: (...args) => { - const { udid, value: text } = resolveStringArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/pasteboard"), { - text, - }); - }, - pasteboardGet: async (...args) => { - const { udid } = resolveNoArgDeviceCall(args); - const result = await requestJson( - endpoint, - "GET", - simulatorPath(udid, "/pasteboard"), - ); - return result.text ?? ""; - }, - chromeProfile: (...args) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestJson( - endpoint, - "GET", - simulatorPath(udid, "/chrome-profile"), - ); - }, - logs: async (...args) => { - const { udid, value: logsOptions } = - typeof args[0] === "string" - ? { udid: args[0], value: args[1] } - : { udid: requireUdid(), value: args[0] }; - const result = await requestJson( - endpoint, - "GET", - simulatorPath(udid, `/logs?${logsQuery(logsOptions)}`), - ); - return result.entries ?? []; - }, - tree: (...args) => { - const { udid, value: treeOptions } = - typeof args[0] === "string" - ? { udid: args[0], value: args[1] } - : { udid: requireUdid(), value: args[0] }; - return requestJson( - endpoint, - "GET", - simulatorPath(udid, `/accessibility-tree?${treeQuery(treeOptions)}`), - ); - }, - query: async (...args) => { - const { udid, value: selector, rest } = resolveObjectArgDeviceCall(args); - const [treeOptions] = rest; - const result = await requestJson( - endpoint, - "POST", - simulatorPath(udid, "/query"), - { - selector: selectorPayload(selector), - ...treeOptions, - }, - ); - return result.matches; - }, - assert: (...args) => { - const { udid, value: selector, rest } = resolveObjectArgDeviceCall(args); - const [assertOptions] = rest; - return requestJson(endpoint, "POST", simulatorPath(udid, "/assert"), { - selector: selectorPayload(selector), - ...assertOptions, - }); - }, - waitFor: (...args) => { - const { udid, value: selector, rest } = resolveObjectArgDeviceCall(args); - const [waitOptions] = rest; - return requestJson(endpoint, "POST", simulatorPath(udid, "/wait-for"), { - selector: selectorPayload(selector), - ...waitOptions, - }); - }, - batch: (...args) => { - const { udid, value: steps, rest } = resolveObjectArgDeviceCall(args); - const [continueOnError = false] = rest; - return requestJson(endpoint, "POST", simulatorPath(udid, "/batch"), { - steps, - continueOnError, - }); - }, - screenshot: (...args) => { - const { udid, options } = resolveOptionalObjectDeviceCall(args); - const params = new URLSearchParams(); - if (options?.withBezel ?? options?.bezel) { - params.set("bezel", "true"); - } - const query = params.toString(); - return requestBuffer( - endpoint, - simulatorPath(udid, `/screenshot.png${query ? `?${query}` : ""}`), - ); - }, - record: (...args) => { - const { udid, options } = resolveOptionalObjectDeviceCall(args); - return requestBuffer( - endpoint, - simulatorPath(udid, "/screen-recording"), - "POST", - { - seconds: options?.seconds ?? 5, - }, - ); - }, - close: () => { - if (options.keepDaemon) { - return; - } - if (result.child) { - result.child.kill(); - if (result.isolatedRoot) { - fs.rmSync(result.isolatedRoot, { recursive: true, force: true }); - } - return; - } - if (result.started) { - spawnSync(cliPath, ["daemon", "stop"], { cwd: options.projectRoot }); - } - }, - }; - return session; -} -async function startIsolatedDaemon(cliPath, options) { - const port = options.port ?? (await freePortPair()); - const projectRoot = fs.mkdtempSync( - path.join(os.tmpdir(), "simdeck-test-project-"), - ); - const metadataPath = path.join( - os.tmpdir(), - `simdeck-test-${process.pid}-${Date.now()}-${crypto.randomUUID()}.json`, - ); - const accessToken = crypto.randomBytes(32).toString("hex"); - const child = spawn( - cliPath, - [ - "daemon", - "run", - "--project-root", - projectRoot, - "--metadata-path", - metadataPath, - "--port", - String(port), - "--bind", - "127.0.0.1", - "--access-token", - accessToken, - "--video-codec", - options.videoCodec ?? "software", - ], - { - cwd: options.projectRoot, - stdio: ["ignore", "pipe", "pipe"], - }, - ); - const output = captureChildOutput(child); - const url = `http://127.0.0.1:${port}`; - try { - await waitForHealth(url, child, output); - } catch (error) { - child.kill(); - fs.rmSync(projectRoot, { recursive: true, force: true }); - throw error; - } - return { - ok: true, - projectRoot, - pid: child.pid ?? 0, - url, - started: true, - child, - isolatedRoot: projectRoot, - }; + started: true, + child, + isolatedRoot: projectRoot, + }; } async function waitForHealth(endpoint, child, output) { - const deadline = Date.now() + 60_000; - let lastError; - while (Date.now() < deadline) { - if (child.exitCode !== null) { - throw new Error( - `SimDeck isolated daemon exited with ${child.exitCode}.\n${output()}`, - ); - } - try { - await requestJson(endpoint, "GET", "/api/health"); - return; - } catch (error) { - lastError = error; - await new Promise((resolve) => setTimeout(resolve, 50)); + const deadline = Date.now() + 60_000; + let lastError; + while (Date.now() < deadline) { + if (child.exitCode !== null) { + throw new Error(`SimDeck isolated daemon exited with ${child.exitCode}.\n${output()}`); + } + try { + await requestJson(endpoint, "GET", "/api/health"); + return; + } + catch (error) { + lastError = error; + await new Promise((resolve) => setTimeout(resolve, 50)); + } } - } - throw new Error( - `Timed out waiting for isolated SimDeck daemon: ${lastError instanceof Error ? lastError.message : String(lastError)}\n${output()}`, - ); + throw new Error(`Timed out waiting for isolated SimDeck daemon: ${lastError instanceof Error ? lastError.message : String(lastError)}\n${output()}`); } function captureChildOutput(child) { - const chunks = []; - const append = (source, chunk) => { - chunks.push(`[${source}] ${chunk.toString("utf8")}`); - while (chunks.join("").length > 16_384) { - chunks.shift(); - } - }; - child.stdout?.on("data", (chunk) => append("stdout", chunk)); - child.stderr?.on("data", (chunk) => append("stderr", chunk)); - return () => chunks.join("").trim(); + const chunks = []; + const append = (source, chunk) => { + chunks.push(`[${source}] ${chunk.toString("utf8")}`); + while (chunks.join("").length > 16_384) { + chunks.shift(); + } + }; + child.stdout?.on("data", (chunk) => append("stdout", chunk)); + child.stderr?.on("data", (chunk) => append("stderr", chunk)); + return () => chunks.join("").trim(); } async function freePortPair() { - for (let attempt = 0; attempt < 100; attempt += 1) { - const port = await freePort(); - if (port < 65535 && (await portAvailable(port + 1))) { - return port; + for (let attempt = 0; attempt < 100; attempt += 1) { + const port = await freePort(); + if (port < 65535 && (await portAvailable(port + 1))) { + return port; + } } - } - throw new Error("Unable to allocate adjacent free TCP ports."); + throw new Error("Unable to allocate adjacent free TCP ports."); } function freePort() { - return new Promise((resolve, reject) => { - const server = net.createServer(); - server.listen(0, "127.0.0.1", () => { - const address = server.address(); - if (!address || typeof address === "string") { - server.close(); - reject(new Error("Unable to allocate a free TCP port.")); - return; - } - const port = address.port; - server.close(() => resolve(port)); + return new Promise((resolve, reject) => { + const server = net.createServer(); + server.listen(0, "127.0.0.1", () => { + const address = server.address(); + if (!address || typeof address === "string") { + server.close(); + reject(new Error("Unable to allocate a free TCP port.")); + return; + } + const port = address.port; + server.close(() => resolve(port)); + }); + server.on("error", reject); }); - server.on("error", reject); - }); } function portAvailable(port) { - return new Promise((resolve) => { - const server = net.createServer(); - server.once("error", () => resolve(false)); - server.listen(port, "127.0.0.1", () => { - server.close(() => resolve(true)); + return new Promise((resolve) => { + const server = net.createServer(); + server.once("error", () => resolve(false)); + server.listen(port, "127.0.0.1", () => { + server.close(() => resolve(true)); + }); }); - }); } function runJson(command, args, options = {}) { - const result = spawnSync(command, args, { - cwd: options.cwd, - encoding: "utf8", - maxBuffer: 1024 * 1024, - }); - if (result.status !== 0) { - throw new Error( - result.stderr.trim() || `${command} ${args.join(" ")} failed`, - ); - } - return JSON.parse(result.stdout); + const result = spawnSync(command, args, { + cwd: options.cwd, + encoding: "utf8", + maxBuffer: 1024 * 1024, + }); + if (result.status !== 0) { + throw new Error(result.stderr.trim() || `${command} ${args.join(" ")} failed`); + } + return JSON.parse(result.stdout); } function requestOk(endpoint, pathName, body) { - return requestJson(endpoint, "POST", pathName, body).then(() => undefined); + return requestJson(endpoint, "POST", pathName, body).then(() => undefined); } function requestJson(endpoint, method, pathName, body) { - return requestBuffer(endpoint, pathName, method, body).then((buffer) => - JSON.parse(buffer.toString("utf8")), - ); + return requestBuffer(endpoint, pathName, method, body).then((buffer) => JSON.parse(buffer.toString("utf8"))); } function requestBuffer(endpoint, pathName, method = "GET", body) { - const url = new URL(pathName, endpoint); - const payload = - body === undefined ? undefined : Buffer.from(JSON.stringify(body)); - return new Promise((resolve, reject) => { - const request = http.request( - url, - { - method, - headers: payload - ? { - "content-type": "application/json", - "content-length": String(payload.length), - origin: endpoint, - } - : { origin: endpoint }, - }, - (response) => { - const chunks = []; - response.on("data", (chunk) => chunks.push(chunk)); - response.on("end", () => { - const buffer = Buffer.concat(chunks); - if ( - (response.statusCode ?? 500) < 200 || - (response.statusCode ?? 500) >= 300 - ) { - reject( - new Error( - `${method} ${pathName} returned ${response.statusCode}: ${buffer.toString("utf8") || response.statusMessage || ""}`, - ), - ); - } else { - resolve(buffer); - } + const url = new URL(pathName, endpoint); + const payload = body === undefined ? undefined : Buffer.from(JSON.stringify(body)); + return new Promise((resolve, reject) => { + const request = http.request(url, { + method, + headers: payload + ? { + "content-type": "application/json", + "content-length": String(payload.length), + origin: endpoint, + } + : { origin: endpoint }, + }, (response) => { + const chunks = []; + response.on("data", (chunk) => chunks.push(chunk)); + response.on("end", () => { + const buffer = Buffer.concat(chunks); + if ((response.statusCode ?? 500) < 200 || + (response.statusCode ?? 500) >= 300) { + reject(new Error(`${method} ${pathName} returned ${response.statusCode}: ${buffer.toString("utf8") || response.statusMessage || ""}`)); + } + else { + resolve(buffer); + } + }); }); - }, - ); - request.on("error", reject); - if (payload) { - request.write(payload); - } - request.end(); - }); + request.on("error", reject); + if (payload) { + request.write(payload); + } + request.end(); + }); } function treeQuery(options = {}) { - const params = new URLSearchParams(); - if (options.source) params.set("source", options.source); - if (options.maxDepth !== undefined) - params.set("maxDepth", String(options.maxDepth)); - if (options.includeHidden) params.set("includeHidden", "true"); - return params.toString(); + const params = new URLSearchParams(); + if (options.source) + params.set("source", options.source); + if (options.maxDepth !== undefined) + params.set("maxDepth", String(options.maxDepth)); + if (options.includeHidden) + params.set("includeHidden", "true"); + if (options.interactiveOnly) + params.set("interactiveOnly", "true"); + return params.toString(); } function logsQuery(options = {}) { - const params = new URLSearchParams(); - if (options.backfill !== undefined) - params.set("backfill", String(options.backfill)); - if (options.seconds !== undefined) - params.set("seconds", String(options.seconds)); - if (options.limit !== undefined) params.set("limit", String(options.limit)); - if (options.levels?.length) params.set("levels", options.levels.join(",")); - if (options.processes?.length) - params.set("processes", options.processes.join(",")); - if (options.q) params.set("q", options.q); - return params.toString(); + const params = new URLSearchParams(); + if (options.backfill !== undefined) + params.set("backfill", String(options.backfill)); + if (options.seconds !== undefined) + params.set("seconds", String(options.seconds)); + if (options.limit !== undefined) + params.set("limit", String(options.limit)); + if (options.levels?.length) + params.set("levels", options.levels.join(",")); + if (options.processes?.length) + params.set("processes", options.processes.join(",")); + if (options.q) + params.set("q", options.q); + return params.toString(); } function selectorPayload(selector) { - return { - id: selector.id, - label: selector.label, - value: selector.value, - elementType: selector.type, - }; + return { + text: selector.text, + id: selector.id, + label: selector.label, + value: selector.value, + elementType: selector.type, + index: selector.index, + enabled: selector.enabled, + checked: selector.checked, + focused: selector.focused, + selected: selector.selected, + regex: selector.regex, + }; } diff --git a/packages/simdeck-test/src/index.ts b/packages/simdeck-test/src/index.ts index 71d22c57..24bbe28a 100644 --- a/packages/simdeck-test/src/index.ts +++ b/packages/simdeck-test/src/index.ts @@ -22,18 +22,27 @@ export type QueryOptions = { | "nativescript" | "react-native" | "flutter" + | "swiftui" | "uikit" | "native-ax" | "android-uiautomator"; maxDepth?: number; includeHidden?: boolean; + interactiveOnly?: boolean; }; export type ElementSelector = { + text?: string; id?: string; label?: string; value?: string; type?: string; + index?: number; + enabled?: boolean; + checked?: boolean; + focused?: boolean; + selected?: boolean; + regex?: boolean; }; export type TapOptions = QueryOptions & { @@ -86,6 +95,8 @@ export type SimDeckSession = { endpoint: string; pid: number; projectRoot: string; + udid?: string; + device(udid: string): SimDeckSession; list(): Promise; boot: DeviceMethod<[], Promise>; shutdown: DeviceMethod<[], Promise>; @@ -143,6 +154,10 @@ export type SimDeckSession = { [selector: ElementSelector, options?: QueryOptions], Promise >; + assertNot: DeviceMethod< + [selector: ElementSelector, options?: QueryOptions], + Promise + >; waitFor: DeviceMethod< [ selector: ElementSelector, @@ -150,6 +165,26 @@ export type SimDeckSession = { ], Promise >; + waitForNot: DeviceMethod< + [ + selector: ElementSelector, + options?: QueryOptions & { timeoutMs?: number; pollMs?: number }, + ], + Promise + >; + scrollUntilVisible: DeviceMethod< + [ + selector: ElementSelector, + options?: QueryOptions & { + timeoutMs?: number; + pollMs?: number; + direction?: "up" | "down" | "left" | "right"; + durationMs?: number; + steps?: number; + }, + ], + Promise + >; batch: DeviceMethod< [steps: unknown[], continueOnError?: boolean], Promise @@ -182,400 +217,482 @@ export async function connect( cwd: options.projectRoot, }); const endpoint = result.url; - const defaultUdid = options.udid; - const simulatorPath = (udid: string, suffix: string) => - `/api/simulators/${encodeURIComponent(udid)}${suffix}`; - const requireUdid = (udid?: string) => { - const resolved = udid ?? defaultUdid; - if (!resolved) { - throw new Error( - "This SimDeck session method requires a UDID. Pass one as the first argument or call connect({ udid }).", - ); - } - return resolved; - }; - const resolveNoArgDeviceCall = (args: unknown[]) => ({ - udid: requireUdid(typeof args[0] === "string" ? args[0] : undefined), - }); - const resolveStringArgDeviceCall = (args: unknown[]) => { - if ( - args.length >= 2 && - typeof args[0] === "string" && - typeof args[1] === "string" - ) { - return { udid: args[0], value: args[1] as string, rest: args.slice(2) }; - } - return { - udid: requireUdid(), - value: args[0] as string, - rest: args.slice(1), + const createSession = (defaultUdid?: string): SimDeckSession => { + const simulatorPath = (udid: string, suffix: string) => + `/api/simulators/${encodeURIComponent(udid)}${suffix}`; + const requireUdid = (udid?: string) => { + const resolved = udid ?? defaultUdid; + if (!resolved) { + throw new Error( + "This SimDeck session method requires a UDID. Pass one as the first argument or call connect({ udid }).", + ); + } + return resolved; }; - }; - const resolveObjectArgDeviceCall = (args: unknown[]) => { - if (typeof args[0] === "string") { - return { udid: args[0], value: args[1] as T, rest: args.slice(2) }; - } - return { udid: requireUdid(), value: args[0] as T, rest: args.slice(1) }; - }; - const resolveOptionalObjectDeviceCall = (args: unknown[]) => { - if (typeof args[0] === "string") { - return { udid: args[0], options: args[1] as T | undefined }; - } - return { udid: requireUdid(), options: args[0] as T | undefined }; - }; - const session: SimDeckSession = { - endpoint, - pid: result.pid, - projectRoot: result.projectRoot, - list: () => requestJson(endpoint, "GET", "/api/simulators"), - boot: (...args: [] | [string]) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestJson(endpoint, "POST", simulatorPath(udid, "/boot"), null); - }, - shutdown: (...args: [] | [string]) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestJson( - endpoint, - "POST", - simulatorPath(udid, "/shutdown"), - null, - ); - }, - erase: (...args: [] | [string]) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestJson(endpoint, "POST", simulatorPath(udid, "/erase"), null); - }, - install: (...args) => { - const { udid, value: appPath } = resolveStringArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/install"), { - appPath, - }); - }, - uninstall: (...args) => { - const { udid, value: bundleId } = resolveStringArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/uninstall"), { - bundleId, - }); - }, - launch: (...args) => { - const { udid, value: bundleId } = resolveStringArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/launch"), { - bundleId, - }); - }, - openUrl: (...args) => { - const { udid, value: url } = resolveStringArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/open-url"), { - url, - }); - }, - tap: (...args) => { - const [udid, x, y] = - typeof args[0] === "string" - ? [args[0], args[1] as number, args[2] as number] - : [requireUdid(), args[0] as number, args[1] as number]; - return requestOk(endpoint, simulatorPath(udid, "/tap"), { - x, - y, - normalized: true, - }); - }, - tapElement: (...args) => { - const { - udid, - value: selector, - rest, - } = resolveObjectArgDeviceCall(args); - const [tapOptions] = rest as [TapOptions?]; - return requestOk(endpoint, simulatorPath(udid, "/tap"), { - selector: selectorPayload(selector), - ...tapOptions, - }); - }, - touch: (...args) => { - const [udid, x, y, phase] = - typeof args[0] === "string" - ? [args[0], args[1] as number, args[2] as number, args[3] as string] - : [ - requireUdid(), - args[0] as number, - args[1] as number, - args[2] as string, - ]; - return requestOk(endpoint, simulatorPath(udid, "/touch"), { - x, - y, - phase, - }); - }, - swipe: (...args) => { - const [udid, startX, startY, endX, endY, swipeOptions = {}] = - typeof args[0] === "string" - ? [ - args[0], - args[1] as number, - args[2] as number, - args[3] as number, - args[4] as number, - args[5] as SwipeOptions | undefined, - ] - : [ - requireUdid(), - args[0] as number, - args[1] as number, - args[2] as number, - args[3] as number, - args[4] as SwipeOptions | undefined, - ]; - return requestJson(endpoint, "POST", simulatorPath(udid, "/batch"), { - steps: [ + const resolveNoArgDeviceCall = (args: unknown[]) => ({ + udid: requireUdid(typeof args[0] === "string" ? args[0] : undefined), + }); + const resolveStringArgDeviceCall = (args: unknown[]) => { + if ( + args.length >= 2 && + typeof args[0] === "string" && + typeof args[1] === "string" + ) { + return { udid: args[0], value: args[1] as string, rest: args.slice(2) }; + } + return { + udid: requireUdid(), + value: args[0] as string, + rest: args.slice(1), + }; + }; + const resolveObjectArgDeviceCall = (args: unknown[]) => { + if (typeof args[0] === "string") { + return { udid: args[0], value: args[1] as T, rest: args.slice(2) }; + } + return { udid: requireUdid(), value: args[0] as T, rest: args.slice(1) }; + }; + const resolveOptionalObjectDeviceCall = (args: unknown[]) => { + if (typeof args[0] === "string") { + return { udid: args[0], options: args[1] as T | undefined }; + } + return { udid: requireUdid(), options: args[0] as T | undefined }; + }; + const session: SimDeckSession = { + endpoint, + pid: result.pid, + projectRoot: result.projectRoot, + udid: defaultUdid, + device: (udid: string) => createSession(udid), + list: () => requestJson(endpoint, "GET", "/api/simulators"), + boot: (...args: [] | [string]) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestJson( + endpoint, + "POST", + simulatorPath(udid, "/boot"), + null, + ); + }, + shutdown: (...args: [] | [string]) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestJson( + endpoint, + "POST", + simulatorPath(udid, "/shutdown"), + null, + ); + }, + erase: (...args: [] | [string]) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestJson( + endpoint, + "POST", + simulatorPath(udid, "/erase"), + null, + ); + }, + install: (...args) => { + const { udid, value: appPath } = resolveStringArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/install"), { + appPath, + }); + }, + uninstall: (...args) => { + const { udid, value: bundleId } = resolveStringArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/uninstall"), { + bundleId, + }); + }, + launch: (...args) => { + const { udid, value: bundleId } = resolveStringArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/launch"), { + bundleId, + }); + }, + openUrl: (...args) => { + const { udid, value: url } = resolveStringArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/open-url"), { + url, + }); + }, + tap: (...args) => { + const [udid, x, y] = + typeof args[0] === "string" + ? [args[0], args[1] as number, args[2] as number] + : [requireUdid(), args[0] as number, args[1] as number]; + return requestOk(endpoint, simulatorPath(udid, "/tap"), { + x, + y, + normalized: true, + }); + }, + tapElement: (...args) => { + const { + udid, + value: selector, + rest, + } = resolveObjectArgDeviceCall(args); + const [tapOptions] = rest as [TapOptions?]; + return requestOk(endpoint, simulatorPath(udid, "/tap"), { + selector: selectorPayload(selector), + ...tapOptions, + }); + }, + touch: (...args) => { + const [udid, x, y, phase] = + typeof args[0] === "string" + ? [args[0], args[1] as number, args[2] as number, args[3] as string] + : [ + requireUdid(), + args[0] as number, + args[1] as number, + args[2] as string, + ]; + return requestOk(endpoint, simulatorPath(udid, "/touch"), { + x, + y, + phase, + }); + }, + swipe: (...args) => { + const [udid, startX, startY, endX, endY, swipeOptions = {}] = + typeof args[0] === "string" + ? [ + args[0], + args[1] as number, + args[2] as number, + args[3] as number, + args[4] as number, + args[5] as SwipeOptions | undefined, + ] + : [ + requireUdid(), + args[0] as number, + args[1] as number, + args[2] as number, + args[3] as number, + args[4] as SwipeOptions | undefined, + ]; + return requestJson(endpoint, "POST", simulatorPath(udid, "/batch"), { + steps: [ + { + action: "swipe", + startX, + startY, + endX, + endY, + ...swipeOptions, + }, + ], + }); + }, + gesture: (...args) => { + const { udid, value: preset, rest } = resolveStringArgDeviceCall(args); + const [gestureOptions = {}] = rest as [GestureOptions?]; + return requestJson(endpoint, "POST", simulatorPath(udid, "/batch"), { + steps: [ + { + action: "gesture", + preset, + ...gestureOptions, + }, + ], + }); + }, + typeText: (...args) => { + const { udid, value: text, rest } = resolveStringArgDeviceCall(args); + const [typeOptions = {}] = rest as [TypeTextOptions?]; + return requestJson(endpoint, "POST", simulatorPath(udid, "/batch"), { + steps: [ + { + action: "type", + text, + ...typeOptions, + }, + ], + }); + }, + key: (...args) => { + const [udid, keyCode, modifiers = 0] = + typeof args[0] === "string" + ? [args[0], args[1] as number, args[2] as number | undefined] + : [requireUdid(), args[0] as number, args[1] as number | undefined]; + return requestOk(endpoint, simulatorPath(udid, "/key"), { + keyCode, + modifiers, + }); + }, + keySequence: (...args) => { + const { + udid, + value: keyCodes, + rest, + } = resolveObjectArgDeviceCall(args); + const [keySequenceOptions = {}] = rest as [KeySequenceOptions?]; + return requestOk(endpoint, simulatorPath(udid, "/key-sequence"), { + keyCodes, + ...keySequenceOptions, + }); + }, + button: (...args) => { + const { udid, value: button, rest } = resolveStringArgDeviceCall(args); + const [durationMs = 0] = rest as [number?]; + return requestOk(endpoint, simulatorPath(udid, "/button"), { + button, + durationMs, + }); + }, + home: (...args: [] | [string]) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/home"), null); + }, + dismissKeyboard: (...args: [] | [string]) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestOk( + endpoint, + simulatorPath(udid, "/dismiss-keyboard"), + null, + ); + }, + appSwitcher: (...args: [] | [string]) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/app-switcher"), null); + }, + rotateLeft: (...args: [] | [string]) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/rotate-left"), null); + }, + rotateRight: (...args: [] | [string]) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/rotate-right"), null); + }, + toggleAppearance: (...args: [] | [string]) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestOk( + endpoint, + simulatorPath(udid, "/toggle-appearance"), + null, + ); + }, + pasteboardSet: (...args) => { + const { udid, value: text } = resolveStringArgDeviceCall(args); + return requestOk(endpoint, simulatorPath(udid, "/pasteboard"), { + text, + }); + }, + pasteboardGet: async (...args: [] | [string]) => { + const { udid } = resolveNoArgDeviceCall(args); + const result = await requestJson<{ text?: string }>( + endpoint, + "GET", + simulatorPath(udid, "/pasteboard"), + ); + return result.text ?? ""; + }, + chromeProfile: (...args: [] | [string]) => { + const { udid } = resolveNoArgDeviceCall(args); + return requestJson( + endpoint, + "GET", + simulatorPath(udid, "/chrome-profile"), + ); + }, + logs: async (...args) => { + const { udid, value: logsOptions } = + typeof args[0] === "string" + ? { udid: args[0], value: args[1] as LogsOptions | undefined } + : { + udid: requireUdid(), + value: args[0] as LogsOptions | undefined, + }; + const result = await requestJson<{ entries?: unknown[] }>( + endpoint, + "GET", + simulatorPath(udid, `/logs?${logsQuery(logsOptions)}`), + ); + return result.entries ?? []; + }, + tree: (...args) => { + const { udid, value: treeOptions } = + typeof args[0] === "string" + ? { udid: args[0], value: args[1] as QueryOptions | undefined } + : { + udid: requireUdid(), + value: args[0] as QueryOptions | undefined, + }; + return requestJson( + endpoint, + "GET", + simulatorPath(udid, `/accessibility-tree?${treeQuery(treeOptions)}`), + ); + }, + query: async (...args) => { + const { + udid, + value: selector, + rest, + } = resolveObjectArgDeviceCall(args); + const [treeOptions] = rest as [QueryOptions?]; + const result = await requestJson<{ matches: unknown[] }>( + endpoint, + "POST", + simulatorPath(udid, "/query"), { - action: "swipe", - startX, - startY, - endX, - endY, - ...swipeOptions, + selector: selectorPayload(selector), + ...treeOptions, }, - ], - }); - }, - gesture: (...args) => { - const { udid, value: preset, rest } = resolveStringArgDeviceCall(args); - const [gestureOptions = {}] = rest as [GestureOptions?]; - return requestJson(endpoint, "POST", simulatorPath(udid, "/batch"), { - steps: [ + ); + return result.matches; + }, + assert: (...args) => { + const { + udid, + value: selector, + rest, + } = resolveObjectArgDeviceCall(args); + const [assertOptions] = rest as [QueryOptions?]; + return requestJson(endpoint, "POST", simulatorPath(udid, "/assert"), { + selector: selectorPayload(selector), + ...assertOptions, + }); + }, + assertNot: (...args) => { + const { + udid, + value: selector, + rest, + } = resolveObjectArgDeviceCall(args); + const [assertOptions] = rest as [QueryOptions?]; + return requestJson( + endpoint, + "POST", + simulatorPath(udid, "/assert-not"), { - action: "gesture", - preset, - ...gestureOptions, + selector: selectorPayload(selector), + ...assertOptions, }, - ], - }); - }, - typeText: (...args) => { - const { udid, value: text, rest } = resolveStringArgDeviceCall(args); - const [typeOptions = {}] = rest as [TypeTextOptions?]; - return requestJson(endpoint, "POST", simulatorPath(udid, "/batch"), { - steps: [ + ); + }, + waitFor: (...args) => { + const { + udid, + value: selector, + rest, + } = resolveObjectArgDeviceCall(args); + const [waitOptions] = rest as [ + (QueryOptions & { timeoutMs?: number; pollMs?: number })?, + ]; + return requestJson(endpoint, "POST", simulatorPath(udid, "/wait-for"), { + selector: selectorPayload(selector), + ...waitOptions, + }); + }, + waitForNot: (...args) => { + const { + udid, + value: selector, + rest, + } = resolveObjectArgDeviceCall(args); + const [waitOptions] = rest as [ + (QueryOptions & { timeoutMs?: number; pollMs?: number })?, + ]; + return requestJson( + endpoint, + "POST", + simulatorPath(udid, "/wait-for-not"), { - action: "type", - text, - ...typeOptions, + selector: selectorPayload(selector), + ...waitOptions, }, - ], - }); - }, - key: (...args) => { - const [udid, keyCode, modifiers = 0] = - typeof args[0] === "string" - ? [args[0], args[1] as number, args[2] as number | undefined] - : [requireUdid(), args[0] as number, args[1] as number | undefined]; - return requestOk(endpoint, simulatorPath(udid, "/key"), { - keyCode, - modifiers, - }); - }, - keySequence: (...args) => { - const { - udid, - value: keyCodes, - rest, - } = resolveObjectArgDeviceCall(args); - const [keySequenceOptions = {}] = rest as [KeySequenceOptions?]; - return requestOk(endpoint, simulatorPath(udid, "/key-sequence"), { - keyCodes, - ...keySequenceOptions, - }); - }, - button: (...args) => { - const { udid, value: button, rest } = resolveStringArgDeviceCall(args); - const [durationMs = 0] = rest as [number?]; - return requestOk(endpoint, simulatorPath(udid, "/button"), { - button, - durationMs, - }); - }, - home: (...args: [] | [string]) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/home"), null); - }, - dismissKeyboard: (...args: [] | [string]) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestOk( - endpoint, - simulatorPath(udid, "/dismiss-keyboard"), - null, - ); - }, - appSwitcher: (...args: [] | [string]) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/app-switcher"), null); - }, - rotateLeft: (...args: [] | [string]) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/rotate-left"), null); - }, - rotateRight: (...args: [] | [string]) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/rotate-right"), null); - }, - toggleAppearance: (...args: [] | [string]) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestOk( - endpoint, - simulatorPath(udid, "/toggle-appearance"), - null, - ); - }, - pasteboardSet: (...args) => { - const { udid, value: text } = resolveStringArgDeviceCall(args); - return requestOk(endpoint, simulatorPath(udid, "/pasteboard"), { - text, - }); - }, - pasteboardGet: async (...args: [] | [string]) => { - const { udid } = resolveNoArgDeviceCall(args); - const result = await requestJson<{ text?: string }>( - endpoint, - "GET", - simulatorPath(udid, "/pasteboard"), - ); - return result.text ?? ""; - }, - chromeProfile: (...args: [] | [string]) => { - const { udid } = resolveNoArgDeviceCall(args); - return requestJson( - endpoint, - "GET", - simulatorPath(udid, "/chrome-profile"), - ); - }, - logs: async (...args) => { - const { udid, value: logsOptions } = - typeof args[0] === "string" - ? { udid: args[0], value: args[1] as LogsOptions | undefined } - : { udid: requireUdid(), value: args[0] as LogsOptions | undefined }; - const result = await requestJson<{ entries?: unknown[] }>( - endpoint, - "GET", - simulatorPath(udid, `/logs?${logsQuery(logsOptions)}`), - ); - return result.entries ?? []; - }, - tree: (...args) => { - const { udid, value: treeOptions } = - typeof args[0] === "string" - ? { udid: args[0], value: args[1] as QueryOptions | undefined } - : { udid: requireUdid(), value: args[0] as QueryOptions | undefined }; - return requestJson( - endpoint, - "GET", - simulatorPath(udid, `/accessibility-tree?${treeQuery(treeOptions)}`), - ); - }, - query: async (...args) => { - const { - udid, - value: selector, - rest, - } = resolveObjectArgDeviceCall(args); - const [treeOptions] = rest as [QueryOptions?]; - const result = await requestJson<{ matches: unknown[] }>( - endpoint, - "POST", - simulatorPath(udid, "/query"), - { - selector: selectorPayload(selector), - ...treeOptions, - }, - ); - return result.matches; - }, - assert: (...args) => { - const { - udid, - value: selector, - rest, - } = resolveObjectArgDeviceCall(args); - const [assertOptions] = rest as [QueryOptions?]; - return requestJson(endpoint, "POST", simulatorPath(udid, "/assert"), { - selector: selectorPayload(selector), - ...assertOptions, - }); - }, - waitFor: (...args) => { - const { - udid, - value: selector, - rest, - } = resolveObjectArgDeviceCall(args); - const [waitOptions] = rest as [ - (QueryOptions & { timeoutMs?: number; pollMs?: number })?, - ]; - return requestJson(endpoint, "POST", simulatorPath(udid, "/wait-for"), { - selector: selectorPayload(selector), - ...waitOptions, - }); - }, - batch: (...args) => { - const { - udid, - value: steps, - rest, - } = resolveObjectArgDeviceCall(args); - const [continueOnError = false] = rest as [boolean?]; - return requestJson(endpoint, "POST", simulatorPath(udid, "/batch"), { - steps, - continueOnError, - }); - }, - screenshot: ( - ...args: [string, ScreenshotOptions?] | [ScreenshotOptions?] - ) => { - const { udid, options } = - resolveOptionalObjectDeviceCall(args); - const params = new URLSearchParams(); - if (options?.withBezel ?? options?.bezel) { - params.set("bezel", "true"); - } - const query = params.toString(); - return requestBuffer( - endpoint, - simulatorPath(udid, `/screenshot.png${query ? `?${query}` : ""}`), - ); - }, - record: ( - ...args: [string, ScreenRecordingOptions?] | [ScreenRecordingOptions?] - ) => { - const { udid, options } = - resolveOptionalObjectDeviceCall(args); - return requestBuffer( - endpoint, - simulatorPath(udid, "/screen-recording"), - "POST", - { - seconds: options?.seconds ?? 5, - }, - ); - }, - close: () => { - if (options.keepDaemon) { - return; - } - if (result.child) { - result.child.kill(); - if (result.isolatedRoot) { - fs.rmSync(result.isolatedRoot, { recursive: true, force: true }); + ); + }, + scrollUntilVisible: (...args) => { + const { + udid, + value: selector, + rest, + } = resolveObjectArgDeviceCall(args); + const [scrollOptions] = rest as [ + | (QueryOptions & { + timeoutMs?: number; + pollMs?: number; + direction?: "up" | "down" | "left" | "right"; + durationMs?: number; + steps?: number; + }) + | undefined, + ]; + return requestJson( + endpoint, + "POST", + simulatorPath(udid, "/scroll-until-visible"), + { + selector: selectorPayload(selector), + ...scrollOptions, + }, + ); + }, + batch: (...args) => { + const { + udid, + value: steps, + rest, + } = resolveObjectArgDeviceCall(args); + const [continueOnError = false] = rest as [boolean?]; + return requestJson(endpoint, "POST", simulatorPath(udid, "/batch"), { + steps, + continueOnError, + }); + }, + screenshot: ( + ...args: [string, ScreenshotOptions?] | [ScreenshotOptions?] + ) => { + const { udid, options } = + resolveOptionalObjectDeviceCall(args); + const params = new URLSearchParams(); + if (options?.withBezel ?? options?.bezel) { + params.set("bezel", "true"); } - return; - } - if (result.started) { - spawnSync(cliPath, ["daemon", "stop"], { cwd: options.projectRoot }); - } - }, + const query = params.toString(); + return requestBuffer( + endpoint, + simulatorPath(udid, `/screenshot.png${query ? `?${query}` : ""}`), + ); + }, + record: ( + ...args: [string, ScreenRecordingOptions?] | [ScreenRecordingOptions?] + ) => { + const { udid, options } = + resolveOptionalObjectDeviceCall(args); + return requestBuffer( + endpoint, + simulatorPath(udid, "/screen-recording"), + "POST", + { + seconds: options?.seconds ?? 5, + }, + ); + }, + close: () => { + if (options.keepDaemon) { + return; + } + if (result.child) { + result.child.kill(); + if (result.isolatedRoot) { + fs.rmSync(result.isolatedRoot, { recursive: true, force: true }); + } + return; + } + if (result.started) { + spawnSync(cliPath, ["daemon", "stop"], { cwd: options.projectRoot }); + } + }, + }; + return session; }; - return session; + return createSession(options.udid); } async function startIsolatedDaemon( @@ -807,6 +924,7 @@ function treeQuery(options: QueryOptions = {}): string { if (options.maxDepth !== undefined) params.set("maxDepth", String(options.maxDepth)); if (options.includeHidden) params.set("includeHidden", "true"); + if (options.interactiveOnly) params.set("interactiveOnly", "true"); return params.toString(); } @@ -826,11 +944,18 @@ function logsQuery(options: LogsOptions = {}): string { function selectorPayload( selector: ElementSelector, -): Record { +): Record { return { + text: selector.text, id: selector.id, label: selector.label, value: selector.value, elementType: selector.type, + index: selector.index, + enabled: selector.enabled, + checked: selector.checked, + focused: selector.focused, + selected: selector.selected, + regex: selector.regex, }; } diff --git a/scripts/integration/cli.mjs b/scripts/integration/cli.mjs index bb100a27..1f888fb5 100644 --- a/scripts/integration/cli.mjs +++ b/scripts/integration/cli.mjs @@ -137,20 +137,26 @@ async function main() { await measuredStep("CLI list", () => assertSimulatorListed(simulatorUDID), { phase: phaseSetup, }); + await measuredStep( + "CLI use default simulator", + () => { + const selection = simdeckJson(["use", simulatorUDID]); + if (selection.udid !== simulatorUDID) { + throw new Error(`simdeck use selected ${JSON.stringify(selection)}`); + } + }, + { phase: phaseSetup }, + ); await measuredStep( "CLI chrome-profile", - () => - assertJson( - simdeckJson(["chrome-profile", simulatorUDID]), - "chrome-profile", - ), + () => assertJson(simdeckJson(["chrome-profile"]), "chrome-profile"), { phase: phaseSetup }, ); await measuredStep( "CLI logs", () => assertJson( - simdeckJson(["logs", simulatorUDID, "--seconds", "1", "--limit", "1"]), + simdeckJson(["logs", "--seconds", "1", "--limit", "1"]), "logs", ), { phase: phaseSetup }, @@ -159,7 +165,7 @@ async function main() { await measuredStep( "CLI install fixture", async () => { - simdeckJson(["install", simulatorUDID, fixture.appPath]); + simdeckJson(["install", fixture.appPath]); preapproveFixtureUrlScheme(); }, { phase: phaseSetup }, @@ -177,7 +183,6 @@ async function main() { const agentTree = await measuredStep("server describe agent", () => simdeckText([ "describe", - simulatorUDID, "--source", "native-ax", "--format", @@ -189,6 +194,28 @@ async function main() { if (!agentTree.includes("source:") || !agentTree.includes("- ")) { throw new Error("agent describe output did not look like a hierarchy"); } + const interactiveTree = await measuredStep( + "server describe agent interactive", + () => + simdeckText([ + "describe", + "--source", + "native-ax", + "--format", + "agent", + "--max-depth", + "8", + "--interactive", + ]), + ); + if ( + !interactiveTree.includes("source:") || + !interactiveTree.includes("Continue") + ) { + throw new Error( + "interactive agent describe did not include fixture controls", + ); + } await runRestControls(); await runCliControls(); @@ -196,7 +223,7 @@ async function main() { await measuredStep( "CLI screenshot file", async () => { - simdeckJson(["screenshot", simulatorUDID, "--output", screenshotPath]); + simdeckJson(["screenshot", "--output", screenshotPath]); assertPng(screenshotPath); }, { phase: phaseCommandSmoke }, @@ -209,10 +236,11 @@ async function main() { stdoutPng, runBuffer( simdeck, - ["--server-url", serverUrl, "screenshot", simulatorUDID, "--stdout"], + ["--server-url", serverUrl, "screenshot", "--stdout"], { timeoutMs: 300_000, maxBuffer: 64 * 1024 * 1024, + env: { HOME: tempRoot }, }, ), ); @@ -226,7 +254,6 @@ async function main() { async () => { simdeckJson([ "screenshot", - simulatorUDID, "--with-bezel", "--output", bezeledScreenshotPath, @@ -239,14 +266,7 @@ async function main() { await measuredStep( "CLI screen recording", async () => { - simdeckJson([ - "record", - simulatorUDID, - "--seconds", - "1", - "--output", - recordingPath, - ]); + simdeckJson(["record", "--seconds", "1", "--output", recordingPath]); assertMp4(recordingPath); }, { phase: phaseCommandSmoke }, @@ -255,14 +275,14 @@ async function main() { await measuredStep( "CLI pasteboard set", async () => { - simdeckJson(["pasteboard", "set", simulatorUDID, "simdeck integration"]); + simdeckJson(["pasteboard", "set", "simdeck integration"]); }, { phase: phaseCommandSmoke }, ); await measuredStep( "CLI pasteboard get", async () => { - const pasteboard = simdeckJson(["pasteboard", "get", simulatorUDID]); + const pasteboard = simdeckJson(["pasteboard", "get"]); if (pasteboard.text !== "simdeck integration") { throw new Error( `pasteboard round-trip failed: ${JSON.stringify(pasteboard)}`, @@ -277,14 +297,14 @@ async function main() { await measuredStep( "CLI type file", async () => { - simdeckJson(["type", simulatorUDID, "--file", fileInput]); + simdeckJson(["type", "--file", fileInput]); }, { phase: phaseCommandSmoke }, ); await measuredStep( "CLI type stdin", async () => { - simdeckJson(["type", simulatorUDID, "--stdin"], { + simdeckJson(["type", "--stdin"], { input: "stdin input", }); }, @@ -296,7 +316,6 @@ async function main() { async () => { const batch = simdeckJson([ "batch", - simulatorUDID, "--step", "button home", "--step", @@ -316,7 +335,7 @@ async function main() { await measuredStep( "CLI uninstall fixture", - () => simdeckJson(["uninstall", simulatorUDID, fixtureBundleId]), + () => simdeckJson(["uninstall", fixtureBundleId]), { phase: phaseSimulatorLifecycle }, ); await measuredStep( @@ -482,6 +501,15 @@ async function runCliControls() { }, { expectFixture: true, expectText: "URL Opened" }, ); + await cliStep( + "CLI tap label shorthand", + ["tap", "Continue", "--wait-timeout-ms", "15000", "--duration-ms", "30"], + { + timeoutMs: 180_000, + maxElapsedMs: 60_000, + }, + { expectFixture: true, expectText: "Continue Tapped", attempts: 8 }, + ); await cliStep( "CLI tap fixture text field", [ @@ -497,7 +525,7 @@ async function runCliControls() { { timeoutMs: 180_000, maxElapsedMs: 60_000 }, { expectFixture: true, - expectText: "URL Opened", + expectText: "Continue Tapped", attempts: 6, delayMs: 1_500, }, @@ -623,6 +651,20 @@ async function runRestControls() { }, { phase: phaseSetup }, ); + await measuredStep( + "REST accessibility-tree interactive", + async () => { + const tree = await httpJson( + "GET", + `/api/simulators/${simulatorUDID}/accessibility-tree?source=native-ax&maxDepth=8&interactiveOnly=true`, + ); + assertRoots(tree, "REST accessibility-tree interactive"); + if (tree.interactiveOnly !== true) { + throw new Error("interactive tree did not report interactiveOnly=true"); + } + }, + { phase: phaseSetup }, + ); await measuredStep( "REST chrome-profile", async () => { @@ -1640,6 +1682,7 @@ function simdeckText(args, options = {}) { timeoutMs: options.timeoutMs ?? 120_000, maxElapsedMs: options.maxElapsedMs, input: options.input, + env: { HOME: tempRoot, ...(options.env ?? {}) }, }); } @@ -1654,6 +1697,7 @@ function runText(command, args, options = {}) { cwd: root, encoding: "utf8", input: options.input, + env: options.env ? { ...process.env, ...options.env } : process.env, timeout: options.timeoutMs ?? 120_000, }); if (result.status !== 0) { @@ -1679,6 +1723,7 @@ function runBuffer(command, args, options = {}) { cwd: root, encoding: "buffer", maxBuffer: options.maxBuffer ?? 16 * 1024 * 1024, + env: options.env ? { ...process.env, ...options.env } : process.env, timeout: options.timeoutMs ?? 120_000, }); if (result.status !== 0) { diff --git a/server/Cargo.lock b/server/Cargo.lock index 341ce51a..706464cc 100644 --- a/server/Cargo.lock +++ b/server/Cargo.lock @@ -2093,6 +2093,19 @@ dependencies = [ "serde", ] +[[package]] +name = "serde_yaml" +version = "0.9.34+deprecated" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "6a8b1a1a2ebf674015cc02edccce75287f1a0130d394307b36743c2f5d504b47" +dependencies = [ + "indexmap 2.14.0", + "itoa", + "ryu", + "serde", + "unsafe-libyaml", +] + [[package]] name = "sha1" version = "0.10.6" @@ -2167,9 +2180,11 @@ dependencies = [ "plist", "prost", "qrcode", + "regex", "roxmltree", "serde", "serde_json", + "serde_yaml", "sha2", "thiserror 2.0.18", "tokio", @@ -2704,6 +2719,12 @@ dependencies = [ "subtle", ] +[[package]] +name = "unsafe-libyaml" +version = "0.2.11" +source = "registry+https://github.com/rust-lang/crates.io-index" +checksum = "673aac59facbab8a9007c7f6108d11f63b603f7cabff99fabf650fea5c32b861" + [[package]] name = "untrusted" version = "0.9.0" diff --git a/server/Cargo.toml b/server/Cargo.toml index 26701016..31099f82 100644 --- a/server/Cargo.toml +++ b/server/Cargo.toml @@ -18,8 +18,10 @@ plist = "1.7" prost = "0.13" qrcode = "0.14" roxmltree = "0.20" +regex = "1.11" serde = { version = "1.0", features = ["derive"] } serde_json = "1.0" +serde_yaml = "0.9" sha2 = "0.10" thiserror = "2.0" tokio = { version = "1.42", features = ["fs", "io-util", "macros", "process", "rt-multi-thread", "signal", "sync", "time"] } diff --git a/server/src/accessibility.rs b/server/src/accessibility.rs new file mode 100644 index 00000000..1615a0fe --- /dev/null +++ b/server/src/accessibility.rs @@ -0,0 +1,212 @@ +use serde_json::{Map, Value}; + +pub fn interactive_accessibility_snapshot(snapshot: &Value) -> Value { + let mut output = snapshot.as_object().cloned().unwrap_or_default(); + let roots = snapshot + .get("roots") + .and_then(Value::as_array) + .map(|roots| { + roots + .iter() + .filter_map(interactive_accessibility_node) + .collect::>() + }) + .unwrap_or_default(); + + output.insert("roots".to_owned(), Value::Array(roots)); + output.insert("interactiveOnly".to_owned(), Value::Bool(true)); + Value::Object(output) +} + +fn interactive_accessibility_node(node: &Value) -> Option { + let object = node.as_object()?; + let children = node + .get("children") + .and_then(Value::as_array) + .map(|children| { + children + .iter() + .filter_map(interactive_accessibility_node) + .collect::>() + }) + .unwrap_or_default(); + + if !is_interactive_accessibility_node(node) && children.is_empty() { + return None; + } + + let mut output = object.clone(); + if children.is_empty() { + output.remove("children"); + } else { + output.insert("children".to_owned(), Value::Array(children)); + } + Some(Value::Object(output)) +} + +fn is_interactive_accessibility_node(node: &Value) -> bool { + if bool_field(node, &["hidden", "isHidden"]).unwrap_or(false) { + return false; + } + if numeric_field(node, &["alpha"]).is_some_and(|alpha| alpha <= 0.01) { + return false; + } + + if has_actionable_action(node) { + return true; + } + if bool_field( + node, + &[ + "clickable", + "focusable", + "isUserInteractionEnabled", + "scrollable", + "checked", + "selected", + ], + ) + .unwrap_or(false) + { + return true; + } + + string_field( + node, + &[ + "type", + "role", + "className", + "elementType", + "displayName", + "widgetType", + ], + ) + .is_some_and(|role| role_looks_interactive(&role)) +} + +fn has_actionable_action(node: &Value) -> bool { + for actions in [ + node.get("actions"), + node.get("custom_actions"), + node.get("control") + .and_then(|control| control.get("actions")), + ] + .into_iter() + .flatten() + { + if actions + .as_array() + .into_iter() + .flatten() + .filter_map(Value::as_str) + .any(action_looks_interactive) + { + return true; + } + } + false +} + +fn action_looks_interactive(action: &str) -> bool { + let action = action.trim().to_ascii_lowercase(); + !action.is_empty() + && !matches!( + action.as_str(), + "describe" | "getproperties" | "get_properties" | "highlight" + ) +} + +fn role_looks_interactive(role: &str) -> bool { + let role = role.to_ascii_lowercase(); + [ + "button", + "cell", + "checkbox", + "collection", + "combobox", + "control", + "edittext", + "link", + "menu", + "picker", + "radio", + "scroll", + "search", + "segmented", + "select", + "slider", + "stepper", + "switch", + "tab", + "table", + "textfield", + "text field", + "textinput", + "text input", + "toggle", + "webview", + ] + .iter() + .any(|needle| role.contains(needle)) +} + +fn bool_field(node: &Value, fields: &[&str]) -> Option { + fields.iter().find_map(|field| nested_bool(node, field)) +} + +fn numeric_field(node: &Value, fields: &[&str]) -> Option { + fields.iter().find_map(|field| nested_number(node, field)) +} + +fn string_field(node: &Value, fields: &[&str]) -> Option { + fields.iter().find_map(|field| nested_string(node, field)) +} + +fn nested_bool(node: &Value, field: &str) -> Option { + node.get(field) + .and_then(Value::as_bool) + .or_else(|| { + nested_object(node, "accessibility").and_then(|object| bool_from_map(object, field)) + }) + .or_else(|| nested_object(node, "control").and_then(|object| bool_from_map(object, field))) +} + +fn nested_number(node: &Value, field: &str) -> Option { + node.get(field) + .and_then(Value::as_f64) + .or_else(|| { + nested_object(node, "accessibility").and_then(|object| number_from_map(object, field)) + }) + .or_else(|| { + nested_object(node, "control").and_then(|object| number_from_map(object, field)) + }) +} + +fn nested_string(node: &Value, field: &str) -> Option { + node.get(field) + .and_then(Value::as_str) + .map(str::to_owned) + .or_else(|| { + nested_object(node, "accessibility").and_then(|object| string_from_map(object, field)) + }) + .or_else(|| { + nested_object(node, "control").and_then(|object| string_from_map(object, field)) + }) +} + +fn nested_object<'a>(node: &'a Value, field: &str) -> Option<&'a Map> { + node.get(field).and_then(Value::as_object) +} + +fn bool_from_map(object: &Map, field: &str) -> Option { + object.get(field).and_then(Value::as_bool) +} + +fn number_from_map(object: &Map, field: &str) -> Option { + object.get(field).and_then(Value::as_f64) +} + +fn string_from_map(object: &Map, field: &str) -> Option { + object.get(field).and_then(Value::as_str).map(str::to_owned) +} diff --git a/server/src/api/routes.rs b/server/src/api/routes.rs index c88e6cd3..64527451 100644 --- a/server/src/api/routes.rs +++ b/server/src/api/routes.rs @@ -1,3 +1,4 @@ +use crate::accessibility::interactive_accessibility_snapshot; use crate::android::{self, AndroidBridge, AndroidEmulatorSpec}; use crate::api::json::json; use crate::auth; @@ -30,6 +31,7 @@ use axum::routing::{get, post}; use axum::{Json, Router}; use bytes::{Bytes, BytesMut}; use futures::{SinkExt, StreamExt}; +use regex::Regex; use serde::Deserialize; use serde_json::Map; use serde_json::{json as json_value, Value}; @@ -575,11 +577,18 @@ struct ChromeButtonPngQuery { #[derive(Deserialize, Clone, Default)] #[serde(rename_all = "camelCase")] struct ElementSelectorPayload { + text: Option, id: Option, label: Option, value: Option, #[serde(alias = "type")] element_type: Option, + index: Option, + enabled: Option, + checked: Option, + focused: Option, + selected: Option, + regex: Option, } #[derive(Deserialize, Clone)] @@ -605,6 +614,21 @@ struct WaitForPayload { poll_ms: Option, } +#[derive(Deserialize, Clone)] +#[serde(rename_all = "camelCase")] +struct ScrollUntilVisiblePayload { + #[serde(default)] + selector: ElementSelectorPayload, + source: Option, + max_depth: Option, + include_hidden: Option, + timeout_ms: Option, + poll_ms: Option, + direction: Option, + duration_ms: Option, + steps: Option, +} + #[derive(Deserialize, Clone)] #[serde(rename_all = "camelCase")] struct TapElementPayload { @@ -642,6 +666,8 @@ enum BatchStep { Tap(TapElementPayload), WaitFor(WaitForPayload), Assert(WaitForPayload), + AssertNot(WaitForPayload), + ScrollUntilVisible(ScrollUntilVisiblePayload), Key { key_code: u16, modifiers: Option, @@ -719,6 +745,7 @@ struct AccessibilityTreeQuery { source: Option, max_depth: Option, include_hidden: Option, + interactive_only: Option, } #[derive(Deserialize)] @@ -863,6 +890,18 @@ pub fn router(state: AppState) -> Router { .route("/api/simulators/{udid}/query", post(accessibility_query)) .route("/api/simulators/{udid}/wait-for", post(wait_for_element)) .route("/api/simulators/{udid}/assert", post(assert_element)) + .route( + "/api/simulators/{udid}/wait-for-not", + post(wait_for_not_element), + ) + .route( + "/api/simulators/{udid}/assert-not", + post(assert_not_element), + ) + .route( + "/api/simulators/{udid}/scroll-until-visible", + post(scroll_until_visible), + ) .route("/api/simulators/{udid}/batch", post(run_batch)) .route("/api/simulators/{udid}/touch", post(send_touch)) .route("/api/simulators/{udid}/edge-touch", post(send_edge_touch)) @@ -2558,6 +2597,7 @@ async fn accessibility_query( payload.source.as_deref(), payload.max_depth, payload.include_hidden.unwrap_or(false), + false, ) .await?; let matches = query_compact_elements( @@ -2589,6 +2629,30 @@ async fn assert_element( wait_for_element_payload(state, udid, payload).await } +async fn wait_for_not_element( + State(state): State, + Path(udid): Path, + Json(payload): Json, +) -> Result, AppError> { + wait_for_absent_element_payload(state, udid, payload).await +} + +async fn assert_not_element( + State(state): State, + Path(udid): Path, + Json(payload): Json, +) -> Result, AppError> { + wait_for_absent_element_payload(state, udid, payload).await +} + +async fn scroll_until_visible( + State(state): State, + Path(udid): Path, + Json(payload): Json, +) -> Result, AppError> { + scroll_until_visible_payload(state, udid, payload).await +} + async fn run_batch( State(state): State, Path(udid): Path, @@ -4014,6 +4078,7 @@ async fn accessibility_tree( query.source.as_deref(), query.max_depth, query.include_hidden.unwrap_or(false), + query.interactive_only.unwrap_or(false), ) .await?, )) @@ -4025,6 +4090,7 @@ async fn accessibility_tree_value( source: Option<&str>, max_depth: Option, include_hidden: bool, + interactive_only: bool, ) -> Result { if android::is_android_id(&udid) { let requested_source = source @@ -4038,6 +4104,9 @@ async fn accessibility_tree_value( if let Some(source) = requested_source { tree["requestedSource"] = Value::String(source); } + if interactive_only { + tree = interactive_accessibility_snapshot(&tree); + } Ok(tree) }) .await; @@ -4052,11 +4121,16 @@ async fn accessibility_tree_value( match accessibility_snapshot(state.clone(), udid.clone(), None, max_depth).await { Ok(snapshot) => snapshot, Err(error) => { - return Ok(empty_accessibility_tree( + let snapshot = empty_accessibility_tree( SOURCE_NATIVE_AX, &available_sources, suppress_native_ax_translation_error(&error.to_string()), - )); + ); + return Ok(if interactive_only { + interactive_accessibility_snapshot(&snapshot) + } else { + snapshot + }); } }; merge_connected_sources_for_pid( @@ -4067,6 +4141,11 @@ async fn accessibility_tree_value( ) .await; let snapshot = attach_available_sources(native_snapshot, &available_sources); + let snapshot = if interactive_only { + interactive_accessibility_snapshot(&snapshot) + } else { + snapshot + }; return Ok(snapshot); } @@ -4087,6 +4166,7 @@ async fn accessibility_tree_value( hierarchy_source, max_depth, include_hidden, + interactive_only, ) .await { @@ -4115,11 +4195,13 @@ async fn accessibility_tree_value( } else { None }; - Ok(attach_tree_metadata( - snapshot, - &available_sources, - fallback_reason, - )) + let snapshot = + attach_tree_metadata(snapshot, &available_sources, fallback_reason); + Ok(if interactive_only { + interactive_accessibility_snapshot(&snapshot) + } else { + snapshot + }) } Err(_inspector_error) => { let mut available_sources = available_sources_with_native_ax(Some(&session)); @@ -4130,6 +4212,7 @@ async fn accessibility_tree_value( InAppHierarchySource::Automatic, Some(0), include_hidden, + false, ) .await { @@ -4139,15 +4222,29 @@ async fn accessibility_tree_value( } match accessibility_snapshot(state.clone(), udid.clone(), None, max_depth).await { - Ok(native_snapshot) => Ok(attach_available_sources( - trim_tree_depth(native_snapshot, max_depth), - &available_sources, - )), - Err(native_ax_error) => Ok(empty_accessibility_tree( - SOURCE_NATIVE_AX, - &available_sources, - suppress_native_ax_translation_error(&native_ax_error.to_string()), - )), + Ok(native_snapshot) => { + let snapshot = attach_available_sources( + trim_tree_depth(native_snapshot, max_depth), + &available_sources, + ); + Ok(if interactive_only { + interactive_accessibility_snapshot(&snapshot) + } else { + snapshot + }) + } + Err(native_ax_error) => { + let snapshot = empty_accessibility_tree( + SOURCE_NATIVE_AX, + &available_sources, + suppress_native_ax_translation_error(&native_ax_error.to_string()), + ); + Ok(if interactive_only { + interactive_accessibility_snapshot(&snapshot) + } else { + snapshot + }) + } } } } @@ -4155,15 +4252,29 @@ async fn accessibility_tree_value( Err(_inspector_error) => { let available_sources = available_sources_with_native_ax(None); match accessibility_snapshot(state.clone(), udid.clone(), None, max_depth).await { - Ok(native_snapshot) => Ok(attach_available_sources( - trim_tree_depth(native_snapshot, max_depth), - &available_sources, - )), - Err(native_ax_error) => Ok(empty_accessibility_tree( - SOURCE_NATIVE_AX, - &available_sources, - suppress_native_ax_translation_error(&native_ax_error.to_string()), - )), + Ok(native_snapshot) => { + let snapshot = attach_available_sources( + trim_tree_depth(native_snapshot, max_depth), + &available_sources, + ); + Ok(if interactive_only { + interactive_accessibility_snapshot(&snapshot) + } else { + snapshot + }) + } + Err(native_ax_error) => { + let snapshot = empty_accessibility_tree( + SOURCE_NATIVE_AX, + &available_sources, + suppress_native_ax_translation_error(&native_ax_error.to_string()), + ); + Ok(if interactive_only { + interactive_accessibility_snapshot(&snapshot) + } else { + snapshot + }) + } } } } @@ -4237,6 +4348,7 @@ async fn perform_tap_payload( payload.source.as_deref(), payload.max_depth, payload.include_hidden.unwrap_or(false), + false, ) .await?; normalize_screen_point_from_snapshot(&snapshot, x, y)? @@ -4295,6 +4407,40 @@ async fn wait_for_element_payload( }))) } +async fn wait_for_absent_element_payload( + state: AppState, + udid: String, + payload: WaitForPayload, +) -> Result, AppError> { + let started = Instant::now(); + let timeout_ms = payload.timeout_ms.unwrap_or(5_000); + let poll_ms = payload.poll_ms.unwrap_or(100).max(10); + let deadline = Instant::now() + Duration::from_millis(timeout_ms); + loop { + let snapshot = accessibility_tree_value( + state.clone(), + udid.clone(), + payload.source.as_deref(), + payload.max_depth, + payload.include_hidden.unwrap_or(false), + false, + ) + .await?; + if first_matching_element(&snapshot, &payload.selector).is_none() { + return Ok(json(json_value!({ + "ok": true, + "elapsedMs": started.elapsed().as_millis() as u64, + }))); + } + if timeout_ms == 0 || Instant::now() >= deadline { + return Err(AppError::bad_request( + "Accessibility element still matched the selector.", + )); + } + tokio::time::sleep(Duration::from_millis(poll_ms)).await; + } +} + async fn wait_for_snapshot_match( state: AppState, udid: String, @@ -4310,6 +4456,7 @@ async fn wait_for_snapshot_match( payload.source.as_deref(), payload.max_depth, payload.include_hidden.unwrap_or(false), + false, ) .await?; if first_matching_element(&snapshot, &payload.selector).is_some() { @@ -4322,6 +4469,153 @@ async fn wait_for_snapshot_match( } } +async fn scroll_until_visible_payload( + state: AppState, + udid: String, + payload: ScrollUntilVisiblePayload, +) -> Result, AppError> { + let started = Instant::now(); + let timeout_ms = payload.timeout_ms.unwrap_or(10_000); + let poll_ms = payload.poll_ms.unwrap_or(100).max(10); + let deadline = Instant::now() + Duration::from_millis(timeout_ms); + let mut scroll_count = 0usize; + loop { + let snapshot = accessibility_tree_value( + state.clone(), + udid.clone(), + payload.source.as_deref(), + payload.max_depth, + payload.include_hidden.unwrap_or(false), + false, + ) + .await?; + if let Some(found) = first_matching_element(&snapshot, &payload.selector) { + return Ok(json(json_value!({ + "ok": true, + "elapsedMs": started.elapsed().as_millis() as u64, + "scrollCount": scroll_count, + "match": compact_accessibility_node(&found), + }))); + } + if timeout_ms == 0 || Instant::now() >= deadline { + return Err(AppError::not_found("No accessibility element matched.")); + } + let scroll_plan = scroll_input_plan_for_udid(&udid, &payload)?; + perform_scroll_input(state.clone(), udid.clone(), scroll_plan).await?; + scroll_count += 1; + tokio::time::sleep(Duration::from_millis(poll_ms)).await; + } +} + +#[derive(Clone, Copy, Debug, PartialEq)] +struct NormalizedSwipe { + start_x: f64, + start_y: f64, + end_x: f64, + end_y: f64, + duration_ms: u64, + steps: u32, +} + +#[derive(Clone, Copy, Debug, PartialEq, Eq)] +enum ScrollInputBackend { + Android, + Native, +} + +#[derive(Clone, Copy, Debug, PartialEq)] +struct ScrollInputPlan { + backend: ScrollInputBackend, + swipe: NormalizedSwipe, +} + +fn scroll_input_plan_for_udid( + udid: &str, + payload: &ScrollUntilVisiblePayload, +) -> Result { + let (start_x, start_y, end_x, end_y) = + normalized_scroll_coordinates(payload.direction.as_deref())?; + Ok(ScrollInputPlan { + backend: if android::is_android_id(udid) { + ScrollInputBackend::Android + } else { + ScrollInputBackend::Native + }, + swipe: NormalizedSwipe { + start_x, + start_y, + end_x, + end_y, + duration_ms: payload.duration_ms.unwrap_or(350), + steps: payload.steps.unwrap_or(12).max(1), + }, + }) +} + +async fn perform_scroll_input( + state: AppState, + udid: String, + plan: ScrollInputPlan, +) -> Result<(), AppError> { + let swipe = plan.swipe; + match plan.backend { + ScrollInputBackend::Android => { + run_android_action(state, move |android| { + android.send_swipe( + &udid, + swipe.start_x, + swipe.start_y, + swipe.end_x, + swipe.end_y, + swipe.duration_ms, + ) + }) + .await + } + ScrollInputBackend::Native => { + run_bridge_action(state, move |bridge| { + if bridge_simulator_is_tvos(&bridge, &udid) { + let key_code = tvos_remote_key_for_touch_motion( + swipe.start_x, + swipe.start_y, + swipe.end_x, + swipe.end_y, + ); + return press_tvos_remote_key(&bridge, &udid, key_code); + } + let input = bridge.create_input_session(&udid)?; + let delay = Duration::from_millis(swipe.duration_ms / u64::from(swipe.steps)); + input.send_touch(swipe.start_x, swipe.start_y, "began")?; + for step in 1..swipe.steps { + let t = f64::from(step) / f64::from(swipe.steps); + input.send_touch( + swipe.start_x + (swipe.end_x - swipe.start_x) * t, + swipe.start_y + (swipe.end_y - swipe.start_y) * t, + "moved", + )?; + std::thread::sleep(delay); + } + input.send_touch(swipe.end_x, swipe.end_y, "ended") + }) + .await + } + } +} + +fn normalized_scroll_coordinates( + direction: Option<&str>, +) -> Result<(f64, f64, f64, f64), AppError> { + match direction.unwrap_or("down").to_ascii_lowercase().as_str() { + "down" => Ok((0.5, 0.78, 0.5, 0.22)), + "up" => Ok((0.5, 0.22, 0.5, 0.78)), + "left" => Ok((0.78, 0.5, 0.22, 0.5)), + "right" => Ok((0.22, 0.5, 0.78, 0.5)), + other => Err(AppError::bad_request(format!( + "Unsupported scroll direction `{other}`." + ))), + } +} + async fn run_batch_step(state: AppState, udid: String, step: BatchStep) -> Result { match step { BatchStep::Sleep { ms, seconds } => { @@ -4346,6 +4640,18 @@ async fn run_batch_step(state: AppState, udid: String, step: BatchStep) -> Resul .ok_or_else(|| AppError::not_found("No accessibility element matched."))?; Ok(json_value!({ "action": "assert", "match": compact_accessibility_node(&found) })) } + BatchStep::AssertNot(payload) => { + let Json(_) = wait_for_absent_element_payload(state, udid, payload).await?; + Ok(json_value!({ "action": "assertNot" })) + } + BatchStep::ScrollUntilVisible(payload) => { + let Json(result) = scroll_until_visible_payload(state, udid, payload).await?; + Ok(json_value!({ + "action": "scrollUntilVisible", + "match": result.get("match").cloned().unwrap_or(Value::Null), + "scrollCount": result.get("scrollCount").cloned().unwrap_or(Value::Null), + })) + } BatchStep::Key { key_code, modifiers, @@ -4781,6 +5087,7 @@ async fn run_batch_step(state: AppState, udid: String, step: BatchStep) -> Resul source.as_deref(), max_depth, include_hidden.unwrap_or(false), + false, ) .await?; Ok(json_value!({ @@ -4799,12 +5106,16 @@ fn query_compact_elements( let mut matches = Vec::new(); if let Some(roots) = snapshot.get("roots").and_then(Value::as_array) { for root in roots { - collect_query_matches(root, selector, limit, &mut matches); - if matches.len() >= limit { + let target_limit = selector.index.map(|index| index + 1).unwrap_or(limit); + collect_query_matches(root, selector, target_limit, &mut matches); + if matches.len() >= target_limit { break; } } } + if let Some(index) = selector.index { + return matches.into_iter().nth(index).into_iter().collect(); + } matches } @@ -4832,6 +5143,16 @@ fn collect_query_matches( fn first_matching_element(snapshot: &Value, selector: &ElementSelectorPayload) -> Option { let roots = snapshot.get("roots")?.as_array()?; + if let Some(index) = selector.index { + let mut matches = Vec::new(); + for root in roots { + collect_query_matches(root, selector, index + 1, &mut matches); + if matches.len() > index { + break; + } + } + return matches.into_iter().nth(index); + } for root in roots { if let Some(found) = first_matching_node(root, selector) { return Some(found.clone()); @@ -4864,48 +5185,122 @@ fn element_matches_selector(node: &Value, selector: &ElementSelectorPayload) -> if selector_is_empty(selector) { return true; } - selector - .element_type - .as_ref() - .is_none_or(|expected| string_fields_match(node, expected, &["type", "role", "className"])) - && selector.id.as_ref().is_none_or(|expected| { - string_fields_match( - node, - expected, - &[ - "AXIdentifier", - "AXUniqueId", - "inspectorId", - "id", - "identifier", - ], - ) - }) - && selector.label.as_ref().is_none_or(|expected| { - string_fields_match( - node, - expected, - &["AXLabel", "label", "title", "text", "name"], - ) - }) - && selector - .value - .as_ref() - .is_none_or(|expected| string_fields_match(node, expected, &["AXValue", "value"])) + let use_regex = selector.regex.unwrap_or(false); + selector.element_type.as_ref().is_none_or(|expected| { + string_fields_match(node, expected, use_regex, &["type", "role", "className"]) + }) && selector.id.as_ref().is_none_or(|expected| { + string_fields_match( + node, + expected, + use_regex, + &[ + "AXIdentifier", + "AXUniqueId", + "inspectorId", + "id", + "identifier", + ], + ) + }) && selector.text.as_ref().is_none_or(|expected| { + string_fields_match( + node, + expected, + use_regex, + &["AXLabel", "label", "title", "text", "name"], + ) + }) && selector.label.as_ref().is_none_or(|expected| { + string_fields_match( + node, + expected, + use_regex, + &["AXLabel", "label", "title", "text", "name"], + ) + }) && selector.value.as_ref().is_none_or(|expected| { + string_fields_match(node, expected, use_regex, &["AXValue", "value"]) + }) && selector.enabled.is_none_or(|expected| { + bool_fields_match( + node, + expected, + &[ + "enabled", + "AXEnabled", + "isEnabled", + "isUserInteractionEnabled", + ], + ) + }) && selector.checked.is_none_or(|expected| { + bool_or_state_fields_match( + node, + expected, + &["checked", "isChecked", "AXChecked"], + &["AXValue", "value"], + &["1", "true", "yes", "on", "checked", "selected"], + ) + }) && selector.focused.is_none_or(|expected| { + bool_fields_match(node, expected, &["focused", "isFocused", "AXFocused"]) + }) && selector.selected.is_none_or(|expected| { + bool_or_state_fields_match( + node, + expected, + &["selected", "isSelected", "AXSelected"], + &["AXValue", "value"], + &["selected", "1", "true", "yes", "on"], + ) + }) } fn selector_is_empty(selector: &ElementSelectorPayload) -> bool { - selector.id.is_none() + selector.text.is_none() + && selector.id.is_none() && selector.label.is_none() && selector.value.is_none() && selector.element_type.is_none() + && selector.enabled.is_none() + && selector.checked.is_none() + && selector.focused.is_none() + && selector.selected.is_none() } -fn string_fields_match(node: &Value, expected: &str, fields: &[&str]) -> bool { +fn string_fields_match(node: &Value, expected: &str, use_regex: bool, fields: &[&str]) -> bool { + let regex = use_regex.then(|| Regex::new(expected).ok()).flatten(); fields .iter() .filter_map(|field| node.get(*field).and_then(Value::as_str)) - .any(|value| value == expected) + .any(|value| { + if let Some(regex) = regex.as_ref() { + regex.is_match(value) + } else { + value == expected + } + }) +} + +fn bool_fields_match(node: &Value, expected: bool, fields: &[&str]) -> bool { + fields + .iter() + .find_map(|field| node.get(*field).and_then(Value::as_bool)) + .is_some_and(|value| value == expected) +} + +fn bool_or_state_fields_match( + node: &Value, + expected: bool, + bool_fields: &[&str], + string_fields: &[&str], + truthy_values: &[&str], +) -> bool { + if bool_fields_match(node, expected, bool_fields) { + return true; + } + string_fields + .iter() + .filter_map(|field| node.get(*field).and_then(Value::as_str)) + .any(|value| { + let truthy = truthy_values + .iter() + .any(|truthy| value.eq_ignore_ascii_case(truthy)); + truthy == expected + }) } fn tap_point_from_snapshot( @@ -4916,6 +5311,7 @@ fn tap_point_from_snapshot( .get("roots") .and_then(Value::as_array) .ok_or_else(|| AppError::not_found("Accessibility snapshot does not contain roots."))?; + let mut seen_matches = 0usize; for root in roots { let root_frame = root .get("frame") @@ -4923,7 +5319,7 @@ fn tap_point_from_snapshot( .ok_or_else(|| AppError::not_found("Accessibility root does not expose a frame."))?; let root_width = number_field(root_frame, "width")?; let root_height = number_field(root_frame, "height")?; - if let Some(node) = first_matching_node(root, selector) { + if let Some(node) = indexed_matching_node(root, selector, &mut seen_matches) { let frame = node .get("frame") .or_else(|| node.get("frameInScreen")) @@ -4939,6 +5335,30 @@ fn tap_point_from_snapshot( Err(AppError::not_found("No accessibility element matched.")) } +fn indexed_matching_node<'a>( + node: &'a Value, + selector: &ElementSelectorPayload, + seen_matches: &mut usize, +) -> Option<&'a Value> { + if element_matches_selector(node, selector) { + if selector.index.unwrap_or(0) == *seen_matches { + return Some(node); + } + *seen_matches += 1; + } + for child in node + .get("children") + .and_then(Value::as_array) + .into_iter() + .flatten() + { + if let Some(found) = indexed_matching_node(child, selector, seen_matches) { + return Some(found); + } + } + None +} + fn normalize_screen_point_from_snapshot( snapshot: &Value, x: f64, @@ -6116,16 +6536,19 @@ async fn run_in_app_inspector_hierarchy( source: InAppHierarchySource, max_depth: Option, include_hidden: bool, + interactive_only: bool, ) -> Result { let max_depth = max_depth.unwrap_or(80); let params = match source { InAppHierarchySource::Automatic => json_value!({ "includeHidden": include_hidden, "maxDepth": max_depth, + "interactiveOnly": interactive_only, }), InAppHierarchySource::UIKit => json_value!({ "includeHidden": include_hidden, "maxDepth": max_depth, + "interactiveOnly": interactive_only, "source": "uikit", }), }; @@ -7093,12 +7516,13 @@ mod tests { normalize_screen_point_from_snapshot, normalized_gesture_coordinates, parse_lsof_tcp_listener, parse_ui_application_service_line, process_identifier_from_accessibility_snapshot, resolved_stream_quality_limits, - split_filter_values, stream_quality_profile, suppress_native_ax_translation_error, - tap_point_from_snapshot, trim_tree_depth, ui_application_foreground_score, - AccessibilityHierarchySource, ElementSelectorPayload, InspectorSession, - InspectorSessionTransport, StreamClientForegroundRegistry, StreamQualityLimits, - StreamQualityPayload, UIKitApplicationServiceDetails, SOURCE_FLUTTER, SOURCE_NATIVE_AX, - SOURCE_NATIVE_SCRIPT, SOURCE_REACT_NATIVE, SOURCE_SWIFTUI, SOURCE_UIKIT, + scroll_input_plan_for_udid, split_filter_values, stream_quality_profile, + suppress_native_ax_translation_error, tap_point_from_snapshot, trim_tree_depth, + ui_application_foreground_score, AccessibilityHierarchySource, ElementSelectorPayload, + InspectorSession, InspectorSessionTransport, ScrollInputBackend, ScrollUntilVisiblePayload, + StreamClientForegroundRegistry, StreamQualityLimits, StreamQualityPayload, + UIKitApplicationServiceDetails, SOURCE_FLUTTER, SOURCE_NATIVE_AX, SOURCE_NATIVE_SCRIPT, + SOURCE_REACT_NATIVE, SOURCE_SWIFTUI, SOURCE_UIKIT, }; use crate::inspector::PublishedInspector; use crate::metrics::counters::ClientStreamStats; @@ -7108,10 +7532,17 @@ mod tests { fn selector() -> ElementSelectorPayload { ElementSelectorPayload { + text: None, id: Some("continue-button".to_owned()), label: Some("Continue".to_owned()), value: None, element_type: Some("Button".to_owned()), + index: None, + enabled: None, + checked: None, + focused: None, + selected: None, + regex: None, } } @@ -7316,6 +7747,29 @@ mod tests { assert!(normalized_gesture_coordinates("orbit", None).is_err()); } + #[test] + fn scroll_until_visible_plans_android_swipe_for_android_ids() { + let payload = ScrollUntilVisiblePayload { + selector: ElementSelectorPayload::default(), + source: Some("android-uiautomator".to_owned()), + max_depth: None, + include_hidden: None, + timeout_ms: None, + poll_ms: None, + direction: Some("down".to_owned()), + duration_ms: Some(225), + steps: Some(7), + }; + + let plan = scroll_input_plan_for_udid("android:Pixel_8", &payload).unwrap(); + + assert_eq!(plan.backend, ScrollInputBackend::Android); + assert_eq!(plan.swipe.start_y, 0.78); + assert_eq!(plan.swipe.end_y, 0.22); + assert_eq!(plan.swipe.duration_ms, 225); + assert_eq!(plan.swipe.steps, 7); + } + #[test] fn compact_accessibility_snapshot_removes_nested_noise_but_keeps_identity() { let compact = compact_accessibility_snapshot(&accessibility_snapshot()); diff --git a/server/src/main.rs b/server/src/main.rs index 091f4bcd..d6635633 100644 --- a/server/src/main.rs +++ b/server/src/main.rs @@ -1,3 +1,4 @@ +mod accessibility; mod android; mod api; mod auth; @@ -17,6 +18,7 @@ mod static_files; mod transport; mod webkit; +use accessibility::interactive_accessibility_snapshot; use anyhow::Context; use api::routes::{router, AppState}; use axum::Router; @@ -35,6 +37,7 @@ use performance::PerformanceRegistry; use qrcode::{render::unicode, QrCode}; use serde::{Deserialize, Serialize}; use serde_json::Value; +use serde_yaml::Value as YamlValue; use simulators::registry::SessionRegistry; use std::collections::{hash_map::DefaultHasher, HashMap, HashSet}; use std::env; @@ -77,6 +80,13 @@ const SERVER_HEALTH_WATCHDOG_HTTP_FAILURE_THRESHOLD: usize = 3; struct Cli { #[arg(long, global = true, hide = true)] server_url: Option, + #[arg( + long, + global = true, + value_name = "SIMULATOR_NAME_OR_UDID", + help = "Override the simulator target for this command" + )] + device: Option, #[command(subcommand)] command: Command, } @@ -138,6 +148,10 @@ enum Command { #[command(subcommand)] command: ProviderCommand, }, + Maestro { + #[command(subcommand)] + command: MaestroCommand, + }, #[command(hide = true)] Serve { #[arg(long, default_value_t = 4310)] @@ -176,50 +190,54 @@ enum Command { #[arg(long, value_enum, default_value_t = ListFormat::CompactJson)] format: ListFormat, }, - Boot { + Use { + #[arg(value_name = "UDID")] udid: String, }, + Boot { + udid: Option, + }, Shutdown { - udid: String, + udid: Option, }, OpenUrl { - udid: String, - url: String, + #[arg(value_name = "UDID_OR_URL", num_args = 1..=2)] + args: Vec, }, Launch { - udid: String, - bundle_id: String, + #[arg(value_name = "UDID_OR_BUNDLE_ID", num_args = 1..=2)] + args: Vec, }, ToggleAppearance { - udid: String, + udid: Option, }, Erase { - udid: String, + udid: Option, }, Install { - udid: String, - app_path: String, + #[arg(value_name = "UDID_OR_APP_PATH", num_args = 1..=2)] + args: Vec, }, Uninstall { - udid: String, - bundle_id: String, + #[arg(value_name = "UDID_OR_BUNDLE_ID", num_args = 1..=2)] + args: Vec, }, Pasteboard { #[command(subcommand)] command: PasteboardCommand, }, Logs { - udid: String, + udid: Option, #[arg(long, default_value_t = 30.0)] seconds: f64, #[arg(long, default_value_t = 200)] limit: usize, }, Processes { - udid: String, + udid: Option, }, Stats { - udid: String, + udid: Option, #[arg(long)] pid: Option, #[arg(long)] @@ -228,14 +246,14 @@ enum Command { interval: f64, }, Sample { - udid: String, + udid: Option, #[arg(long)] pid: Option, #[arg(long, default_value_t = 3)] seconds: u64, }, Screenshot { - udid: String, + udid: Option, #[arg(short, long)] output: Option, #[arg(long)] @@ -244,7 +262,7 @@ enum Command { with_bezel: bool, }, Record { - udid: String, + udid: Option, #[arg(short, long)] output: Option, #[arg(long)] @@ -253,13 +271,13 @@ enum Command { seconds: f64, }, Stream { - udid: String, + udid: Option, #[arg(long, default_value_t = 0)] frames: u64, }, #[command(name = "describe")] DescribeUi { - udid: String, + udid: Option, #[arg(long, value_parser = parse_point)] point: Option<(f64, f64)>, #[arg(long, value_enum, default_value_t = DescribeUiFormat::Json)] @@ -270,13 +288,14 @@ enum Command { max_depth: Option, #[arg(long)] include_hidden: bool, + #[arg(short = 'i', long = "interactive", visible_alias = "interactive-only")] + interactive_only: bool, #[arg(long)] direct: bool, }, Touch { - udid: String, - x: f64, - y: f64, + #[arg(value_name = "UDID_OR_POINT", num_args = 2..=3)] + args: Vec, #[arg(long, default_value = "began")] phase: String, #[arg(long)] @@ -289,9 +308,8 @@ enum Command { delay_ms: u64, }, Tap { - udid: String, - x: Option, - y: Option, + #[arg(value_name = "UDID_OR_TARGET", num_args = 0..)] + args: Vec, #[arg(long)] id: Option, #[arg(long)] @@ -314,7 +332,7 @@ enum Command { post_delay_ms: u64, }, WaitFor { - udid: String, + udid: Option, #[command(flatten)] selector: SelectorArgs, #[arg(long, value_enum, default_value_t = DescribeUiSource::Auto)] @@ -329,7 +347,7 @@ enum Command { poll_interval_ms: u64, }, Assert { - udid: String, + udid: Option, #[command(flatten)] selector: SelectorArgs, #[arg(long, value_enum, default_value_t = DescribeUiSource::Auto)] @@ -344,11 +362,8 @@ enum Command { poll_interval_ms: u64, }, Swipe { - udid: String, - start_x: f64, - start_y: f64, - end_x: f64, - end_y: f64, + #[arg(value_name = "UDID_OR_POINTS", num_args = 4..=5)] + args: Vec, #[arg(long)] normalized: bool, #[arg(long, default_value_t = 350)] @@ -361,8 +376,8 @@ enum Command { post_delay_ms: u64, }, Gesture { - udid: String, - preset: String, + #[arg(value_name = "UDID_OR_PRESET", num_args = 1..=2)] + args: Vec, #[arg(long)] screen_width: Option, #[arg(long)] @@ -379,9 +394,8 @@ enum Command { post_delay_ms: u64, }, Pinch { - udid: String, - center_x: Option, - center_y: Option, + #[arg(value_name = "UDID_OR_CENTER", num_args = 0..=3)] + args: Vec, #[arg(long, default_value_t = 160.0)] start_distance: f64, #[arg(long, default_value_t = 80.0)] @@ -396,9 +410,8 @@ enum Command { steps: u32, }, RotateGesture { - udid: String, - center_x: Option, - center_y: Option, + #[arg(value_name = "UDID_OR_CENTER", num_args = 0..=3)] + args: Vec, #[arg(long, default_value_t = 100.0)] radius: f64, #[arg(long, default_value_t = 90.0)] @@ -411,8 +424,8 @@ enum Command { steps: u32, }, Key { - udid: String, - key: String, + #[arg(value_name = "UDID_OR_KEY", num_args = 1..=2)] + args: Vec, #[arg(long, default_value_t = 0)] modifiers: u32, #[arg(long, default_value_t = 0)] @@ -423,22 +436,22 @@ enum Command { post_delay_ms: u64, }, KeySequence { - udid: String, + udid: Option, #[arg(long = "keycodes", alias = "keys")] keycodes: String, #[arg(long, default_value_t = 100)] delay_ms: u64, }, KeyCombo { - udid: String, + udid: Option, #[arg(long)] modifiers: String, #[arg(long)] key: String, }, Type { - udid: String, - text: Option, + #[arg(value_name = "UDID_OR_TEXT", num_args = 0..=2)] + args: Vec, #[arg(long)] stdin: bool, #[arg(long)] @@ -447,18 +460,18 @@ enum Command { delay_ms: u64, }, Button { - udid: String, - button: String, + #[arg(value_name = "UDID_OR_BUTTON", num_args = 1..=2)] + args: Vec, #[arg(long, default_value_t = 0)] duration_ms: u32, }, Crown { - udid: String, + udid: Option, #[arg(long, default_value_t = 50.0)] delta: f64, }, Batch { - udid: String, + udid: Option, #[arg(long = "step")] steps: Vec, #[arg(long)] @@ -469,22 +482,22 @@ enum Command { continue_on_error: bool, }, DismissKeyboard { - udid: String, + udid: Option, }, Home { - udid: String, + udid: Option, }, AppSwitcher { - udid: String, + udid: Option, }, RotateLeft { - udid: String, + udid: Option, }, RotateRight { - udid: String, + udid: Option, }, ChromeProfile { - udid: String, + udid: Option, }, } @@ -621,6 +634,18 @@ enum ProviderCommand { }, } +#[derive(Subcommand)] +enum MaestroCommand { + Test { + #[arg(value_name = "UDID_OR_FLOW", num_args = 1..=2)] + args: Vec, + #[arg(long)] + artifacts_dir: Option, + #[arg(long)] + continue_on_error: bool, + }, +} + #[derive(Subcommand)] enum ServiceCommand { On { @@ -696,11 +721,11 @@ enum CoreSimulatorCommand { #[derive(Subcommand)] enum PasteboardCommand { Get { - udid: String, + udid: Option, }, Set { - udid: String, - text: Option, + #[arg(value_name = "UDID_OR_TEXT", num_args = 0..=2)] + args: Vec, #[arg(long)] stdin: bool, #[arg(long)] @@ -851,6 +876,18 @@ struct DaemonMetadata { local_stream_fps: Option, } +#[derive(Clone, Debug, Serialize, Deserialize)] +#[serde(rename_all = "camelCase")] +struct ProjectDeviceSelection { + project_root: PathBuf, + udid: String, + #[serde(default, skip_serializing_if = "Option::is_none")] + name: Option, + #[serde(default, skip_serializing_if = "Option::is_none")] + runtime_name: Option, + selected_at: u64, +} + fn default_daemon_port() -> u16 { 4310 } @@ -1780,6 +1817,47 @@ fn daemon_log_path_for_root(root: &Path) -> anyhow::Result { .join(format!("{:016x}.log", hasher.finish()))) } +fn read_project_device_selection() -> anyhow::Result> { + let root = project_root()?; + let path = project_device_selection_path_for_root(&root)?; + if !path.exists() { + return Ok(None); + } + let data = fs::read_to_string(&path).with_context(|| format!("read {}", path.display()))?; + let selection = serde_json::from_str::(&data) + .with_context(|| format!("parse simulator selection {}", path.display()))?; + if selection.project_root != root { + return Ok(None); + } + Ok(Some(selection)) +} + +fn write_project_device_selection(selection: &ProjectDeviceSelection) -> anyhow::Result { + let path = project_device_selection_path_for_root(&selection.project_root)?; + if let Some(parent) = path.parent() { + fs::create_dir_all(parent)?; + } + fs::write(&path, serde_json::to_vec_pretty(selection)?) + .with_context(|| format!("write {}", path.display()))?; + Ok(path) +} + +fn project_device_selection_path_for_root(root: &Path) -> anyhow::Result { + let mut hasher = DefaultHasher::new(); + root.to_string_lossy().hash(&mut hasher); + Ok(simdeck_user_state_dir() + .join("default-devices") + .join(format!("{:016x}.json", hasher.finish()))) +} + +fn simdeck_user_state_dir() -> PathBuf { + env::var_os("HOME") + .filter(|value| !value.is_empty()) + .map(PathBuf::from) + .map(|home| home.join(".simdeck")) + .unwrap_or_else(|| env::temp_dir().join("simdeck")) +} + fn daemon_metadata_paths() -> anyhow::Result> { let dir = env::temp_dir().join("simdeck"); if !dir.exists() { @@ -1876,10 +1954,12 @@ fn is_known_command(value: &str) -> bool { value, "ui" | "pair" | "daemon" + | "maestro" | "service" | "core-simulator" | "simctl-service" | "list" + | "use" | "boot" | "shutdown" | "open-url" @@ -1920,7 +2000,15 @@ fn is_known_command(value: &str) -> bool { fn run_no_command_action(action: NoCommandAction) -> anyhow::Result<()> { match action { - NoCommandAction::Foreground(selector) => run_foreground_ui(selector), + NoCommandAction::Foreground(selector) => { + let selector = selector.or_else(|| { + read_project_device_selection() + .ok() + .flatten() + .map(|selection| selection.udid) + }); + run_foreground_ui(selector) + } NoCommandAction::Detached => start_detached_daemon(DaemonLaunchOptions::default()), NoCommandAction::Kill => stop_project_daemon(), NoCommandAction::Restart => restart_detached_daemon(DaemonLaunchOptions::default()), @@ -2395,6 +2483,287 @@ fn list_studio_simulators(server_url: &str) -> anyhow::Result, + global_selector: Option<&str>, + explicit_server_url: Option<&str>, +) -> anyhow::Result { + if let Some(udid) = positional.map(str::trim).filter(|value| !value.is_empty()) { + return Ok(udid.to_owned()); + } + + let selector = global_selector + .map(str::to_owned) + .or_else(|| env::var("SIMDECK_DEVICE").ok()) + .or_else(|| env::var("SIMDECK_UDID").ok()) + .map(|value| value.trim().to_owned()) + .filter(|value| !value.is_empty()); + + if let Some(selector) = selector { + if android::is_android_id(&selector) || looks_like_device_selector(&selector) { + return Ok(selector); + } + let server_url = command_service_url(explicit_server_url)?; + if let Some(simulator) = select_studio_simulator(&server_url, &selector)? { + return Ok(simulator.udid); + } + return Ok(selector); + } + + if let Some(selection) = read_project_device_selection()? { + let udid = selection.udid.trim(); + if !udid.is_empty() { + return Ok(udid.to_owned()); + } + } + + let server_url = command_service_url(explicit_server_url)?; + if let Some(simulator) = infer_default_cli_simulator(&server_url)? { + return Ok(simulator.udid); + } + + let simulators = list_studio_simulators(&server_url)?; + let booted = simulators + .iter() + .filter(|simulator| simulator.is_booted) + .collect::>(); + if booted.len() > 1 { + anyhow::bail!( + "Multiple booted simulators are available. Pass a UDID, run `simdeck use `, use --device, or set SIMDECK_DEVICE." + ); + } + if simulators.is_empty() { + anyhow::bail!("No simulators are available. Boot one or pass a UDID explicitly."); + } + anyhow::bail!( + "No default simulator could be inferred. Pass a UDID, run `simdeck use `, use --device, or set SIMDECK_DEVICE." + ) +} + +fn infer_default_cli_simulator( + server_url: &str, +) -> anyhow::Result> { + let simulators = list_studio_simulators(server_url)?; + let booted = simulators + .iter() + .filter(|simulator| simulator.is_booted) + .cloned() + .collect::>(); + if booted.len() == 1 { + return Ok(booted.into_iter().next()); + } + if booted.is_empty() && simulators.len() == 1 { + return Ok(simulators.into_iter().next()); + } + Ok(None) +} + +fn parse_tap_command_args( + args: Vec, + id: Option, + label: Option, + value: Option, + element_type: Option, +) -> anyhow::Result { + let mut target = TapCommandTarget { + selector: ElementSelector { + id, + label, + value, + element_type, + }, + ..Default::default() + }; + + let args = args + .into_iter() + .map(|arg| arg.trim().to_owned()) + .filter(|arg| !arg.is_empty()) + .collect::>(); + + if !target.selector.is_empty() { + match args.as_slice() { + [] => return Ok(target), + [udid] => { + target.udid = Some(udid.clone()); + return Ok(target); + } + _ => anyhow::bail!( + "tap accepts at most one positional UDID when selector flags are used." + ), + } + } + + if args.is_empty() { + return Ok(target); + } + + let (udid, target_args) = if args.len() >= 2 && looks_like_device_selector(&args[0]) { + (Some(args[0].clone()), &args[1..]) + } else { + (None, args.as_slice()) + }; + target.udid = udid; + + if target_args.len() == 2 { + if let (Some(x), Some(y)) = ( + parse_f64_arg(&target_args[0]), + parse_f64_arg(&target_args[1]), + ) { + target.x = Some(x); + target.y = Some(y); + return Ok(target); + } + } + + if target_args.len() == 1 && parse_f64_arg(&target_args[0]).is_some() { + anyhow::bail!("tap requires both x and y coordinates."); + } + if target_args.iter().any(|arg| parse_f64_arg(arg).is_some()) { + anyhow::bail!("tap coordinates must be provided as exactly two numeric values."); + } + + target.selector.label = Some(target_args.join(" ")); + Ok(target) +} + +fn project_device_selection_for_selector( + selector: &str, + explicit_server_url: Option<&str>, +) -> anyhow::Result { + let selector = selector.trim(); + if selector.is_empty() { + anyhow::bail!("simdeck use requires a simulator UDID or name."); + } + + let project_root = project_root()?; + if android::is_android_id(selector) || looks_like_device_selector(selector) { + return Ok(ProjectDeviceSelection { + project_root, + udid: selector.to_owned(), + name: None, + runtime_name: None, + selected_at: now_secs(), + }); + } + + let server_url = command_service_url(explicit_server_url)?; + let matched = select_studio_simulator(&server_url, selector)?; + if let Some(simulator) = matched { + return Ok(ProjectDeviceSelection { + project_root, + udid: simulator.udid, + name: Some(simulator.name), + runtime_name: simulator.runtime_name, + selected_at: now_secs(), + }); + } + + anyhow::bail!("No simulator matched {selector:?}. Run `simdeck list` to see available UDIDs.") +} + +fn parse_optional_udid_value_args( + command: &str, + args: Vec, + value_name: &str, +) -> anyhow::Result<(Option, String)> { + let args = clean_cli_args(args); + match args.as_slice() { + [value] => Ok((None, value.clone())), + [udid, value] => Ok((Some(udid.clone()), value.clone())), + [] => anyhow::bail!("{command} requires {value_name}."), + _ => anyhow::bail!("{command} accepts either {value_name} or UDID {value_name}."), + } +} + +fn parse_optional_udid_text_args( + command: &str, + args: Vec, + has_non_positional_input: bool, +) -> anyhow::Result<(Option, Option)> { + let args = clean_cli_args(args); + if has_non_positional_input { + return match args.as_slice() { + [] => Ok((None, None)), + [udid] => Ok((Some(udid.clone()), None)), + _ => anyhow::bail!( + "{command} accepts at most one positional UDID with --stdin or --file." + ), + }; + } + match args.as_slice() { + [] => Ok((None, None)), + [text] => Ok((None, Some(text.clone()))), + [udid, text] => Ok((Some(udid.clone()), Some(text.clone()))), + _ => anyhow::bail!("{command} accepts either TEXT or UDID TEXT. Quote multi-word text."), + } +} + +fn parse_optional_udid_f64_args( + command: &str, + args: Vec, + expected_values: usize, +) -> anyhow::Result<(Option, Vec)> { + let args = clean_cli_args(args); + let (udid, values) = match args.len() { + len if len == expected_values => (None, args.as_slice()), + len if len == expected_values + 1 => (Some(args[0].clone()), &args[1..]), + _ => anyhow::bail!( + "{command} accepts either {expected_values} numeric values or UDID plus {expected_values} numeric values." + ), + }; + let mut parsed = Vec::with_capacity(values.len()); + for value in values { + parsed.push(parse_f64_arg(value).ok_or_else(|| { + anyhow::anyhow!("{command} expected a finite number, got {value:?}.") + })?); + } + Ok((udid, parsed)) +} + +fn parse_optional_udid_point_args( + command: &str, + args: Vec, +) -> anyhow::Result<(Option, Option, Option)> { + let args = clean_cli_args(args); + match args.as_slice() { + [] => Ok((None, None, None)), + [udid] => Ok((Some(udid.clone()), None, None)), + [x, y] => Ok(( + None, + Some(parse_required_f64_arg(command, x)?), + Some(parse_required_f64_arg(command, y)?), + )), + [udid, x, y] => Ok(( + Some(udid.clone()), + Some(parse_required_f64_arg(command, x)?), + Some(parse_required_f64_arg(command, y)?), + )), + _ => anyhow::bail!("{command} accepts [UDID] or [UDID] CENTER_X CENTER_Y."), + } +} + +fn parse_required_f64_arg(command: &str, value: &str) -> anyhow::Result { + parse_f64_arg(value) + .ok_or_else(|| anyhow::anyhow!("{command} expected a finite number, got {value:?}.")) +} + +fn clean_cli_args(args: Vec) -> Vec { + args.into_iter() + .map(|arg| arg.trim().to_owned()) + .filter(|arg| !arg.is_empty()) + .collect() +} + +fn parse_f64_arg(value: &str) -> Option { + value.parse::().ok().filter(|value| value.is_finite()) +} + +fn looks_like_device_selector(value: &str) -> bool { + android::is_android_id(value) + || (value.len() == 36 && value.chars().all(|ch| ch.is_ascii_hexdigit() || ch == '-')) +} + fn studio_provider_bridge_script() -> anyhow::Result { let mut candidates = Vec::new(); if let Ok(root) = project_root() { @@ -2550,11 +2919,19 @@ fn main() -> anyhow::Result<()> { let cli = Cli::parse(); let explicit_server_url = cli.server_url.clone(); + let device_selector = cli.device.clone(); let service_url = explicit_server_url .clone() .or_else(|| env::var("SIMDECK_SERVER_URL").ok()) .filter(|value| !value.trim().is_empty()); let bridge = NativeBridge; + let resolve_device_udid = |udid: Option<&str>| -> anyhow::Result { + resolve_cli_device_udid( + udid, + device_selector.as_deref(), + explicit_server_url.as_deref(), + ) + }; match cli.command { Command::Ui { @@ -2743,6 +3120,27 @@ fn main() -> anyhow::Result<()> { }), }, Command::Provider { command } => run_provider_command(command), + Command::Maestro { command } => match command { + MaestroCommand::Test { + args, + artifacts_dir, + continue_on_error, + } => { + let (udid, flow) = parse_optional_udid_value_args("maestro test", args, "FLOW")?; + let udid = resolve_device_udid(udid.as_deref())?; + let flow = PathBuf::from(flow); + let service_url = command_service_url(explicit_server_url.as_deref())?; + let report = + run_maestro_flow(&service_url, &udid, &flow, artifacts_dir, continue_on_error)?; + let ok = report.get("ok").and_then(Value::as_bool).unwrap_or(false); + println_json(&report)?; + if ok { + Ok(()) + } else { + anyhow::bail!("Maestro-compatible flow failed.") + } + } + }, Command::Serve { port, bind, @@ -2869,7 +3267,23 @@ fn main() -> anyhow::Result<()> { print_list_simulators(&simulators, format)?; Ok(()) } + Command::Use { udid } => { + let selection = + project_device_selection_for_selector(&udid, explicit_server_url.as_deref())?; + let path = write_project_device_selection(&selection)?; + println_json(&serde_json::json!({ + "ok": true, + "action": "use", + "udid": selection.udid, + "name": selection.name, + "runtimeName": selection.runtime_name, + "projectRoot": selection.project_root, + "path": path, + }))?; + Ok(()) + } Command::Boot { udid } => { + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; service_post_ok(&service_url, &udid, "boot", &Value::Null)?; println!( @@ -2881,6 +3295,7 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::Shutdown { udid } => { + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; service_post_ok(&service_url, &udid, "shutdown", &Value::Null)?; println!( @@ -2891,7 +3306,9 @@ fn main() -> anyhow::Result<()> { ); Ok(()) } - Command::OpenUrl { udid, url } => { + Command::OpenUrl { args } => { + let (udid, url) = parse_optional_udid_value_args("open-url", args, "URL")?; + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; service_open_url(&service_url, &udid, &url)?; println!( @@ -2902,7 +3319,9 @@ fn main() -> anyhow::Result<()> { ); Ok(()) } - Command::Launch { udid, bundle_id } => { + Command::Launch { args } => { + let (udid, bundle_id) = parse_optional_udid_value_args("launch", args, "BUNDLE_ID")?; + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; service_launch(&service_url, &udid, &bundle_id)?; println!( @@ -2914,6 +3333,7 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::ToggleAppearance { udid } => { + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; service_post_ok(&service_url, &udid, "toggle-appearance", &Value::Null)?; println_json( @@ -2922,12 +3342,15 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::Erase { udid } => { + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; service_post_ok(&service_url, &udid, "erase", &Value::Null)?; println_json(&serde_json::json!({ "ok": true, "udid": udid, "action": "erase" }))?; Ok(()) } - Command::Install { udid, app_path } => { + Command::Install { args } => { + let (udid, app_path) = parse_optional_udid_value_args("install", args, "APP_PATH")?; + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; service_post_ok( &service_url, @@ -2940,7 +3363,9 @@ fn main() -> anyhow::Result<()> { )?; Ok(()) } - Command::Uninstall { udid, bundle_id } => { + Command::Uninstall { args } => { + let (udid, bundle_id) = parse_optional_udid_value_args("uninstall", args, "BUNDLE_ID")?; + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; service_post_ok( &service_url, @@ -2955,6 +3380,7 @@ fn main() -> anyhow::Result<()> { } Command::Pasteboard { command } => match command { PasteboardCommand::Get { udid } => { + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; let text = service_get_json( &service_url, @@ -2967,12 +3393,14 @@ fn main() -> anyhow::Result<()> { println_json(&serde_json::json!({ "udid": udid, "text": text }))?; Ok(()) } - PasteboardCommand::Set { - udid, - text, - stdin, - file, - } => { + PasteboardCommand::Set { args, stdin, file } => { + let has_non_positional_input = stdin || file.is_some(); + let (udid, text) = parse_optional_udid_text_args( + "pasteboard set", + args, + has_non_positional_input, + )?; + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; let text = read_text_input(text, stdin, file)?; service_post_ok( @@ -2992,6 +3420,7 @@ fn main() -> anyhow::Result<()> { seconds, limit, } => { + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; let filters = native::bridge::LogFilters::new(Vec::new(), Vec::new(), String::new()); let _ = filters; @@ -3009,6 +3438,7 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::Processes { udid } => { + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; let processes = service_get_json( &service_url, @@ -3023,6 +3453,7 @@ fn main() -> anyhow::Result<()> { watch, interval, } => { + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; if watch { run_stats_watch(&service_url, &udid, pid, interval)?; @@ -3033,6 +3464,7 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::Sample { udid, pid, seconds } => { + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; let pid = match pid { Some(pid) => pid, @@ -3061,6 +3493,7 @@ fn main() -> anyhow::Result<()> { stdout, with_bezel, } => { + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; let query = if with_bezel { "?bezel=true" } else { "" }; let png = service_get_bytes( @@ -3094,6 +3527,7 @@ fn main() -> anyhow::Result<()> { stdout, seconds, } => { + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; let mp4 = service_post_bytes( &service_url, @@ -3120,7 +3554,10 @@ fn main() -> anyhow::Result<()> { } Ok(()) } - Command::Stream { udid, frames } => run_stream_stdout(&bridge, udid, frames), + Command::Stream { udid, frames } => { + let udid = resolve_device_udid(udid.as_deref())?; + run_stream_stdout(&bridge, udid, frames) + } Command::DescribeUi { udid, point, @@ -3128,9 +3565,19 @@ fn main() -> anyhow::Result<()> { source, max_depth, include_hidden, + interactive_only, direct, } => { - let service_url = command_service_url(explicit_server_url.as_deref())?; + let udid = resolve_cli_device_udid( + udid.as_deref(), + device_selector.as_deref(), + explicit_server_url.as_deref(), + )?; + let service_url = if direct { + String::new() + } else { + command_service_url(explicit_server_url.as_deref())? + }; let snapshot = describe_ui_snapshot( &bridge, &udid, @@ -3138,6 +3585,7 @@ fn main() -> anyhow::Result<()> { source, max_depth, include_hidden, + interactive_only, direct, &service_url, )?; @@ -3145,15 +3593,17 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::Touch { - udid, - x, - y, + args, phase, normalized, down, up, delay_ms, } => { + let (udid, points) = parse_optional_udid_f64_args("touch", args, 2)?; + let udid = resolve_device_udid(udid.as_deref())?; + let x = points[0]; + let y = points[1]; let android_device = android::is_android_id(&udid); if android_device && !normalized { anyhow::bail!("Android touch coordinates require --normalized."); @@ -3205,9 +3655,7 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::Tap { - udid, - x, - y, + args, id, label, value, @@ -3219,8 +3667,29 @@ fn main() -> anyhow::Result<()> { pre_delay_ms, post_delay_ms, } => { + let target = parse_tap_command_args(args, id, label, value, element_type)?; + let uses_inferred_device = target.udid.is_none(); + let uses_selector = !target.selector.is_empty(); + let udid = resolve_cli_device_udid( + target.udid.as_deref(), + device_selector.as_deref(), + explicit_server_url.as_deref(), + )?; + let x = target.x; + let y = target.y; + let ElementSelector { + id, + label, + value, + element_type, + } = target.selector; + let preferred_service_url = if uses_inferred_device || uses_selector { + Some(command_service_url(explicit_server_url.as_deref())?) + } else { + service_url.clone() + }; let command_server_url = - command_service_url_for_udid(&udid, &explicit_server_url, &service_url)?; + command_service_url_for_udid(&udid, &explicit_server_url, &preferred_service_url)?; if let (Some(server_url), Some(x), Some(y), true, None, None, None, None) = ( command_server_url.as_deref(), x, @@ -3295,6 +3764,7 @@ fn main() -> anyhow::Result<()> { timeout_ms, poll_interval_ms, } => { + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; let result = service_wait_for_selector( &service_url, @@ -3319,6 +3789,7 @@ fn main() -> anyhow::Result<()> { timeout_ms, poll_interval_ms, } => { + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; let result = service_wait_for_selector( &service_url, @@ -3335,17 +3806,19 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::Swipe { - udid, - start_x, - start_y, - end_x, - end_y, + args, normalized, duration_ms, steps, pre_delay_ms, post_delay_ms, } => { + let (udid, points) = parse_optional_udid_f64_args("swipe", args, 4)?; + let udid = resolve_device_udid(udid.as_deref())?; + let start_x = points[0]; + let start_y = points[1]; + let end_x = points[2]; + let end_y = points[3]; let android_device = android::is_android_id(&udid); if android_device && !normalized { anyhow::bail!("Android swipe coordinates require --normalized."); @@ -3405,8 +3878,7 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::Gesture { - udid, - preset, + args, screen_width, screen_height, normalized, @@ -3415,6 +3887,8 @@ fn main() -> anyhow::Result<()> { pre_delay_ms, post_delay_ms, } => { + let (udid, preset) = parse_optional_udid_value_args("gesture", args, "PRESET")?; + let udid = resolve_device_udid(udid.as_deref())?; let android_device = android::is_android_id(&udid); let command_server_url = command_service_url_for_udid(&udid, &explicit_server_url, &service_url)?; @@ -3494,9 +3968,7 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::Pinch { - udid, - center_x, - center_y, + args, start_distance, end_distance, angle_degrees, @@ -3504,6 +3976,8 @@ fn main() -> anyhow::Result<()> { duration_ms, steps, } => { + let (udid, center_x, center_y) = parse_optional_udid_point_args("pinch", args)?; + let udid = resolve_device_udid(udid.as_deref())?; if android::is_android_id(&udid) { anyhow::bail!("Android pinch gestures are not supported by the ADB input bridge."); } @@ -3523,15 +3997,16 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::RotateGesture { - udid, - center_x, - center_y, + args, radius, degrees, normalized, duration_ms, steps, } => { + let (udid, center_x, center_y) = + parse_optional_udid_point_args("rotate-gesture", args)?; + let udid = resolve_device_udid(udid.as_deref())?; if android::is_android_id(&udid) { anyhow::bail!("Android rotate gestures are not supported by the ADB input bridge."); } @@ -3554,13 +4029,14 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::Key { - udid, - key, + args, modifiers, duration_ms, pre_delay_ms, post_delay_ms, } => { + let (udid, key) = parse_optional_udid_value_args("key", args, "KEY")?; + let udid = resolve_device_udid(udid.as_deref())?; let key_code = parse_hid_key(&key)?; sleep_ms(pre_delay_ms); let command_server_url = @@ -3584,6 +4060,7 @@ fn main() -> anyhow::Result<()> { keycodes, delay_ms, } => { + let udid = resolve_device_udid(udid.as_deref())?; let keys = parse_key_list(&keycodes)?; let command_server_url = command_service_url_for_udid(&udid, &explicit_server_url, &service_url)?; @@ -3608,6 +4085,7 @@ fn main() -> anyhow::Result<()> { modifiers, key, } => { + let udid = resolve_device_udid(udid.as_deref())?; let modifier_mask = parse_modifier_mask(&modifiers)?; let key_code = parse_hid_key(&key)?; let command_server_url = @@ -3621,12 +4099,15 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::Type { - udid, - text, + args, stdin, file, delay_ms, } => { + let has_non_positional_input = stdin || file.is_some(); + let (udid, text) = + parse_optional_udid_text_args("type", args, has_non_positional_input)?; + let udid = resolve_device_udid(udid.as_deref())?; let text = read_text_input(text, stdin, file)?; if android::is_android_id(&udid) { let server_url = command_service_url(explicit_server_url.as_deref())?; @@ -3646,11 +4127,9 @@ fn main() -> anyhow::Result<()> { println_json(&serde_json::json!({ "ok": true, "udid": udid, "action": "type" }))?; Ok(()) } - Command::Button { - udid, - button, - duration_ms, - } => { + Command::Button { args, duration_ms } => { + let (udid, button) = parse_optional_udid_value_args("button", args, "BUTTON")?; + let udid = resolve_device_udid(udid.as_deref())?; let command_server_url = command_service_url_for_udid(&udid, &explicit_server_url, &service_url)?; if let Some(server_url) = command_server_url.as_deref() { @@ -3664,6 +4143,7 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::Crown { udid, delta } => { + let udid = resolve_device_udid(udid.as_deref())?; if let Some(server_url) = service_url.as_deref() { service_crown(server_url, &udid, delta)?; } else { @@ -3681,6 +4161,7 @@ fn main() -> anyhow::Result<()> { stdin, continue_on_error, } => { + let udid = resolve_device_udid(udid.as_deref())?; let command_server_url = command_service_url_for_udid(&udid, &explicit_server_url, &service_url)?; let report = if let Some(server_url) = command_server_url.as_deref() { @@ -3698,6 +4179,7 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::DismissKeyboard { udid } => { + let udid = resolve_device_udid(udid.as_deref())?; let command_server_url = command_service_url_for_udid(&udid, &explicit_server_url, &service_url)?; if let Some(server_url) = command_server_url.as_deref() { @@ -3714,6 +4196,7 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::Home { udid } => { + let udid = resolve_device_udid(udid.as_deref())?; let command_server_url = command_service_url_for_udid(&udid, &explicit_server_url, &service_url)?; if let Some(server_url) = command_server_url.as_deref() { @@ -3725,6 +4208,7 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::AppSwitcher { udid } => { + let udid = resolve_device_udid(udid.as_deref())?; let command_server_url = command_service_url_for_udid(&udid, &explicit_server_url, &service_url)?; if let Some(server_url) = command_server_url.as_deref() { @@ -3738,6 +4222,7 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::RotateLeft { udid } => { + let udid = resolve_device_udid(udid.as_deref())?; let command_server_url = command_service_url_for_udid(&udid, &explicit_server_url, &service_url)?; if let Some(server_url) = command_server_url.as_deref() { @@ -3751,6 +4236,7 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::RotateRight { udid } => { + let udid = resolve_device_udid(udid.as_deref())?; let command_server_url = command_service_url_for_udid(&udid, &explicit_server_url, &service_url)?; if let Some(server_url) = command_server_url.as_deref() { @@ -3764,6 +4250,7 @@ fn main() -> anyhow::Result<()> { Ok(()) } Command::ChromeProfile { udid } => { + let udid = resolve_device_udid(udid.as_deref())?; let service_url = command_service_url(explicit_server_url.as_deref())?; let profile = service_get_json( &service_url, @@ -3964,7 +4451,7 @@ fn now_secs() -> u64 { .as_secs() } -#[derive(Clone, Debug, Default)] +#[derive(Clone, Debug, Default, PartialEq)] struct ElementSelector { id: Option, label: Option, @@ -3972,6 +4459,23 @@ struct ElementSelector { element_type: Option, } +impl ElementSelector { + fn is_empty(&self) -> bool { + self.id.is_none() + && self.label.is_none() + && self.value.is_none() + && self.element_type.is_none() + } +} + +#[derive(Clone, Debug, Default, PartialEq)] +struct TapCommandTarget { + udid: Option, + x: Option, + y: Option, + selector: ElementSelector, +} + #[derive(Clone, Copy, Debug)] struct GestureCoordinates { start_x: f64, @@ -4258,6 +4762,7 @@ fn describe_ui_snapshot( source: DescribeUiSource, max_depth: Option, include_hidden: bool, + interactive_only: bool, direct: bool, server_url: &str, ) -> anyhow::Result { @@ -4281,6 +4786,7 @@ fn describe_ui_snapshot( source, max_depth, include_hidden, + interactive_only, server_url, ) { Ok(snapshot) => return Ok(snapshot), @@ -4297,7 +4803,12 @@ fn describe_ui_snapshot( ); } - Ok(bridge.accessibility_snapshot_with_max_depth(udid, point, max_depth)?) + let snapshot = bridge.accessibility_snapshot_with_max_depth(udid, point, max_depth)?; + Ok(if interactive_only && point.is_none() { + interactive_accessibility_snapshot(&snapshot) + } else { + snapshot + }) } fn fetch_service_accessibility_tree( @@ -4305,6 +4816,7 @@ fn fetch_service_accessibility_tree( source: DescribeUiSource, max_depth: Option, include_hidden: bool, + interactive_only: bool, server_url: &str, ) -> anyhow::Result { let mut query = vec![format!("source={}", source.as_query_value())]; @@ -4314,6 +4826,9 @@ fn fetch_service_accessibility_tree( if include_hidden { query.push("includeHidden=true".to_owned()); } + if interactive_only { + query.push("interactiveOnly=true".to_owned()); + } let path = format!( "/api/simulators/{}/accessibility-tree?{}", url_path_component(udid), @@ -5671,6 +6186,428 @@ fn parse_modifier_mask(value: &str) -> Result { Ok(mask) } +fn run_maestro_flow( + server_url: &str, + udid: &str, + flow: &Path, + artifacts_dir: Option, + continue_on_error: bool, +) -> anyhow::Result { + let raw = fs::read_to_string(flow) + .with_context(|| format!("read Maestro flow {}", flow.display()))?; + let yaml = parse_maestro_flow_yaml(&raw) + .with_context(|| format!("parse Maestro flow {}", flow.display()))?; + let commands = maestro_commands_from_flow(&yaml)?; + let artifact_root = artifacts_dir.unwrap_or_else(|| { + PathBuf::from("simdeck-artifacts").join( + flow.file_stem() + .and_then(|value| value.to_str()) + .unwrap_or("maestro-flow"), + ) + }); + fs::create_dir_all(&artifact_root)?; + + let mut steps = Vec::new(); + let mut failures = Vec::new(); + for (index, command) in commands.iter().enumerate() { + let started = Instant::now(); + let result = run_maestro_command(server_url, udid, command, &artifact_root); + match result { + Ok(detail) => steps.push(serde_json::json!({ + "index": index, + "ok": true, + "command": maestro_command_name(command), + "elapsedMs": started.elapsed().as_millis() as u64, + "detail": detail, + })), + Err(error) => { + let message = error.to_string(); + let screenshot = + capture_maestro_failure_screenshot(server_url, udid, &artifact_root, index + 1) + .ok(); + steps.push(serde_json::json!({ + "index": index, + "ok": false, + "command": maestro_command_name(command), + "elapsedMs": started.elapsed().as_millis() as u64, + "error": message, + "screenshot": screenshot, + })); + failures.push(message); + if !continue_on_error { + break; + } + } + } + } + + Ok(serde_json::json!({ + "ok": failures.is_empty(), + "flow": flow, + "udid": udid, + "steps": steps, + "failureCount": failures.len(), + "artifactsDir": artifact_root, + })) +} + +fn parse_maestro_flow_yaml(raw: &str) -> anyhow::Result { + let mut documents = Vec::new(); + for document in serde_yaml::Deserializer::from_str(raw) { + documents.push(YamlValue::deserialize(document)?); + } + match documents.len() { + 0 => Err(anyhow::anyhow!("Maestro flow is empty.")), + 1 => Ok(documents.remove(0)), + _ => { + let app_id = documents + .first() + .and_then(|value| yaml_string_or_field(value, "appId")); + let mut commands = documents + .pop() + .ok_or_else(|| anyhow::anyhow!("Maestro flow is empty."))?; + if let Some(app_id) = app_id { + fill_empty_launch_app_commands(&mut commands, &app_id); + } + Ok(commands) + } + } +} + +fn fill_empty_launch_app_commands(commands: &mut YamlValue, app_id: &str) { + let Some(commands) = commands.as_sequence_mut() else { + return; + }; + for command in commands { + if command.as_str() == Some("launchApp") { + let mut mapping = serde_yaml::Mapping::new(); + mapping.insert( + YamlValue::String("launchApp".to_owned()), + YamlValue::String(app_id.to_owned()), + ); + *command = YamlValue::Mapping(mapping); + continue; + } + let Some(mapping) = command.as_mapping_mut() else { + continue; + }; + let key = YamlValue::String("launchApp".to_owned()); + let Some(value) = mapping.get_mut(&key) else { + continue; + }; + if value.is_null() || value.as_mapping().is_some_and(|mapping| mapping.is_empty()) { + *value = YamlValue::String(app_id.to_owned()); + } + } +} + +fn maestro_commands_from_flow(flow: &YamlValue) -> anyhow::Result> { + match flow { + YamlValue::Sequence(commands) => Ok(commands.clone()), + YamlValue::Mapping(mapping) => mapping + .get(YamlValue::String("commands".to_owned())) + .and_then(YamlValue::as_sequence) + .cloned() + .ok_or_else(|| { + anyhow::anyhow!("Maestro flow must be a command list or contain `commands`.") + }), + _ => Err(anyhow::anyhow!( + "Maestro flow must be a command list or contain `commands`." + )), + } +} + +fn run_maestro_command( + server_url: &str, + udid: &str, + command: &YamlValue, + artifacts_dir: &Path, +) -> anyhow::Result { + let null_value = YamlValue::Null; + let (name, value) = if let Some(name) = command.as_str() { + (name, &null_value) + } else { + let Some(mapping) = command.as_mapping() else { + anyhow::bail!("Maestro command must be a string or mapping."); + }; + if mapping.len() != 1 { + anyhow::bail!("Maestro command must contain exactly one action."); + } + let (name, value) = mapping.iter().next().unwrap(); + ( + name.as_str() + .ok_or_else(|| anyhow::anyhow!("Maestro command name must be a string."))?, + value, + ) + }; + match name { + "launchApp" => { + let bundle_id = maestro_bundle_id(value)?; + service_launch(server_url, udid, &bundle_id)?; + Ok(serde_json::json!({ "bundleId": bundle_id })) + } + "openLink" => { + let url = yaml_string_or_field(value, "link") + .or_else(|| yaml_string_or_field(value, "url")) + .ok_or_else(|| anyhow::anyhow!("openLink requires a URL."))?; + service_open_url(server_url, udid, &url)?; + Ok(serde_json::json!({ "url": url })) + } + "tapOn" => { + let body = maestro_tap_body(value)?; + service_tap_element(server_url, udid, body)?; + Ok(Value::Null) + } + "inputText" => { + let text = yaml_string_or_field(value, "text") + .ok_or_else(|| anyhow::anyhow!("inputText requires text."))?; + service_batch( + server_url, + udid, + vec![serde_json::json!({ "action": "type", "text": text })], + false, + )?; + Ok(Value::Null) + } + "eraseText" => { + let count = yaml_u64_or_field(value, "charactersToErase").unwrap_or(64); + let keys = vec![42u16; count as usize]; + service_key_sequence(server_url, udid, &keys, 5)?; + Ok(serde_json::json!({ "charactersToErase": count })) + } + "pressKey" => { + let key = yaml_string_or_field(value, "key") + .ok_or_else(|| anyhow::anyhow!("pressKey requires a key."))?; + service_key(server_url, udid, parse_hid_key(&key)?, 0)?; + Ok(serde_json::json!({ "key": key })) + } + "assertVisible" => { + let selector = maestro_selector(value)?; + service_wait_for(server_url, udid, "assert", selector, 5_000)?; + Ok(Value::Null) + } + "assertNotVisible" => { + let selector = maestro_selector(value)?; + service_wait_for(server_url, udid, "assert-not", selector, 5_000)?; + Ok(Value::Null) + } + "scrollUntilVisible" => { + let selector_value = yaml_field(value, "element").unwrap_or(value); + let selector = maestro_selector(selector_value)?; + let direction = + yaml_string_or_field(value, "direction").unwrap_or_else(|| "down".to_owned()); + service_post_ok( + server_url, + udid, + "scroll-until-visible", + &serde_json::json!({ + "selector": selector, + "direction": direction, + "timeoutMs": yaml_u64_or_field(value, "timeout").unwrap_or(10_000), + }), + )?; + Ok(Value::Null) + } + "swipe" => { + let direction = + yaml_string_or_field(value, "direction").unwrap_or_else(|| "up".to_owned()); + let preset = match direction.to_ascii_lowercase().as_str() { + "up" => "scroll-up", + "down" => "scroll-down", + "left" => "scroll-left", + "right" => "scroll-right", + _ => anyhow::bail!("Unsupported Maestro swipe direction `{direction}`."), + }; + service_batch( + server_url, + udid, + vec![serde_json::json!({ "action": "gesture", "preset": preset })], + false, + )?; + Ok(serde_json::json!({ "direction": direction })) + } + "takeScreenshot" => { + let name = yaml_string_or_field(value, "path") + .or_else(|| yaml_string_or_field(value, "name")) + .unwrap_or_else(|| "screenshot".to_owned()); + let path = artifacts_dir.join(format!("{}.png", name.trim_end_matches(".png"))); + let png = service_get_bytes( + server_url, + &format!( + "/api/simulators/{}/screenshot.png", + url_path_component(udid) + ), + )?; + fs::write(&path, png)?; + Ok(serde_json::json!({ "path": path })) + } + "waitForAnimationToEnd" | "waitForAnimationToEnd:" => { + sleep_ms( + yaml_u64_or_field(value, "timeout") + .unwrap_or(1_000) + .min(10_000), + ); + Ok(Value::Null) + } + other => Err(anyhow::anyhow!( + "Unsupported Maestro command `{other}` in this compatibility runner." + )), + } +} + +fn maestro_command_name(command: &YamlValue) -> String { + if let Some(name) = command.as_str() { + return name.to_owned(); + } + command + .as_mapping() + .and_then(|mapping| mapping.keys().next()) + .and_then(YamlValue::as_str) + .unwrap_or("unknown") + .to_owned() +} + +fn maestro_bundle_id(value: &YamlValue) -> anyhow::Result { + yaml_string_or_field(value, "appId") + .or_else(|| yaml_string_or_field(value, "bundleId")) + .ok_or_else(|| anyhow::anyhow!("launchApp requires `appId` or `bundleId`.")) +} + +fn maestro_tap_body(value: &YamlValue) -> anyhow::Result { + if let Some(point) = yaml_field(value, "point") + .and_then(YamlValue::as_str) + .map(str::to_owned) + { + let (x, y) = parse_maestro_point(&point)?; + return Ok(serde_json::json!({ "x": x, "y": y, "normalized": true })); + } + Ok(serde_json::json!({ + "selector": maestro_selector(value)?, + "waitTimeoutMs": yaml_u64_or_field(value, "timeout").unwrap_or(5_000), + })) +} + +fn maestro_selector(value: &YamlValue) -> anyhow::Result { + if let Some(text) = value.as_str() { + return Ok(serde_json::json!({ "text": text, "regex": true })); + } + let Some(mapping) = value.as_mapping() else { + anyhow::bail!("Selector must be a string or mapping."); + }; + let text = yaml_string_field(mapping, "text"); + let explicit_regex = yaml_bool_field(mapping, "regex"); + let use_regex = explicit_regex.unwrap_or_else(|| text.is_some()); + let id = yaml_string_field(mapping, "id").map(|id| { + if use_regex && explicit_regex != Some(true) { + anchored_regex_literal(&id) + } else { + id + } + }); + Ok(serde_json::json!({ + "text": text, + "id": id, + "label": yaml_string_field(mapping, "label"), + "value": yaml_string_field(mapping, "value"), + "elementType": yaml_string_field(mapping, "type"), + "index": yaml_u64_field(mapping, "index"), + "enabled": yaml_bool_field(mapping, "enabled"), + "checked": yaml_bool_field(mapping, "checked"), + "focused": yaml_bool_field(mapping, "focused"), + "selected": yaml_bool_field(mapping, "selected"), + "regex": use_regex, + })) +} + +fn anchored_regex_literal(value: &str) -> String { + format!("^{}$", regex::escape(value)) +} + +fn service_wait_for( + server_url: &str, + udid: &str, + action: &str, + selector: Value, + timeout_ms: u64, +) -> anyhow::Result<()> { + service_post_ok( + server_url, + udid, + action, + &serde_json::json!({ "selector": selector, "timeoutMs": timeout_ms }), + ) +} + +fn parse_maestro_point(point: &str) -> anyhow::Result<(f64, f64)> { + let (x, y) = point + .split_once(',') + .ok_or_else(|| anyhow::anyhow!("point must be `x,y`."))?; + let parse = |value: &str| -> anyhow::Result { + let value = value.trim(); + if let Some(percent) = value.strip_suffix('%') { + Ok(percent.parse::()? / 100.0) + } else { + Ok(value.parse::()?) + } + }; + Ok((parse(x)?, parse(y)?)) +} + +fn capture_maestro_failure_screenshot( + server_url: &str, + udid: &str, + artifacts_dir: &Path, + step: usize, +) -> anyhow::Result { + let path = artifacts_dir.join(format!("failure-step-{step}.png")); + let png = service_get_bytes( + server_url, + &format!( + "/api/simulators/{}/screenshot.png", + url_path_component(udid) + ), + )?; + fs::write(&path, png)?; + Ok(path) +} + +fn yaml_field<'a>(value: &'a YamlValue, field: &str) -> Option<&'a YamlValue> { + value.as_mapping()?.get(YamlValue::String(field.to_owned())) +} + +fn yaml_string_or_field(value: &YamlValue, field: &str) -> Option { + value.as_str().map(str::to_owned).or_else(|| { + yaml_field(value, field) + .and_then(YamlValue::as_str) + .map(str::to_owned) + }) +} + +fn yaml_u64_or_field(value: &YamlValue, field: &str) -> Option { + value + .as_u64() + .or_else(|| yaml_field(value, field).and_then(YamlValue::as_u64)) +} + +fn yaml_string_field(mapping: &serde_yaml::Mapping, field: &str) -> Option { + mapping + .get(YamlValue::String(field.to_owned())) + .and_then(YamlValue::as_str) + .map(str::to_owned) +} + +fn yaml_u64_field(mapping: &serde_yaml::Mapping, field: &str) -> Option { + mapping + .get(YamlValue::String(field.to_owned())) + .and_then(YamlValue::as_u64) +} + +fn yaml_bool_field(mapping: &serde_yaml::Mapping, field: &str) -> Option { + mapping + .get(YamlValue::String(field.to_owned())) + .and_then(YamlValue::as_bool) +} + fn run_batch( bridge: &NativeBridge, udid: &str, @@ -5786,6 +6723,25 @@ fn batch_line_to_json_step(line: &str) -> anyhow::Result { "timeoutMs": args.value("timeout-ms").or_else(|| args.value("wait-timeout-ms")).and_then(|value| value.parse::().ok()).unwrap_or(5_000), "pollMs": args.value("poll-interval-ms").and_then(|value| value.parse::().ok()).unwrap_or(100), }), + "assert-not" | "assertNot" | "wait-for-not" | "waitForNot" => serde_json::json!({ + "action": "assertNot", + "selector": batch_selector_json(&args), + "source": args.value("source"), + "maxDepth": args.value("max-depth").and_then(|value| value.parse::().ok()), + "includeHidden": args.flag("include-hidden"), + "timeoutMs": args.value("timeout-ms").or_else(|| args.value("wait-timeout-ms")).and_then(|value| value.parse::().ok()).unwrap_or(5_000), + "pollMs": args.value("poll-interval-ms").and_then(|value| value.parse::().ok()).unwrap_or(100), + }), + "scroll-until-visible" | "scrollUntilVisible" => serde_json::json!({ + "action": "scrollUntilVisible", + "selector": batch_selector_json(&args), + "source": args.value("source"), + "maxDepth": args.value("max-depth").and_then(|value| value.parse::().ok()), + "includeHidden": args.flag("include-hidden"), + "timeoutMs": args.value("timeout-ms").or_else(|| args.value("wait-timeout-ms")).and_then(|value| value.parse::().ok()).unwrap_or(10_000), + "pollMs": args.value("poll-interval-ms").and_then(|value| value.parse::().ok()).unwrap_or(100), + "direction": args.value("direction").unwrap_or("down"), + }), "key" => serde_json::json!({ "action": "key", "keyCode": parse_hid_key(tokens.get(1).map(String::as_str).unwrap_or(""))?, @@ -6233,10 +7189,17 @@ fn required_f64(args: &StepOptions, key: &str) -> Result Value { serde_json::json!({ + "text": args.value("text"), "id": args.value("id"), "label": args.value("label"), "value": args.value("value"), "elementType": args.value("element-type"), + "index": args.value("index").and_then(|value| value.parse::().ok()), + "enabled": args.value("enabled").and_then(parse_bool_value), + "checked": args.value("checked").and_then(parse_bool_value), + "focused": args.value("focused").and_then(parse_bool_value), + "selected": args.value("selected").and_then(parse_bool_value), + "regex": args.flag("regex"), }) } @@ -6249,6 +7212,14 @@ fn batch_selector_from_args(args: &StepOptions) -> ElementSelector { } } +fn parse_bool_value(value: &str) -> Option { + match value.to_ascii_lowercase().as_str() { + "true" | "1" | "yes" | "on" => Some(true), + "false" | "0" | "no" | "off" => Some(false), + _ => None, + } +} + fn wait_for_batch_selector( bridge: &NativeBridge, udid: &str, @@ -6875,19 +7846,24 @@ fn default_client_root() -> anyhow::Result { #[cfg(test)] mod tests { use super::{ - batch_line_to_json_step, daemon_matches_launch_options, http_url_for_host, is_tailscale_ip, - normalize_accessibility_point_for_display, parse_workspace_daemon_process_line, - render_qr_code, server_health_watchdog_should_restart, service_post_error_is_retryable, - simdeck_pair_url, studio_daemon_restart_args, workspace_daemon_process_is_current, Cli, - Command, DaemonCommand, DaemonLaunchOptions, DaemonMetadata, PairingAddress, - ServiceCommand, StreamQualityProfileArg, StudioExposeOptions, VideoCodecMode, - WorkspaceDaemonProcess, DEFAULT_LOCAL_STREAM_QUALITY_PROFILE, - SERVER_HEALTH_WATCHDOG_FAILURE_THRESHOLD, SERVER_HEALTH_WATCHDOG_HTTP_FAILURE_THRESHOLD, + batch_line_to_json_step, daemon_matches_launch_options, http_url_for_host, + interactive_accessibility_snapshot, is_tailscale_ip, maestro_commands_from_flow, + maestro_selector, normalize_accessibility_point_for_display, parse_maestro_flow_yaml, + parse_maestro_point, parse_optional_udid_f64_args, parse_optional_udid_text_args, + parse_optional_udid_value_args, parse_tap_command_args, + parse_workspace_daemon_process_line, render_agent_accessibility_tree, render_qr_code, + run_maestro_command, server_health_watchdog_should_restart, + service_post_error_is_retryable, simdeck_pair_url, studio_daemon_restart_args, + workspace_daemon_process_is_current, Cli, Command, DaemonCommand, DaemonLaunchOptions, + DaemonMetadata, ElementSelector, PairingAddress, ServiceCommand, StreamQualityProfileArg, + StudioExposeOptions, TapCommandTarget, VideoCodecMode, WorkspaceDaemonProcess, YamlValue, + DEFAULT_LOCAL_STREAM_QUALITY_PROFILE, SERVER_HEALTH_WATCHDOG_FAILURE_THRESHOLD, + SERVER_HEALTH_WATCHDOG_HTTP_FAILURE_THRESHOLD, }; use clap::Parser; use std::collections::HashMap; use std::net::{IpAddr, Ipv4Addr}; - use std::path::PathBuf; + use std::path::{Path, PathBuf}; fn daemon_metadata_for_test( port: u16, @@ -7277,6 +8253,216 @@ mod tests { )); } + #[test] + fn describe_interactive_flag_prunes_agent_tree_but_keeps_context() { + let parsed = + Cli::try_parse_from(["simdeck", "describe", "sim-1", "--format", "agent", "-i"]) + .unwrap(); + let Command::DescribeUi { + interactive_only, .. + } = parsed.command + else { + panic!("expected describe command"); + }; + assert!(interactive_only); + + let snapshot = serde_json::json!({ + "source": "native-ax", + "roots": [{ + "type": "Window", + "children": [{ + "type": "View", + "AXLabel": "Static wrapper", + "children": [{ + "type": "Button", + "AXLabel": "Continue", + "enabled": true, + "children": [] + }, { + "type": "Label", + "AXLabel": "Read only", + "children": [] + }] + }] + }] + }); + + let pruned = interactive_accessibility_snapshot(&snapshot); + let output = render_agent_accessibility_tree(&pruned); + + assert!(output.contains("- Window")); + assert!(output.contains("- View: Static wrapper")); + assert!(output.contains("- Button: Continue")); + assert!(!output.contains("Read only")); + } + + #[test] + fn tap_single_positional_arg_is_label_shorthand() { + let parsed = Cli::try_parse_from(["simdeck", "tap", "Continue"]).unwrap(); + let Command::Tap { args, .. } = parsed.command else { + panic!("expected tap command"); + }; + let target = parse_tap_command_args(args, None, None, None, None).unwrap(); + + assert_eq!( + target, + TapCommandTarget { + udid: None, + x: None, + y: None, + selector: ElementSelector { + label: Some("Continue".to_owned()), + ..Default::default() + } + } + ); + } + + #[test] + fn tap_legacy_udid_coordinates_still_parse() { + let udid = "00000000-0000-0000-0000-000000000001"; + let target = parse_tap_command_args( + vec![udid.to_owned(), "120".to_owned(), "240".to_owned()], + None, + None, + None, + None, + ) + .unwrap(); + + assert_eq!(target.udid.as_deref(), Some(udid)); + assert_eq!(target.x, Some(120.0)); + assert_eq!(target.y, Some(240.0)); + assert!(target.selector.is_empty()); + } + + #[test] + fn tap_legacy_udid_label_shorthand_still_parse() { + let udid = "00000000-0000-0000-0000-000000000001"; + let target = parse_tap_command_args( + vec![udid.to_owned(), "Continue".to_owned()], + None, + None, + None, + None, + ) + .unwrap(); + + assert_eq!(target.udid.as_deref(), Some(udid)); + assert_eq!(target.selector.label.as_deref(), Some("Continue")); + } + + #[test] + fn global_device_flag_is_available_for_agent_shortcuts() { + let parsed = + Cli::try_parse_from(["simdeck", "--device", "iPhone 16", "tap", "Continue"]).unwrap(); + + assert_eq!(parsed.device.as_deref(), Some("iPhone 16")); + } + + #[test] + fn use_command_accepts_udid_selector() { + let parsed = + Cli::try_parse_from(["simdeck", "use", "00000000-0000-0000-0000-000000000001"]) + .unwrap(); + + let Command::Use { udid } = parsed.command else { + panic!("expected use command"); + }; + assert_eq!(udid, "00000000-0000-0000-0000-000000000001"); + } + + #[test] + fn device_commands_accept_omitted_udid() { + let parsed = Cli::try_parse_from(["simdeck", "boot"]).unwrap(); + let Command::Boot { udid } = parsed.command else { + panic!("expected boot command"); + }; + assert_eq!(udid, None); + + let parsed = Cli::try_parse_from(["simdeck", "home"]).unwrap(); + let Command::Home { udid } = parsed.command else { + panic!("expected home command"); + }; + assert_eq!(udid, None); + + let parsed = Cli::try_parse_from(["simdeck", "screenshot", "--stdout"]).unwrap(); + let Command::Screenshot { udid, stdout, .. } = parsed.command else { + panic!("expected screenshot command"); + }; + assert_eq!(udid, None); + assert!(stdout); + } + + #[test] + fn payload_commands_keep_legacy_udid_but_allow_default_device() { + let parsed = Cli::try_parse_from(["simdeck", "launch", "com.example.App"]).unwrap(); + let Command::Launch { args } = parsed.command else { + panic!("expected launch command"); + }; + let (udid, bundle_id) = + parse_optional_udid_value_args("launch", args, "BUNDLE_ID").unwrap(); + assert_eq!(udid, None); + assert_eq!(bundle_id, "com.example.App"); + + let parsed = + Cli::try_parse_from(["simdeck", "launch", "SIM-1", "com.example.App"]).unwrap(); + let Command::Launch { args } = parsed.command else { + panic!("expected launch command"); + }; + let (udid, bundle_id) = + parse_optional_udid_value_args("launch", args, "BUNDLE_ID").unwrap(); + assert_eq!(udid.as_deref(), Some("SIM-1")); + assert_eq!(bundle_id, "com.example.App"); + } + + #[test] + fn coordinate_commands_keep_legacy_udid_but_allow_default_device() { + let parsed = Cli::try_parse_from(["simdeck", "touch", "120", "240"]).unwrap(); + let Command::Touch { args, .. } = parsed.command else { + panic!("expected touch command"); + }; + let (udid, points) = parse_optional_udid_f64_args("touch", args, 2).unwrap(); + assert_eq!(udid, None); + assert_eq!(points, vec![120.0, 240.0]); + + let parsed = + Cli::try_parse_from(["simdeck", "swipe", "SIM-1", "10", "20", "30", "40"]).unwrap(); + let Command::Swipe { args, .. } = parsed.command else { + panic!("expected swipe command"); + }; + let (udid, points) = parse_optional_udid_f64_args("swipe", args, 4).unwrap(); + assert_eq!(udid.as_deref(), Some("SIM-1")); + assert_eq!(points, vec![10.0, 20.0, 30.0, 40.0]); + } + + #[test] + fn text_commands_use_positional_text_or_legacy_udid_with_input_flags() { + let parsed = Cli::try_parse_from(["simdeck", "type", "hello"]).unwrap(); + let Command::Type { + args, stdin, file, .. + } = parsed.command + else { + panic!("expected type command"); + }; + let (udid, text) = + parse_optional_udid_text_args("type", args, stdin || file.is_some()).unwrap(); + assert_eq!(udid, None); + assert_eq!(text.as_deref(), Some("hello")); + + let parsed = Cli::try_parse_from(["simdeck", "type", "SIM-1", "--stdin"]).unwrap(); + let Command::Type { + args, stdin, file, .. + } = parsed.command + else { + panic!("expected type command"); + }; + let (udid, text) = + parse_optional_udid_text_args("type", args, stdin || file.is_some()).unwrap(); + assert_eq!(udid.as_deref(), Some("SIM-1")); + assert_eq!(text, None); + } + #[test] fn batch_sleep_positional_duration_defaults_to_milliseconds() { let step = batch_line_to_json_step("sleep 500").unwrap(); @@ -7328,6 +8514,109 @@ mod tests { assert_eq!(step["timeoutMs"], 5000); } + #[test] + fn batch_assert_not_and_scroll_map_to_daemon_actions() { + let assert_not = batch_line_to_json_step("assert-not --text Loading --regex").unwrap(); + assert_eq!(assert_not["action"], "assertNot"); + assert_eq!(assert_not["selector"]["text"], "Loading"); + assert_eq!(assert_not["selector"]["regex"], true); + + let scroll = + batch_line_to_json_step("scroll-until-visible --text Settings --direction down") + .unwrap(); + assert_eq!(scroll["action"], "scrollUntilVisible"); + assert_eq!(scroll["selector"]["text"], "Settings"); + assert_eq!(scroll["direction"], "down"); + } + + #[test] + fn maestro_flow_accepts_config_with_commands() { + let yaml = parse_maestro_flow_yaml( + r#" +appId: com.example.App +--- +- launchApp +- tapOn: Continue +"#, + ) + .unwrap(); + let commands = maestro_commands_from_flow(&yaml).unwrap(); + assert_eq!(commands.len(), 2); + assert_eq!(commands[0]["launchApp"].as_str(), Some("com.example.App")); + } + + #[test] + fn maestro_selector_maps_text_and_state() { + let yaml: YamlValue = serde_yaml::from_str( + r#" +text: Continue.* +enabled: true +index: 1 +"#, + ) + .unwrap(); + let selector = maestro_selector(&yaml).unwrap(); + assert_eq!(selector["text"], "Continue.*"); + assert_eq!(selector["enabled"], true); + assert_eq!(selector["index"], 1); + assert_eq!(selector["regex"], true); + } + + #[test] + fn maestro_selector_keeps_id_literals_exact_by_default() { + let yaml: YamlValue = serde_yaml::from_str("id: login.button").unwrap(); + let selector = maestro_selector(&yaml).unwrap(); + + assert_eq!(selector["id"], "login.button"); + assert_eq!(selector["regex"], false); + } + + #[test] + fn maestro_selector_escapes_literal_ids_when_text_requires_regex() { + let yaml: YamlValue = serde_yaml::from_str( + r#" +text: Continue.* +id: login.button +"#, + ) + .unwrap(); + let selector = maestro_selector(&yaml).unwrap(); + + assert_eq!(selector["text"], "Continue.*"); + assert_eq!(selector["id"], "^login\\.button$"); + assert_eq!(selector["regex"], true); + } + + #[test] + fn maestro_swipe_rejects_unknown_directions() { + let command: YamlValue = serde_yaml::from_str( + r#" +swipe: + direction: rigth +"#, + ) + .unwrap(); + let error = + run_maestro_command("http://127.0.0.1:9", "test-udid", &command, Path::new(".")) + .unwrap_err(); + + assert!(error + .to_string() + .contains("Unsupported Maestro swipe direction `rigth`")); + } + + #[test] + fn maestro_percent_points_become_normalized_coordinates() { + assert_eq!(parse_maestro_point("50%,75%").unwrap(), (0.5, 0.75)); + } + + #[test] + fn maestro_tap_on_string_maps_to_text_selector() { + let yaml: YamlValue = serde_yaml::from_str("Continue").unwrap(); + let body = super::maestro_tap_body(&yaml).unwrap(); + assert_eq!(body["selector"]["text"], "Continue"); + } + #[test] fn server_health_watchdog_restarts_when_http_listener_is_unhealthy() { assert!(server_health_watchdog_should_restart( diff --git a/skills/simdeck/SKILL.md b/skills/simdeck/SKILL.md index 7b677138..984c3ee5 100644 --- a/skills/simdeck/SKILL.md +++ b/skills/simdeck/SKILL.md @@ -47,23 +47,28 @@ If Browser Use is not available, only then use `simdeck ui --open` - it would op ## Device And App -Device commands take `` immediately after the command. +Start by choosing a project default device. `simdeck use ` stores the +selection for the current workspace/CWD so later commands can omit the UDID. +Explicit UDIDs, `--device`, `SIMDECK_DEVICE`, and `SIMDECK_UDID` still work for +one-off overrides. Prefer short forms in agent loops, such as +`simdeck tap "Continue"` and `simdeck describe --format agent --max-depth 2`. ```bash simdeck list simdeck list --format json +simdeck use simdeck boot -simdeck shutdown -simdeck erase +simdeck shutdown +simdeck erase simdeck core-simulator restart -simdeck install /path/to/App.app -simdeck install /path/to/App.ipa +simdeck install /path/to/App.app +simdeck install /path/to/App.ipa simdeck install android: /path/to/app.apk -simdeck launch com.example.App -simdeck uninstall com.example.App -simdeck open-url myapp://route -simdeck open-url https://example.com -simdeck toggle-appearance +simdeck launch com.example.App +simdeck uninstall com.example.App +simdeck open-url myapp://route +simdeck open-url https://example.com +simdeck toggle-appearance ``` `simdeck list` defaults to compact JSON for token-efficient agent selection. @@ -80,32 +85,35 @@ AVDs from the Android SDK. Use targeted checks for test loops. `describe` is a diagnostic snapshot of the whole hierarchy. For verification, prefer the daemon APIs exposed by `simdeck/test`: `query`, `waitFor`, `assert`, selector `tap`, and `batch`. ```bash -simdeck describe -simdeck describe --format agent --max-depth 4 -simdeck describe --format compact-json -simdeck describe --point 120,240 -simdeck describe --source auto -simdeck describe --source nativescript -simdeck describe --source react-native -simdeck describe --source flutter -simdeck describe --source uikit -simdeck describe --source native-ax -simdeck describe --source android-uiautomator -simdeck describe --direct -simdeck wait-for --label "Welcome" --timeout-ms 5000 -simdeck assert --id login.button --source auto --max-depth 8 +simdeck describe +simdeck describe --format agent --max-depth 4 +simdeck describe --format agent --max-depth 4 --interactive +simdeck describe --format compact-json +simdeck describe --point 120,240 +simdeck describe --source auto +simdeck describe --source nativescript +simdeck describe --source react-native +simdeck describe --source flutter +simdeck describe --source uikit +simdeck describe --source native-ax +simdeck describe --source android-uiautomator +simdeck describe --direct +simdeck wait-for --label "Welcome" --timeout-ms 5000 +simdeck assert --id login.button --source auto --max-depth 8 ``` Use `--source auto` with the project daemon. Use `--direct` or `--source native-ax` for the private CoreSimulator accessibility bridge. Use `--source android-uiautomator` for Android emulator UIAutomator hierarchies. NativeScript, React Native, and Flutter inspector runtimes can add richer hierarchy data. For Android IDs, `describe` uses `uiautomator dump`; use `--format agent` or `--format compact-json` the same way as iOS. +Use `--interactive` or `-i` when an agent only needs controls and actionable framework nodes; SimDeck keeps ancestor context so the output is still navigable. Prefer selectors, coordinates only when needed. Selector taps go through the daemon and wait for the element server-side. ```bash -simdeck tap --id LoginButton --wait-timeout-ms 5000 -simdeck tap --label "Continue" --element-type Button -simdeck tap 120 240 +simdeck tap --id LoginButton --wait-timeout-ms 5000 +simdeck tap --label "Continue" --element-type Button +simdeck tap 120 240 +simdeck tap "Continue" ``` For persistent app integration tests, use `simdeck/test` instead of shelling out repeatedly: @@ -133,46 +141,46 @@ Use `tree()`/`describe` only when a test needs to print the whole UI for debuggi ## Interact ```bash -simdeck tap 120 240 -simdeck touch 0.5 0.5 --phase began --normalized -simdeck touch 0.5 0.5 --phase ended --normalized -simdeck touch 120 240 --down --up --delay-ms 800 -simdeck swipe 200 700 200 200 -simdeck swipe 200 700 200 200 --duration-ms 500 --pre-delay-ms 100 --post-delay-ms 250 -simdeck gesture scroll-up -simdeck gesture scroll-down -simdeck gesture swipe-from-left-edge -simdeck gesture swipe-from-right-edge -simdeck pinch --start-distance 160 --end-distance 80 -simdeck pinch --start-distance 0.20 --end-distance 0.35 --normalized --duration-ms 250 --steps 8 -simdeck rotate-gesture --radius 100 --degrees 90 -simdeck rotate-gesture --radius 0.12 --degrees 45 --normalized --duration-ms 250 --steps 8 -simdeck type 'hello' -simdeck type --stdin -simdeck type --file message.txt -simdeck key enter -simdeck key 42 --duration-ms 500 -simdeck key-sequence --keycodes h,e,l,l,o --delay-ms 75 -simdeck key-combo --modifiers cmd,shift --key z -simdeck dismiss-keyboard -simdeck button home -simdeck button lock --duration-ms 1000 -simdeck button side-button -simdeck button volume-up -simdeck button volume-down -simdeck button action --duration-ms 1000 -simdeck button mute -simdeck button digital-crown -simdeck crown --delta 50 -simdeck button left-side-button -simdeck button siri -simdeck button apple-pay -simdeck home -simdeck app-switcher -simdeck rotate-left -simdeck rotate-right -simdeck pasteboard set 'text' -simdeck pasteboard get +simdeck tap 120 240 +simdeck touch 0.5 0.5 --phase began --normalized +simdeck touch 0.5 0.5 --phase ended --normalized +simdeck touch 120 240 --down --up --delay-ms 800 +simdeck swipe 200 700 200 200 +simdeck swipe 200 700 200 200 --duration-ms 500 --pre-delay-ms 100 --post-delay-ms 250 +simdeck gesture scroll-up +simdeck gesture scroll-down +simdeck gesture swipe-from-left-edge +simdeck gesture swipe-from-right-edge +simdeck pinch --start-distance 160 --end-distance 80 +simdeck pinch --start-distance 0.20 --end-distance 0.35 --normalized --duration-ms 250 --steps 8 +simdeck rotate-gesture --radius 100 --degrees 90 +simdeck rotate-gesture --radius 0.12 --degrees 45 --normalized --duration-ms 250 --steps 8 +simdeck type 'hello' +simdeck type --stdin +simdeck type --file message.txt +simdeck key enter +simdeck key 42 --duration-ms 500 +simdeck key-sequence --keycodes h,e,l,l,o --delay-ms 75 +simdeck key-combo --modifiers cmd,shift --key z +simdeck dismiss-keyboard +simdeck button home +simdeck button lock --duration-ms 1000 +simdeck button side-button +simdeck button volume-up +simdeck button volume-down +simdeck button action --duration-ms 1000 +simdeck button mute +simdeck button digital-crown +simdeck crown --delta 50 +simdeck button left-side-button +simdeck button siri +simdeck button apple-pay +simdeck home +simdeck app-switcher +simdeck rotate-left +simdeck rotate-right +simdeck pasteboard set 'text' +simdeck pasteboard get ``` Use `--stdin` or `--file` for text with quotes, newlines, shell variables, or shell-sensitive characters. @@ -180,9 +188,9 @@ Use `--stdin` or `--file` for text with quotes, newlines, shell variables, or sh ## Timing, Batch ```bash -simdeck tap --label "Continue" --wait-timeout-ms 5000 -simdeck swipe 200 700 200 200 --pre-delay-ms 100 --post-delay-ms 250 -simdeck button lock --duration-ms 1000 +simdeck tap --label "Continue" --wait-timeout-ms 5000 +simdeck swipe 200 700 200 200 --pre-delay-ms 100 --post-delay-ms 250 +simdeck button lock --duration-ms 1000 ``` Prefer to use `wait-for` or `assert` in a batch to wait for UI state instead of fixed delays. `sleep 500` in a batch waits 500 ms. Use `sleep 0.5s` or `sleep --seconds 0.5` when you want to write seconds explicitly. @@ -190,19 +198,19 @@ Prefer to use `wait-for` or `assert` in a batch to wait for UI state instead of Use `batch` when steps are known; use discrete commands when a later step depends on parsing previous output. ```bash -simdeck batch \ +simdeck batch \ --step "tap --label Continue --wait-timeout-ms 5000" \ --step "type 'hello world'" \ --step "gesture scroll-down" \ --step "pinch --start-distance 0.20 --end-distance 0.35 --normalized" ``` -Batch rules: one source (`--step`, `--file`, or `--stdin`); keep `` at batch level; ordered steps; fail-fast by default; `--continue-on-error` for best effort. Step commands: `tap`, `wait-for`, `assert`, `swipe`, `gesture`, `pinch`, `rotate-gesture`, `touch`, `type`, `button`, `key`, `key-sequence`, `key-combo`, `sleep`. +Batch rules: one source (`--step`, `--file`, or `--stdin`); set the default with `simdeck use ` or keep `` at batch level; ordered steps; fail-fast by default; `--continue-on-error` for best effort. Step commands: `tap`, `wait-for`, `assert`, `swipe`, `gesture`, `pinch`, `rotate-gesture`, `touch`, `type`, `button`, `key`, `key-sequence`, `key-combo`, `sleep`. For JS tests, batch can combine action and verification without extra CLI process startup: ```ts -await simdeck.batch(udid, [ +await simdeck.batch([ { action: "tap", selector: { label: "Continue" }, waitTimeoutMs: 5000 }, { action: "waitFor", @@ -213,27 +221,33 @@ await simdeck.batch(udid, [ ]); ``` +For app-style flows, SimDeck can run a practical subset of Maestro YAML: + +```bash +simdeck maestro test flow.yaml --artifacts-dir artifacts/maestro +``` + ## Evidence ```bash -simdeck screenshot --output screen.png -simdeck screenshot --with-bezel --output screen-bezel.png -simdeck screenshot --stdout > screen.png -simdeck record --seconds 5 --output screen-recording.mp4 -simdeck record --seconds 5 --stdout > screen-recording.mp4 -simdeck logs --seconds 30 --limit 200 -simdeck chrome-profile -simdeck processes -simdeck stats -simdeck stats --watch -simdeck sample --seconds 3 +simdeck screenshot --output screen.png +simdeck screenshot --with-bezel --output screen-bezel.png +simdeck screenshot --stdout > screen.png +simdeck record --seconds 5 --output screen-recording.mp4 +simdeck record --seconds 5 --stdout > screen-recording.mp4 +simdeck logs --seconds 30 --limit 200 +simdeck chrome-profile +simdeck processes +simdeck stats +simdeck stats --watch +simdeck sample --seconds 3 ``` Use screenshots for still evidence, `--with-bezel` when the device frame matters, and `record` for short MP4 screen recordings. Use `stats` for simulator app CPU, memory, disk write, network receive/send rates, connections, hang, and crash/termination signals. Use `sample` only when a short CPU stack capture is worth the extra pause. Prefer describe for token-efficient state dumps, if they have enough context. ## Default Loop -1. Start UI, list, boot/select ``, open viewer if in-app browser available +1. Start UI, list, `simdeck use `, boot/select the device, open viewer if in-app browser available 2. Build with project tools; install and launch with SimDeck. 3. Use one `describe --format agent --max-depth 4` to understand an unfamiliar screen. 4. Interact with selectors first; use coordinates only when needed.