diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 8ccadea..4e86d43 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -130,6 +130,15 @@ "./skills/flutter-read-logs-usage", "./skills/read-logs" ] + }, + { + "name": "lean-a11y-vo-log", + "source": "./plugins/lean-a11y-vo-log", + "description": "Capture a timestamped transcript of everything VoiceOver speaks (and each keyboard nav gesture) during a manual screen-reader session on macOS, then diagnose it into an accessibility bug report via /lean-a11y-vo-log.", + "skills": [ + "./skills/lean-a11y-vo-log-usage", + "./skills/lean-a11y-vo-log" + ] } ] } diff --git a/README.md b/README.md index 9733f55..0e60226 100644 --- a/README.md +++ b/README.md @@ -55,6 +55,7 @@ Most plugins are pure rules and skills with no setup. A few need one-time toolin - [`flutter-patrol`](plugins/flutter-patrol/) - Patrol CLI and Patrol MCP - [`flutter-marionette`](plugins/flutter-marionette/) - Marionette MCP and app-side binding +- [`lean-a11y-vo-log`](plugins/lean-a11y-vo-log/) - VoiceOver AppleScript control, macOS Automation permission, and (optional) Karabiner gesture rules ## Available plugins @@ -83,6 +84,7 @@ Most plugins are pure rules and skills with no setup. A few need one-time toolin - [`flutter-patrol`](plugins/flutter-patrol/) - Patrol test architecture, key conventions, and Patrol MCP workflow for AI-assisted E2E work - [`flutter-marionette`](plugins/flutter-marionette/) - runtime interaction with a live debug app through Marionette MCP for exploration, smoke checks, and UI debugging - [`flutter-read-logs`](plugins/flutter-read-logs/) - read the running app's latest `flutter run` logs as on-demand context via `/read-logs` +- [`lean-a11y-vo-log`](plugins/lean-a11y-vo-log/) - capture a text transcript of everything VoiceOver speaks (and each keyboard nav gesture) during a manual screen-reader session on macOS, then diagnose it via `/lean-a11y-vo-log` ### Every plugin has a `-usage` skill diff --git a/plugins/lean-a11y-vo-log/.claude-plugin/plugin.json b/plugins/lean-a11y-vo-log/.claude-plugin/plugin.json new file mode 100644 index 0000000..353e3dd --- /dev/null +++ b/plugins/lean-a11y-vo-log/.claude-plugin/plugin.json @@ -0,0 +1,20 @@ +{ + "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json", + "name": "lean-a11y-vo-log", + "displayName": "A11y VoiceOver Log", + "description": "Capture a timestamped transcript of everything VoiceOver speaks — and each keyboard nav gesture — during a manual screen-reader session on macOS, then diagnose it into an accessibility bug report. A text alternative to screen recording.", + "version": "0.1.0", + "author": { + "name": "LeanCode" + }, + "homepage": "https://github.com/leancodepl/ai-plugins", + "repository": "https://github.com/leancodepl/ai-plugins", + "keywords": [ + "accessibility", + "a11y", + "voiceover", + "screen-reader", + "macos" + ], + "skills": "./skills/" +} diff --git a/plugins/lean-a11y-vo-log/CHANGELOG.md b/plugins/lean-a11y-vo-log/CHANGELOG.md new file mode 100644 index 0000000..db04506 --- /dev/null +++ b/plugins/lean-a11y-vo-log/CHANGELOG.md @@ -0,0 +1,7 @@ +# Changelog + +## 0.1.0 + +- Initial `lean-a11y-vo-log` plugin. +- `skills/lean-a11y-vo-log/SKILL.md` — spawn a Terminal logger that captures a timestamped transcript of everything VoiceOver speaks (and each keyboard nav gesture, via Karabiner) during a manual screen-reader session, then diagnose the announcement timeline into an accessibility bug report, via `/lean-a11y-vo-log`. `vo_log.sh` auto-starts VoiceOver if it's off; `start_log_terminal.sh` reuses the launch window instead of opening a second Terminal. +- `skills/lean-a11y-vo-log-usage/SKILL.md` — explain what the plugin does, when to reach for it, and route to the README setup. diff --git a/plugins/lean-a11y-vo-log/README.md b/plugins/lean-a11y-vo-log/README.md new file mode 100644 index 0000000..62b0255 --- /dev/null +++ b/plugins/lean-a11y-vo-log/README.md @@ -0,0 +1,108 @@ +# lean-a11y-vo-log + +Capture a timestamped **text transcript of everything VoiceOver speaks** during a +manual screen-reader session on macOS — and, optionally, each keyboard nav gesture +— then have Claude diagnose it into an accessibility bug report. A text alternative +to recording and re-watching video. + +The developer drives VoiceOver by hand; Claude opens the logging Terminal, reads the +resulting file, and turns the announcement timeline into `phrase → widget → +file:line → fix` chains. Project-agnostic: it logs whatever macOS app is frontmost +(web, native, Flutter). + +## Included assets + +- `skills/lean-a11y-vo-log/SKILL.md` — the workflow (spawn logger → dev runs flow → Claude + diagnoses) plus the announcement-diagnosis table +- `skills/lean-a11y-vo-log-usage/SKILL.md` — routing skill: what the plugin does and when to + reach for it +- `skills/lean-a11y-vo-log/scripts/vo_log.sh` — the poller: reads VoiceOver's "last phrase" + ~6×/s (deduped) and appends `VO` lines +- `skills/lean-a11y-vo-log/scripts/start_log_terminal.sh` — opens the dedicated logging + Terminal window running `vo_log.sh` +- `skills/lean-a11y-vo-log/scripts/vo_gesture.sh` — appends a `GESTURE` line; called by + Karabiner on each VoiceOver nav combo +- `skills/lean-a11y-vo-log/assets/karabiner_vo_gesture_logger.json` — Karabiner rules + template (the `__VO_GESTURE_SH__` placeholder is stamped during setup) + +## How it works + +`vo_log.sh` polls VoiceOver over AppleScript and writes one line per spoken phrase. +Keyboard gestures cannot be captured by a self-contained tap (VoiceOver disables a +`CGEventTap` after the first event), so gesture lines come from **Karabiner-Elements**: +each VoiceOver nav combo (VO+Right/Left/Up/Down, VO+Space, VO+Shift+Down…) runs +`vo_gesture.sh`, which appends a `GESTURE` line to the same log — including for +**silent focus moves** (a gesture with no `VO` line after it = focus moved but +nothing was spoken, the #1 screen-reader smell). + +``` +HH:MM:SS | VO | +HH:MM:SS | GESTURE | vo-right (auto, from Karabiner) +HH:MM:SS | ACTION | +``` + +## Setup (one-time, per machine) + +### 1. Let VoiceOver be driven by AppleScript + +VoiceOver Utility (**VO+F8**) → General → ☑ *"Allow VoiceOver to be controlled with +AppleScript"*. + +### 2. Grant Automation permission + +Whatever app is running Claude Code — Terminal, iTerm, VS Code, … — needs permission +to control **VoiceOver** and **Terminal.app** (the logger always opens its window in +Terminal.app, regardless of where Claude runs). The first run may prompt *" +wants to control Terminal / VoiceOver"* → **Allow**. If it errors instead, add the +entries under System Settings → Privacy & Security → Automation. + +That is enough for **speech logging**. `/lean-a11y-vo-log` now works — you just won't get +`GESTURE` lines until step 3. + +### 3. Keyboard-gesture logging (Karabiner-Elements) + +This is what adds the `GESTURE` lines (and lets Claude spot silent focus moves). +Install [Karabiner-Elements](https://karabiner-elements.pqrs.org/) first — that +install is the only manual step. The rest below (copy the logger to a stable spot, +stamp the rules) is plain shell, so **if you trust Claude, just ask it to do this +setup for you** — paste this section or say *"do the lean-a11y-vo-log Karabiner setup"* +and it runs the commands. It only writes to `~/.local/bin/` and +`~/.config/karabiner/`; you still enable the rule yourself in the Karabiner UI at +the end. Prefer to do it by hand? Wire the bundled rules to a **stable** copy of the +gesture logger — stable so plugin updates (which land under a versioned path) don't +break Karabiner: + +```bash +# Resolve the installed plugin's files (latest version wins) +GEST_SRC="$(find ~/.claude/plugins/cache -path '*/lean-a11y-vo-log/skills/lean-a11y-vo-log/scripts/vo_gesture.sh' | sort | tail -1)" +RULES_SRC="$(find ~/.claude/plugins/cache -path '*/lean-a11y-vo-log/skills/lean-a11y-vo-log/assets/karabiner_vo_gesture_logger.json' | sort | tail -1)" + +# a) Copy the gesture logger to a stable location that survives plugin updates +mkdir -p ~/.local/bin +cp "$GEST_SRC" ~/.local/bin/vo_gesture.sh +chmod +x ~/.local/bin/vo_gesture.sh + +# b) Stamp that absolute path into the rules and drop them where Karabiner imports from +mkdir -p ~/.config/karabiner/assets/complex_modifications +sed "s#__VO_GESTURE_SH__#$HOME/.local/bin/vo_gesture.sh#g" "$RULES_SRC" \ + > ~/.config/karabiner/assets/complex_modifications/vo_gesture_logger.json +``` + +Then in **Karabiner-Elements → Settings → Complex Modifications → Add rule**, enable +the *"VoiceOver gesture logger"* rules. + +> Re-run step 3 only if you later change `vo_gesture.sh` itself; a normal plugin +> version bump does not require it, because Karabiner points at the stable copy. + +## Usage + +``` +/lean-a11y-vo-log +``` + +Claude opens the logging Terminal (starting VoiceOver for you if it's off), you run +the buggy flow, then say **"read the log"** and Claude writes up the bugs. Type a +short note + Enter in the log window to record a visual observation as an `ACTION` +line. To turn VoiceOver off again when done, press **Cmd+F5** — the logger stays +quiet while it's off. (Set `VO_NO_AUTOSTART=1` if you'd rather enable VoiceOver +yourself before starting.) diff --git a/plugins/lean-a11y-vo-log/skills/lean-a11y-vo-log-usage/SKILL.md b/plugins/lean-a11y-vo-log/skills/lean-a11y-vo-log-usage/SKILL.md new file mode 100644 index 0000000..bb0b06a --- /dev/null +++ b/plugins/lean-a11y-vo-log/skills/lean-a11y-vo-log-usage/SKILL.md @@ -0,0 +1,51 @@ +--- +name: lean-a11y-vo-log-usage +description: Explain what the `lean-a11y-vo-log` plugin does and how to use it. Use when the user invokes `/lean-a11y-vo-log-usage`, asks what this plugin covers, or needs help with VoiceOver speech logging, gesture capture, or the one-time setup. +--- + +# lean-a11y-vo-log Usage + +## How to respond + +- If the user invoked this skill without a concrete task, start by explaining what + the plugin is for and when it beats a screen recording. +- Point to the one-time setup in the plugin `README.md` (VoiceOver AppleScript + control, Automation permission, optional Karabiner gesture rules). +- If the user already has a concrete logging task, hand off to the `lean-a11y-vo-log` + skill and do the work. +- Do not reply with filler like "skill loaded" or "ready for the task" before + explaining the plugin. + +## What this plugin does + +- Captures a **timestamped text transcript** of everything VoiceOver speaks during + a manual screen-reader session on macOS — a text alternative to recording video. +- Optionally interleaves **keyboard nav gestures** (via Karabiner-Elements), so the + transcript shows exactly what the dev did between announcements — including silent + focus moves (a gesture with no VO line after it), the #1 screen-reader smell. +- Then **diagnoses** the announcement timeline into an accessibility bug report, + chaining each bad announcement to the likely widget, file, and fix. +- Is **project-agnostic**: it logs whatever macOS app is frontmost (web, native, + Flutter). + +## When to reach for it + +The *dev* drives VoiceOver by hand and Claude reads the resulting transcript. Best +for reproducing a bug the dev already feels, or capturing a real human navigation +flow — without recording and re-watching a video. + +## Setup + +One-time, per machine — see the plugin `README.md`: + +1. VoiceOver → allow AppleScript control. +2. Grant Automation permission to the app running Claude Code (whatever terminal + that is) so it can control VoiceOver + Terminal. +3. **Dependency for `GESTURE` lines:** install Karabiner-Elements and import the + bundled rules (pointed at a stable copy of `vo_gesture.sh`). Speech logging works + without it; keyboard-gesture logging does not. + +## Example usage + +- `/lean-a11y-vo-log` — Claude opens the logging Terminal, you run your flow, then say + "read the log" and Claude writes up the bugs. diff --git a/plugins/lean-a11y-vo-log/skills/lean-a11y-vo-log/SKILL.md b/plugins/lean-a11y-vo-log/skills/lean-a11y-vo-log/SKILL.md new file mode 100644 index 0000000..61a481e --- /dev/null +++ b/plugins/lean-a11y-vo-log/skills/lean-a11y-vo-log/SKILL.md @@ -0,0 +1,91 @@ +--- +name: lean-a11y-vo-log +description: Capture a text transcript of everything VoiceOver speaks (and each keyboard nav gesture) during a manual screen-reader session on macOS, then diagnose it into an accessibility bug report — a text alternative to screen recording. Claude spawns a dedicated Terminal running the logger, the dev enables VoiceOver and does the flow, then Claude reads the log and turns the announcement timeline into a bug report. Project-agnostic (any macOS app: web, native, Flutter). Trigger on "/lean-a11y-vo-log", "log VoiceOver", "capture VO speech", "record what the screen reader says". +--- + +# lean-a11y-vo-log — VoiceOver speech logging (developer-in-the-loop) + +Turns a manual VoiceOver session into a timestamped transcript, then diagnoses it. +The dev drives VoiceOver by hand; every spoken phrase (and every nav gesture, if +Karabiner logging is set up) lands in one file; Claude reads it and writes up the +bugs. Project-agnostic — it captures whatever app is frontmost. + +**One-time machine setup lives in this plugin's `README.md`, not here** (VoiceOver +AppleScript control, Automation permission, and the Karabiner gesture rules). The +logger script prechecks that setup and prints what is missing, so you rarely need +to explain it — just point the dev at the README's *Setup* section if a check fails. + +## Workflow + +### 1. Start the logger (Claude does this) + +```bash +bash "${CLAUDE_PLUGIN_ROOT}/skills/lean-a11y-vo-log/scripts/start_log_terminal.sh" ~/vo_log.txt +``` + +The script **starts VoiceOver itself** if it's off, then opens a new Terminal window +showing the live transcript. Do **not** activate/steal focus to that window — that +moves the VoiceOver cursor off the app under test. If the script reports it couldn't +start VoiceOver or can't control it over AppleScript, point the dev at the README +*Setup* section (Cmd+F5 to toggle VO by hand; AppleScript-control + Automation +permissions). + +### 2. Dev runs the session (human) + +Tell the developer: +> Logger's live in the new window. Switch to the app, do the flow that's buggy. +> Gestures auto-log if you set up the Karabiner rules; otherwise type a short note +> + Enter in that window when you do something notable (e.g. "opened market +> dropdown"). When you're done, come back and say "read the log". + +### 3. Read & diagnose (Claude does this) + +Extract just this session (from the last `=== requested … ===` marker) and build +the timeline: + +```bash +awk '/=== requested/{buf=""} {buf=buf $0 "\n"} END{printf "%s", buf}' ~/vo_log.txt +``` + +Then analyze the announcement sequence with the table below. For each bad +announcement, state the chain *phrase → focused widget → file:line → property → +fix*. After a fix, re-run this logger to confirm the announcement changed. + +## Diagnosing the transcript + +| Phrase / symptom | Means | Likely cause & fix | +|------------------|-------|--------------------| +| "…, group" + "press Control-Option-Shift-Down" | interactable group, not a transparent region — user must drill in | `Semantics` with `container: true` **and** a `label` wrapping interactive children. Drop them; a landmark `role` alone is transparent | +| "end of, " in the wrong place | hard group edge around a landmark | remove `container: true` | +| GESTURE with no VO line (silent stop) | unlabeled node / covered pane | add `Semantics(label:)`, or `ExcludeSemantics` if decorative; if a whole pane, check an `ExcludeSemantics(excluding:)` gating it | +| container announced with a child's name | name-steal — container unnamed | name the route/region (`namesRoute`) or remove the merging label | +| element read twice per swipe | duplicate focus stop | collapse nested focusables to one | +| title read as plain text | missing heading role | `Semantics(header: true)` / heading-level widget | +| visible label ≠ spoken name | label overrides visible text (2.5.3) | align `Semantics(label:)` with the visible text | + +Useful greps (Flutter): `grep -rn "container: true" lib --include="*.dart"` (the +usual landmark culprit) and `grep -rn "Semantics(" lib/`. + +## Reading signals + +- **GESTURE with no following VO line** → silent focus move. Focus went somewhere + unlabeled/unannounced. Prime suspect for "I don't know where I am". +- **Same phrase repeated** → focus bounced back, or a live-region is chattering. +- **A long stream after one action** → over-verbose node (e.g. a dialog reading its + whole subtree). Note it, but it's often acceptable. + +## Log format + +``` +HH:MM:SS | VO | +HH:MM:SS | GESTURE | vo-right (auto, from Karabiner) +HH:MM:SS | ACTION | +``` + +## Video fallback + +For TalkBack (no macOS scripting) or inherently visual bugs (focus-ring position), +record with the caption panel on (VO+F10 / TalkBack "Display speech output"), then +slice and read it frame by frame: +`ffmpeg -v error -i