a11y-vo-log: VoiceOver speech-logging plugin

.claude-plugin/marketplace.json

@@ -130,6 +130,15 @@
         "./skills/flutter-read-logs-usage",
         "./skills/read-logs"
       ]
+    },
+    {
+      "name": "lean-a11y-vo-log",
+      "source": "./plugins/lean-a11y-vo-log",
+      "description": "Capture a timestamped transcript of everything VoiceOver speaks (and each keyboard nav gesture) during a manual screen-reader session on macOS, then diagnose it into an accessibility bug report via /lean-a11y-vo-log.",
+      "skills": [
+        "./skills/lean-a11y-vo-log-usage",
+        "./skills/lean-a11y-vo-log"
+      ]
     }
   ]
 }

README.md

@@ -55,6 +55,7 @@ Most plugins are pure rules and skills with no setup. A few need one-time toolin
 
 - [`flutter-patrol`](plugins/flutter-patrol/) - Patrol CLI and Patrol MCP
 - [`flutter-marionette`](plugins/flutter-marionette/) - Marionette MCP and app-side binding
+- [`lean-a11y-vo-log`](plugins/lean-a11y-vo-log/) - VoiceOver AppleScript control, macOS Automation permission, and (optional) Karabiner gesture rules
 
 ## Available plugins
 
@@ -83,6 +84,7 @@ Most plugins are pure rules and skills with no setup. A few need one-time toolin
 - [`flutter-patrol`](plugins/flutter-patrol/) - Patrol test architecture, key conventions, and Patrol MCP workflow for AI-assisted E2E work
 - [`flutter-marionette`](plugins/flutter-marionette/) - runtime interaction with a live debug app through Marionette MCP for exploration, smoke checks, and UI debugging
 - [`flutter-read-logs`](plugins/flutter-read-logs/) - read the running app's latest `flutter run` logs as on-demand context via `/read-logs`
+- [`lean-a11y-vo-log`](plugins/lean-a11y-vo-log/) - capture a text transcript of everything VoiceOver speaks (and each keyboard nav gesture) during a manual screen-reader session on macOS, then diagnose it via `/lean-a11y-vo-log`
 
 ### Every plugin has a `-usage` skill
 

plugins/lean-a11y-vo-log/.claude-plugin/plugin.json

@@ -0,0 +1,20 @@
+{
+  "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json",
+  "name": "lean-a11y-vo-log",
+  "displayName": "A11y VoiceOver Log",
+  "description": "Capture a timestamped transcript of everything VoiceOver speaks — and each keyboard nav gesture — during a manual screen-reader session on macOS, then diagnose it into an accessibility bug report. A text alternative to screen recording.",
+  "version": "0.1.0",
+  "author": {
+    "name": "LeanCode"
+  },
+  "homepage": "https://github.com/leancodepl/ai-plugins",
+  "repository": "https://github.com/leancodepl/ai-plugins",
+  "keywords": [
+    "accessibility",
+    "a11y",
+    "voiceover",
+    "screen-reader",
+    "macos"
+  ],
+  "skills": "./skills/"
+}

plugins/lean-a11y-vo-log/CHANGELOG.md

@@ -0,0 +1,7 @@
+# Changelog
+
+## 0.1.0
+
+- Initial `lean-a11y-vo-log` plugin.
+- `skills/lean-a11y-vo-log/SKILL.md` — spawn a Terminal logger that captures a timestamped transcript of everything VoiceOver speaks (and each keyboard nav gesture, via Karabiner) during a manual screen-reader session, then diagnose the announcement timeline into an accessibility bug report, via `/lean-a11y-vo-log`. `vo_log.sh` auto-starts VoiceOver if it's off; `start_log_terminal.sh` reuses the launch window instead of opening a second Terminal.
+- `skills/lean-a11y-vo-log-usage/SKILL.md` — explain what the plugin does, when to reach for it, and route to the README setup.

plugins/lean-a11y-vo-log/README.md

@@ -0,0 +1,108 @@
+# lean-a11y-vo-log
+
+Capture a timestamped **text transcript of everything VoiceOver speaks** during a
+manual screen-reader session on macOS — and, optionally, each keyboard nav gesture
+— then have Claude diagnose it into an accessibility bug report. A text alternative
+to recording and re-watching video.
+
+The developer drives VoiceOver by hand; Claude opens the logging Terminal, reads the
+resulting file, and turns the announcement timeline into `phrase → widget →
+file:line → fix` chains. Project-agnostic: it logs whatever macOS app is frontmost
+(web, native, Flutter).
+
+## Included assets
+
+- `skills/lean-a11y-vo-log/SKILL.md` — the workflow (spawn logger → dev runs flow → Claude
+  diagnoses) plus the announcement-diagnosis table
+- `skills/lean-a11y-vo-log-usage/SKILL.md` — routing skill: what the plugin does and when to
+  reach for it
+- `skills/lean-a11y-vo-log/scripts/vo_log.sh` — the poller: reads VoiceOver's "last phrase"
+  ~6×/s (deduped) and appends `VO` lines
+- `skills/lean-a11y-vo-log/scripts/start_log_terminal.sh` — opens the dedicated logging
+  Terminal window running `vo_log.sh`
+- `skills/lean-a11y-vo-log/scripts/vo_gesture.sh` — appends a `GESTURE` line; called by
+  Karabiner on each VoiceOver nav combo
+- `skills/lean-a11y-vo-log/assets/karabiner_vo_gesture_logger.json` — Karabiner rules
+  template (the `__VO_GESTURE_SH__` placeholder is stamped during setup)
+
+## How it works
+
+`vo_log.sh` polls VoiceOver over AppleScript and writes one line per spoken phrase.
+Keyboard gestures cannot be captured by a self-contained tap (VoiceOver disables a
+`CGEventTap` after the first event), so gesture lines come from **Karabiner-Elements**:
+each VoiceOver nav combo (VO+Right/Left/Up/Down, VO+Space, VO+Shift+Down…) runs
+`vo_gesture.sh`, which appends a `GESTURE` line to the same log — including for
+**silent focus moves** (a gesture with no `VO` line after it = focus moved but
+nothing was spoken, the #1 screen-reader smell).
+
+```
+HH:MM:SS | VO      | <phrase VoiceOver spoke>
+HH:MM:SS | GESTURE | vo-right          (auto, from Karabiner)
+HH:MM:SS | ACTION  | <freeform note the dev typed in the log window>
+```
+
+## Setup (one-time, per machine)
+
+### 1. Let VoiceOver be driven by AppleScript
+
+VoiceOver Utility (**VO+F8**) → General → ☑ *"Allow VoiceOver to be controlled with
+AppleScript"*.
+
+### 2. Grant Automation permission
+
+Whatever app is running Claude Code — Terminal, iTerm, VS Code, … — needs permission
+to control **VoiceOver** and **Terminal.app** (the logger always opens its window in
+Terminal.app, regardless of where Claude runs). The first run may prompt *"<that app>
+wants to control Terminal / VoiceOver"* → **Allow**. If it errors instead, add the
+entries under System Settings → Privacy & Security → Automation.
+
+That is enough for **speech logging**. `/lean-a11y-vo-log` now works — you just won't get
+`GESTURE` lines until step 3.
+
+### 3. Keyboard-gesture logging (Karabiner-Elements)
+
+This is what adds the `GESTURE` lines (and lets Claude spot silent focus moves).
+Install [Karabiner-Elements](https://karabiner-elements.pqrs.org/) first — that
+install is the only manual step. The rest below (copy the logger to a stable spot,
+stamp the rules) is plain shell, so **if you trust Claude, just ask it to do this
+setup for you** — paste this section or say *"do the lean-a11y-vo-log Karabiner setup"*
+and it runs the commands. It only writes to `~/.local/bin/` and
+`~/.config/karabiner/`; you still enable the rule yourself in the Karabiner UI at
+the end. Prefer to do it by hand? Wire the bundled rules to a **stable** copy of the
+gesture logger — stable so plugin updates (which land under a versioned path) don't
+break Karabiner:
+
+```bash
+# Resolve the installed plugin's files (latest version wins)
+GEST_SRC="$(find ~/.claude/plugins/cache -path '*/lean-a11y-vo-log/skills/lean-a11y-vo-log/scripts/vo_gesture.sh' | sort | tail -1)"
+RULES_SRC="$(find ~/.claude/plugins/cache -path '*/lean-a11y-vo-log/skills/lean-a11y-vo-log/assets/karabiner_vo_gesture_logger.json' | sort | tail -1)"
+
+# a) Copy the gesture logger to a stable location that survives plugin updates
+mkdir -p ~/.local/bin
+cp "$GEST_SRC" ~/.local/bin/vo_gesture.sh
+chmod +x ~/.local/bin/vo_gesture.sh
+
+# b) Stamp that absolute path into the rules and drop them where Karabiner imports from
+mkdir -p ~/.config/karabiner/assets/complex_modifications
+sed "s#__VO_GESTURE_SH__#$HOME/.local/bin/vo_gesture.sh#g" "$RULES_SRC" \
+  > ~/.config/karabiner/assets/complex_modifications/vo_gesture_logger.json
+```
+
+Then in **Karabiner-Elements → Settings → Complex Modifications → Add rule**, enable
+the *"VoiceOver gesture logger"* rules.
+
+> Re-run step 3 only if you later change `vo_gesture.sh` itself; a normal plugin
+> version bump does not require it, because Karabiner points at the stable copy.
+
+## Usage
+
+```
+/lean-a11y-vo-log
+```
+
+Claude opens the logging Terminal (starting VoiceOver for you if it's off), you run
+the buggy flow, then say **"read the log"** and Claude writes up the bugs. Type a
+short note + Enter in the log window to record a visual observation as an `ACTION`
+line. To turn VoiceOver off again when done, press **Cmd+F5** — the logger stays
+quiet while it's off. (Set `VO_NO_AUTOSTART=1` if you'd rather enable VoiceOver
+yourself before starting.)

plugins/lean-a11y-vo-log/skills/lean-a11y-vo-log-usage/SKILL.md

@@ -0,0 +1,51 @@
+---
+name: lean-a11y-vo-log-usage
+description: Explain what the `lean-a11y-vo-log` plugin does and how to use it. Use when the user invokes `/lean-a11y-vo-log-usage`, asks what this plugin covers, or needs help with VoiceOver speech logging, gesture capture, or the one-time setup.
+---
+
+# lean-a11y-vo-log Usage
+
+## How to respond
+
+- If the user invoked this skill without a concrete task, start by explaining what
+  the plugin is for and when it beats a screen recording.
+- Point to the one-time setup in the plugin `README.md` (VoiceOver AppleScript
+  control, Automation permission, optional Karabiner gesture rules).
+- If the user already has a concrete logging task, hand off to the `lean-a11y-vo-log`
+  skill and do the work.
+- Do not reply with filler like "skill loaded" or "ready for the task" before
+  explaining the plugin.
+
+## What this plugin does
+
+- Captures a **timestamped text transcript** of everything VoiceOver speaks during
+  a manual screen-reader session on macOS — a text alternative to recording video.
+- Optionally interleaves **keyboard nav gestures** (via Karabiner-Elements), so the
+  transcript shows exactly what the dev did between announcements — including silent
+  focus moves (a gesture with no VO line after it), the #1 screen-reader smell.
+- Then **diagnoses** the announcement timeline into an accessibility bug report,
+  chaining each bad announcement to the likely widget, file, and fix.
+- Is **project-agnostic**: it logs whatever macOS app is frontmost (web, native,
+  Flutter).
+
+## When to reach for it
+
+The *dev* drives VoiceOver by hand and Claude reads the resulting transcript. Best
+for reproducing a bug the dev already feels, or capturing a real human navigation
+flow — without recording and re-watching a video.
+
+## Setup
+
+One-time, per machine — see the plugin `README.md`:
+
+1. VoiceOver → allow AppleScript control.
+2. Grant Automation permission to the app running Claude Code (whatever terminal
+   that is) so it can control VoiceOver + Terminal.
+3. **Dependency for `GESTURE` lines:** install Karabiner-Elements and import the
+   bundled rules (pointed at a stable copy of `vo_gesture.sh`). Speech logging works
+   without it; keyboard-gesture logging does not.
+
+## Example usage
+
+- `/lean-a11y-vo-log` — Claude opens the logging Terminal, you run your flow, then say
+  "read the log" and Claude writes up the bugs.

plugins/lean-a11y-vo-log/skills/lean-a11y-vo-log/SKILL.md

@@ -0,0 +1,91 @@
+---
+name: lean-a11y-vo-log
+description: Capture a text transcript of everything VoiceOver speaks (and each keyboard nav gesture) during a manual screen-reader session on macOS, then diagnose it into an accessibility bug report — a text alternative to screen recording. Claude spawns a dedicated Terminal running the logger, the dev enables VoiceOver and does the flow, then Claude reads the log and turns the announcement timeline into a bug report. Project-agnostic (any macOS app: web, native, Flutter). Trigger on "/lean-a11y-vo-log", "log VoiceOver", "capture VO speech", "record what the screen reader says".
+---
+
+# lean-a11y-vo-log — VoiceOver speech logging (developer-in-the-loop)
+
+Turns a manual VoiceOver session into a timestamped transcript, then diagnoses it.
+The dev drives VoiceOver by hand; every spoken phrase (and every nav gesture, if
+Karabiner logging is set up) lands in one file; Claude reads it and writes up the
+bugs. Project-agnostic — it captures whatever app is frontmost.
+
+**One-time machine setup lives in this plugin's `README.md`, not here** (VoiceOver
+AppleScript control, Automation permission, and the Karabiner gesture rules). The
+logger script prechecks that setup and prints what is missing, so you rarely need
+to explain it — just point the dev at the README's *Setup* section if a check fails.
+
+## Workflow
+
+### 1. Start the logger (Claude does this)
+
+```bash
+bash "${CLAUDE_PLUGIN_ROOT}/skills/lean-a11y-vo-log/scripts/start_log_terminal.sh" ~/vo_log.txt
+```
+
+The script **starts VoiceOver itself** if it's off, then opens a new Terminal window
+showing the live transcript. Do **not** activate/steal focus to that window — that
+moves the VoiceOver cursor off the app under test. If the script reports it couldn't
+start VoiceOver or can't control it over AppleScript, point the dev at the README
+*Setup* section (Cmd+F5 to toggle VO by hand; AppleScript-control + Automation
+permissions).
+
+### 2. Dev runs the session (human)
+
+Tell the developer:
+> Logger's live in the new window. Switch to the app, do the flow that's buggy.
+> Gestures auto-log if you set up the Karabiner rules; otherwise type a short note
+> + Enter in that window when you do something notable (e.g. "opened market
+> dropdown"). When you're done, come back and say "read the log".
+
+### 3. Read & diagnose (Claude does this)
+
+Extract just this session (from the last `=== requested … ===` marker) and build
+the timeline:
+
+```bash
+awk '/=== requested/{buf=""} {buf=buf $0 "\n"} END{printf "%s", buf}' ~/vo_log.txt
+```
+
+Then analyze the announcement sequence with the table below. For each bad
+announcement, state the chain *phrase → focused widget → file:line → property →
+fix*. After a fix, re-run this logger to confirm the announcement changed.
+
+## Diagnosing the transcript
+
+| Phrase / symptom | Means | Likely cause & fix |
+|------------------|-------|--------------------|
+| "…, group" + "press Control-Option-Shift-Down" | interactable group, not a transparent region — user must drill in | `Semantics` with `container: true` **and** a `label` wrapping interactive children. Drop them; a landmark `role` alone is transparent |
+| "end of, <name>" in the wrong place | hard group edge around a landmark | remove `container: true` |
+| GESTURE with no VO line (silent stop) | unlabeled node / covered pane | add `Semantics(label:)`, or `ExcludeSemantics` if decorative; if a whole pane, check an `ExcludeSemantics(excluding:)` gating it |
+| container announced with a child's name | name-steal — container unnamed | name the route/region (`namesRoute`) or remove the merging label |
+| element read twice per swipe | duplicate focus stop | collapse nested focusables to one |
+| title read as plain text | missing heading role | `Semantics(header: true)` / heading-level widget |
+| visible label ≠ spoken name | label overrides visible text (2.5.3) | align `Semantics(label:)` with the visible text |
+
+Useful greps (Flutter): `grep -rn "container: true" lib --include="*.dart"` (the
+usual landmark culprit) and `grep -rn "Semantics(" lib/<feature>`.
+
+## Reading signals
+
+- **GESTURE with no following VO line** → silent focus move. Focus went somewhere
+  unlabeled/unannounced. Prime suspect for "I don't know where I am".
+- **Same phrase repeated** → focus bounced back, or a live-region is chattering.
+- **A long stream after one action** → over-verbose node (e.g. a dialog reading its
+  whole subtree). Note it, but it's often acceptable.
+
+## Log format
+
+```
+HH:MM:SS | VO      | <phrase VoiceOver spoke>
+HH:MM:SS | GESTURE | vo-right          (auto, from Karabiner)
+HH:MM:SS | ACTION  | <freeform note the dev typed>
+```
+
+## Video fallback
+
+For TalkBack (no macOS scripting) or inherently visual bugs (focus-ring position),
+record with the caption panel on (VO+F10 / TalkBack "Display speech output"), then
+slice and read it frame by frame:
+`ffmpeg -v error -i <video> -vf "fps=2" ai_tmp/frames/f_%03d.png` — treat each
+caption exactly like a `VO |` log line. Delete `ai_tmp/` afterwards.