From 919c22dfc95350cc891b9887da59f4c2e1d55aa4 Mon Sep 17 00:00:00 2001 From: Oskar Zajac Date: Fri, 10 Jul 2026 19:00:51 +0200 Subject: [PATCH 1/6] a11y-log: add VoiceOver speech-logging plugin Capture a timestamped transcript of everything VoiceOver speaks (and each keyboard nav gesture, via Karabiner) during a manual screen-reader session on macOS, then diagnose it into an accessibility bug report via /a11y-log. One-time machine setup (VoiceOver AppleScript control, Automation permission, Karabiner gesture rules) lives in the plugin README, not in the skill body, so it never loads into context on invocation. The skill body is orchestration plus the announcement-diagnosis table. The Karabiner rules ship with a placeholder path stamped to a stable copy of vo_gesture.sh during setup, so plugin version bumps (versioned install path) don't break gesture logging. --- .claude-plugin/marketplace.json | 9 ++ README.md | 2 + plugins/a11y-log/.claude-plugin/plugin.json | 20 +++ plugins/a11y-log/README.md | 109 ++++++++++++++ .../a11y-log/skills/a11y-log-usage/SKILL.md | 52 +++++++ plugins/a11y-log/skills/a11y-log/SKILL.md | 97 +++++++++++++ .../assets/karabiner_vo_gesture_logger.json | 133 ++++++++++++++++++ .../a11y-log/scripts/start_log_terminal.sh | 30 ++++ .../skills/a11y-log/scripts/vo_gesture.sh | 17 +++ .../skills/a11y-log/scripts/vo_log.sh | 114 +++++++++++++++ 10 files changed, 583 insertions(+) create mode 100644 plugins/a11y-log/.claude-plugin/plugin.json create mode 100644 plugins/a11y-log/README.md create mode 100644 plugins/a11y-log/skills/a11y-log-usage/SKILL.md create mode 100644 plugins/a11y-log/skills/a11y-log/SKILL.md create mode 100644 plugins/a11y-log/skills/a11y-log/assets/karabiner_vo_gesture_logger.json create mode 100755 plugins/a11y-log/skills/a11y-log/scripts/start_log_terminal.sh create mode 100755 plugins/a11y-log/skills/a11y-log/scripts/vo_gesture.sh create mode 100755 plugins/a11y-log/skills/a11y-log/scripts/vo_log.sh diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json index 8ccadea..74125a4 100644 --- a/.claude-plugin/marketplace.json +++ b/.claude-plugin/marketplace.json @@ -130,6 +130,15 @@ "./skills/flutter-read-logs-usage", "./skills/read-logs" ] + }, + { + "name": "a11y-log", + "source": "./plugins/a11y-log", + "description": "Capture a timestamped transcript of everything VoiceOver speaks (and each keyboard nav gesture) during a manual screen-reader session on macOS, then diagnose it into an accessibility bug report via /a11y-log.", + "skills": [ + "./skills/a11y-log-usage", + "./skills/a11y-log" + ] } ] } diff --git a/README.md b/README.md index 9733f55..f268fb3 100644 --- a/README.md +++ b/README.md @@ -55,6 +55,7 @@ Most plugins are pure rules and skills with no setup. A few need one-time toolin - [`flutter-patrol`](plugins/flutter-patrol/) - Patrol CLI and Patrol MCP - [`flutter-marionette`](plugins/flutter-marionette/) - Marionette MCP and app-side binding +- [`a11y-log`](plugins/a11y-log/) - VoiceOver AppleScript control, macOS Automation permission, and (optional) Karabiner gesture rules ## Available plugins @@ -83,6 +84,7 @@ Most plugins are pure rules and skills with no setup. A few need one-time toolin - [`flutter-patrol`](plugins/flutter-patrol/) - Patrol test architecture, key conventions, and Patrol MCP workflow for AI-assisted E2E work - [`flutter-marionette`](plugins/flutter-marionette/) - runtime interaction with a live debug app through Marionette MCP for exploration, smoke checks, and UI debugging - [`flutter-read-logs`](plugins/flutter-read-logs/) - read the running app's latest `flutter run` logs as on-demand context via `/read-logs` +- [`a11y-log`](plugins/a11y-log/) - capture a text transcript of everything VoiceOver speaks (and each keyboard nav gesture) during a manual screen-reader session on macOS, then diagnose it via `/a11y-log` ### Every plugin has a `-usage` skill diff --git a/plugins/a11y-log/.claude-plugin/plugin.json b/plugins/a11y-log/.claude-plugin/plugin.json new file mode 100644 index 0000000..1e71c2e --- /dev/null +++ b/plugins/a11y-log/.claude-plugin/plugin.json @@ -0,0 +1,20 @@ +{ + "$schema": "https://json.schemastore.org/claude-code-plugin-manifest.json", + "name": "a11y-log", + "displayName": "A11y Log (VoiceOver)", + "description": "Capture a timestamped transcript of everything VoiceOver speaks — and each keyboard nav gesture — during a manual screen-reader session on macOS, then diagnose it into an accessibility bug report. A text alternative to screen recording.", + "version": "0.1.0", + "author": { + "name": "LeanCode" + }, + "homepage": "https://github.com/leancodepl/ai-plugins", + "repository": "https://github.com/leancodepl/ai-plugins", + "keywords": [ + "accessibility", + "a11y", + "voiceover", + "screen-reader", + "macos" + ], + "skills": "./skills/" +} diff --git a/plugins/a11y-log/README.md b/plugins/a11y-log/README.md new file mode 100644 index 0000000..af07f53 --- /dev/null +++ b/plugins/a11y-log/README.md @@ -0,0 +1,109 @@ +# a11y-log + +Capture a timestamped **text transcript of everything VoiceOver speaks** during a +manual screen-reader session on macOS — and, optionally, each keyboard nav gesture +— then have Claude diagnose it into an accessibility bug report. A text alternative +to recording and re-watching video. + +The developer drives VoiceOver by hand; Claude opens the logging Terminal, reads the +resulting file, and turns the announcement timeline into `phrase → widget → +file:line → fix` chains. Project-agnostic: it logs whatever macOS app is frontmost +(web, native, Flutter). + +## Included assets + +- `skills/a11y-log/SKILL.md` — the workflow (spawn logger → dev runs flow → Claude + diagnoses) plus the announcement-diagnosis table +- `skills/a11y-log-usage/SKILL.md` — routing skill: what the plugin does and when to + reach for it vs. `a11y-audit` +- `skills/a11y-log/scripts/vo_log.sh` — the poller: reads VoiceOver's "last phrase" + ~6×/s (deduped) and appends `VO` lines +- `skills/a11y-log/scripts/start_log_terminal.sh` — opens the dedicated logging + Terminal window running `vo_log.sh` +- `skills/a11y-log/scripts/vo_gesture.sh` — appends a `GESTURE` line; called by + Karabiner on each VoiceOver nav combo +- `skills/a11y-log/assets/karabiner_vo_gesture_logger.json` — Karabiner rules + template (the `__VO_GESTURE_SH__` placeholder is stamped during setup) + +## How it works + +`vo_log.sh` polls VoiceOver over AppleScript and writes one line per spoken phrase. +Keyboard gestures cannot be captured by a self-contained tap (VoiceOver disables a +`CGEventTap` after the first event), so gesture lines come from **Karabiner-Elements**: +each VoiceOver nav combo (VO+Right/Left/Up/Down, VO+Space, VO+Shift+Down…) runs +`vo_gesture.sh`, which appends a `GESTURE` line to the same log — including for +**silent focus moves** (a gesture with no `VO` line after it = focus moved but +nothing was spoken, the #1 screen-reader smell). + +``` +HH:MM:SS | VO | +HH:MM:SS | GESTURE | vo-right (auto, from Karabiner) +HH:MM:SS | ACTION | +``` + +## Setup (one-time, per machine) + +### 1. Let VoiceOver be driven by AppleScript + +VoiceOver Utility (**VO+F8**) → General → ☑ *"Allow VoiceOver to be controlled with +AppleScript"*. + +### 2. Grant Automation permission + +The process running Claude Code needs to control **VoiceOver** and **Terminal**. The +first run may prompt *"… wants to control Terminal / VoiceOver"* → **Allow**. If it +errors instead, add them under System Settings → Privacy & Security → Automation. + +That is enough for **speech logging**. `/a11y-log` now works — you just won't get +`GESTURE` lines until step 3. + +### 3. Keyboard-gesture logging (Karabiner-Elements) + +This is what adds the `GESTURE` lines (and lets Claude spot silent focus moves). +Install [Karabiner-Elements](https://karabiner-elements.pqrs.org/), then wire the +bundled rules to a **stable** copy of the gesture logger — stable so plugin updates +(which land under a versioned path) don't break Karabiner: + +```bash +# Resolve the installed plugin's files (latest version wins) +GEST_SRC="$(find ~/.claude/plugins/cache -path '*/a11y-log/skills/a11y-log/scripts/vo_gesture.sh' | sort | tail -1)" +RULES_SRC="$(find ~/.claude/plugins/cache -path '*/a11y-log/skills/a11y-log/assets/karabiner_vo_gesture_logger.json' | sort | tail -1)" + +# a) Copy the gesture logger to a stable location that survives plugin updates +mkdir -p ~/.local/bin +cp "$GEST_SRC" ~/.local/bin/vo_gesture.sh +chmod +x ~/.local/bin/vo_gesture.sh + +# b) Stamp that absolute path into the rules and drop them where Karabiner imports from +mkdir -p ~/.config/karabiner/assets/complex_modifications +sed "s#__VO_GESTURE_SH__#$HOME/.local/bin/vo_gesture.sh#g" "$RULES_SRC" \ + > ~/.config/karabiner/assets/complex_modifications/vo_gesture_logger.json +``` + +Then in **Karabiner-Elements → Settings → Complex Modifications → Add rule**, enable +the *"VoiceOver gesture logger"* rules. + +> Re-run step 3 only if you later change `vo_gesture.sh` itself; a normal plugin +> version bump does not require it, because Karabiner points at the stable copy. + +## Usage + +``` +/a11y-log +``` + +Claude opens the logging Terminal, you enable VoiceOver (**Cmd+F5**) and run the +buggy flow, then say **"read the log"** and Claude writes up the bugs. Type a short +note + Enter in the log window to record a visual observation as an `ACTION` line. + +## Data-handling note + +The transcript contains whatever VoiceOver reads aloud — screen text, labels, and +any on-screen customer data. Reading the log sends that content to the model. Don't +run it against production/real-customer data unless you've accepted that, and delete +`~/vo_log.txt` when done if it captured anything sensitive. + +## Related plugins + +- [`flutter-marionette`](../flutter-marionette/) — AI drives a live Flutter app + (runtime exploration); pairs with autonomous a11y checks. diff --git a/plugins/a11y-log/skills/a11y-log-usage/SKILL.md b/plugins/a11y-log/skills/a11y-log-usage/SKILL.md new file mode 100644 index 0000000..bb83df7 --- /dev/null +++ b/plugins/a11y-log/skills/a11y-log-usage/SKILL.md @@ -0,0 +1,52 @@ +--- +name: a11y-log-usage +description: Explain what the `a11y-log` plugin does and how to use it. Use when the user invokes `/a11y-log-usage`, asks what this plugin covers, or needs help with VoiceOver speech logging, gesture capture, or the one-time setup. +--- + +# a11y-log Usage + +## How to respond + +- If the user invoked this skill without a concrete task, start by explaining what + the plugin is for and when it beats a screen recording. +- Point to the one-time setup in the plugin `README.md` (VoiceOver AppleScript + control, Automation permission, optional Karabiner gesture rules). +- If the user already has a concrete logging task, hand off to the `a11y-log` + skill and do the work. +- Do not reply with filler like "skill loaded" or "ready for the task" before + explaining the plugin. + +## What this plugin does + +- Captures a **timestamped text transcript** of everything VoiceOver speaks during + a manual screen-reader session on macOS — a text alternative to recording video. +- Optionally interleaves **keyboard nav gestures** (via Karabiner-Elements), so the + transcript shows exactly what the dev did between announcements — including silent + focus moves (a gesture with no VO line after it), the #1 screen-reader smell. +- Then **diagnoses** the announcement timeline into an accessibility bug report, + chaining each bad announcement to the likely widget, file, and fix. +- Is **project-agnostic**: it logs whatever macOS app is frontmost (web, native, + Flutter). + +## When to reach for it vs. `a11y-audit` + +- **`a11y-log`** — the *dev* drives VoiceOver by hand and Claude reads the resulting + transcript. Best for reproducing a bug the dev already feels, or capturing a real + human navigation flow. +- **`a11y-audit`** — Claude *operates* VoiceOver itself (autonomous find/fix/verify), + no human swiping. + +## Setup + +One-time, per machine — see the plugin `README.md`: + +1. VoiceOver → allow AppleScript control. +2. Grant Automation permission to the process running Claude Code (Terminal + + VoiceOver). +3. (For gesture lines) import the bundled Karabiner rules pointing at a stable copy + of `vo_gesture.sh`. + +## Example usage + +- `/a11y-log` — Claude opens the logging Terminal, you run your flow, then say + "read the log" and Claude writes up the bugs. diff --git a/plugins/a11y-log/skills/a11y-log/SKILL.md b/plugins/a11y-log/skills/a11y-log/SKILL.md new file mode 100644 index 0000000..cafda00 --- /dev/null +++ b/plugins/a11y-log/skills/a11y-log/SKILL.md @@ -0,0 +1,97 @@ +--- +name: a11y-log +description: Capture a text transcript of everything VoiceOver speaks (and each keyboard nav gesture) during a manual screen-reader session on macOS, then diagnose it into an accessibility bug report — a text alternative to screen recording. Claude spawns a dedicated Terminal running the logger, the dev enables VoiceOver and does the flow, then Claude reads the log and turns the announcement timeline into a bug report. Project-agnostic (any macOS app: web, native, Flutter). Trigger on "/a11y-log", "log VoiceOver", "capture VO speech", "record what the screen reader says". +--- + +# a11y-log — VoiceOver speech logging (developer-in-the-loop) + +Turns a manual VoiceOver session into a timestamped transcript, then diagnoses it. +The dev drives VoiceOver by hand; every spoken phrase (and every nav gesture, if +Karabiner logging is set up) lands in one file; Claude reads it and writes up the +bugs. Project-agnostic — it captures whatever app is frontmost. + +**One-time machine setup lives in this plugin's `README.md`, not here** (VoiceOver +AppleScript control, Automation permission, and the Karabiner gesture rules). The +logger script prechecks that setup and prints what is missing, so you rarely need +to explain it — just point the dev at the README's *Setup* section if a check fails. + +## Workflow + +### 1. Start the logger (Claude does this) + +```bash +bash "${CLAUDE_PLUGIN_ROOT}/skills/a11y-log/scripts/start_log_terminal.sh" ~/vo_log.txt +``` + +A new Terminal window opens showing the live transcript. Do **not** activate/steal +focus to it — that moves the VoiceOver cursor off the app under test. If the +script's precheck fails, tell the dev to press **Cmd+F5** first (Claude must not +toggle VoiceOver — the dev controls it so focus lands where they expect) and see +the README *Setup* section. + +### 2. Dev runs the session (human) + +Tell the developer: +> Logger's live in the new window. Switch to the app, do the flow that's buggy. +> Gestures auto-log if you set up the Karabiner rules; otherwise type a short note +> + Enter in that window when you do something notable (e.g. "opened market +> dropdown"). When you're done, come back and say "read the log". + +### 3. Read & diagnose (Claude does this) + +Extract just this session (from the last `=== requested … ===` marker) and build +the timeline: + +```bash +awk '/=== requested/{buf=""} {buf=buf $0 "\n"} END{printf "%s", buf}' ~/vo_log.txt +``` + +Then analyze the announcement sequence with the table below. For each bad +announcement, state the chain *phrase → focused widget → file:line → property → +fix*. Fixing and verifying continues via the `a11y-audit` skill (the AI re-drives +VoiceOver to prove the fix) or the dev re-runs this logger after the change. + +## Diagnosing the transcript + +| Phrase / symptom | Means | Likely cause & fix | +|------------------|-------|--------------------| +| "…, group" + "press Control-Option-Shift-Down" | interactable group, not a transparent region — user must drill in | `Semantics` with `container: true` **and** a `label` wrapping interactive children. Drop them; a landmark `role` alone is transparent | +| "end of, " in the wrong place | hard group edge around a landmark | remove `container: true` | +| GESTURE with no VO line (silent stop) | unlabeled node / covered pane | add `Semantics(label:)`, or `ExcludeSemantics` if decorative; if a whole pane, check an `ExcludeSemantics(excluding:)` gating it | +| container announced with a child's name | name-steal — container unnamed | name the route/region (`namesRoute`) or remove the merging label | +| element read twice per swipe | duplicate focus stop | collapse nested focusables to one | +| title read as plain text | missing heading role | `Semantics(header: true)` / heading-level widget | +| visible label ≠ spoken name | label overrides visible text (2.5.3) | align `Semantics(label:)` with the visible text | + +Useful greps (Flutter): `grep -rn "container: true" lib --include="*.dart"` (the +usual landmark culprit) and `grep -rn "Semantics(" lib/`. + +## Reading signals + +- **GESTURE with no following VO line** → silent focus move. Focus went somewhere + unlabeled/unannounced. Prime suspect for "I don't know where I am". +- **Same phrase repeated** → focus bounced back, or a live-region is chattering. +- **A long stream after one action** → over-verbose node (e.g. a dialog reading its + whole subtree). Note it, but it's often acceptable. + +## Log format + +``` +HH:MM:SS | VO | +HH:MM:SS | GESTURE | vo-right (auto, from Karabiner) +HH:MM:SS | ACTION | +``` + +## Video fallback + +For TalkBack (no macOS scripting) or inherently visual bugs (focus-ring position), +record with the caption panel on (VO+F10 / TalkBack "Display speech output"), then +slice and read it frame by frame: +`ffmpeg -v error -i