press_key emits stray characters instead of key events on X11 (named keys and modifier chords broken)

[martin152]

Summary

On an X11 session, press_key does not produce the key it is asked for. Named keys like Return and chords like ctrl+a or ctrl+s end up inserting stray characters into the focused app instead of firing the real key. Pointer input, screenshots, window control and the AT-SPI tools all work well. This is specifically the keyboard-by-name path.

It took me a while to isolate, because my first guess (keyboard layout) turned out to be wrong. The short version: raw keycode injection through ydotool is fine, so the fault sits in how key names become events.

Environment

• computer-use-linux 0.4.1, installed via ./install.sh
• Linux Mint 22.3 (Ubuntu 24.04 "noble" base)
• Session: X11, Cinnamon (X-Cinnamon), DISPLAY=:0
• Keyboard layout: US English. Confirmed in the tray. This matters, see below.
• ydotool 1.0.4 (official upstream binary), ydotoold running as a user service, socket at /run/user/1000/.ydotool_socket
• doctor input backends: abs_pointer, portal, ydotool; preferred = abs_pointer
• doctor readiness: all flags true, blockers: []

What happens

Target is a plain xed window, focused by pid. Every result below is confirmed by screenshotting the window afterwards, not by reading a file or the AT-SPI tree (both of those gave me false readings early on, so I stopped trusting them).

• type_text "HELLO FROM COMPUTER-USE-LINUX 12345" lands correctly. Letter typing works.
• press_key "Return" gives no newline. It inserts two stray glyphs and the cursor stays on line 1.
• press_key "ctrl+s" gives no save and no dialog. It inserts junk.
• press_key "ctrl+a" gives no select-all. It inserts junk.

The isolating test

I ran the tool against raw ydotool on the same window, same session, and screenshotted each step.

Raw ydotool, numeric keycodes:

• ydotool key 28:1 28:0 (Enter) produced a real newline. Works.
• ydotool key 29:1 30:1 30:0 29:0 (Ctrl+A) selected the whole buffer. Works.
• ydotool type "RAW_LINE1" came out as RAW%LINE+. Letters are fine, but _ became % and 1 became +.

The tool:

• type_text "CUL_LINE1" came out as CUL%LINE+. Same mangling as raw ydotool type.
• press_key "Return" gave stray chars, no newline.
• press_key "ctrl+a" gave stray chars, no select-all.

Two things fall out of this.

First, ydotool key <numeric keycode> is reliable here. Enter and Ctrl+A both did exactly what they should. So uinput, the daemon, permissions and the injection layer are all healthy.

Second, press_key does not use that reliable path. If it were sending keycode 28 for Return, it would produce a newline like the raw call did. It doesn't. So the name-to-keycode step is where it breaks. My read is that named keys and chords get routed through the ydotool type path (or something that XKB then remaps), rather than through ydotool key with the correct numeric code. I have not read the source, so I can't point at the exact spot.

There is a related but separate wrinkle. ydotool type itself mangles some symbols and digits under X11 (_ to %, 1 to +), and type_text inherits it because it shares that path. Letters survive. It is intermittent rather than constant. My very first type_text with "12345" came out clean, a later one turned "1" into "+". This looks like the usual ydotool-on-X11 behaviour where injected keycodes get reinterpreted by the active XKB layout. The layout here is plain US, so this is not a non-US-layout issue.

Why I think this belongs here and not upstream in ydotool

The raw ydotool key path already works on X11. So a fix could be as small as having press_key translate names and chords into numeric keycodes and send them via ydotool key ..., instead of whatever currently produces stray characters. For the type_text symbol mangling, an X11 session might be better served by xdotool / XTEST, which respects the layout. ydotool is Wayland-first by design, and the README already flags X11 as best-effort, so this feels consistent with that.

Repro sketch

1. X11 session, US layout.
2. Open a text editor and focus it.
3. Call press_key with Return, then ctrl+a, then ctrl+s.
4. Screenshot the window. Stray characters, no newline, no select, no save.
5. For contrast, run ydotool key 28:1 28:0 against the same window and watch a real newline appear.

I have annotated screenshots for each step and can attach them, or run any other combination you want tested. Thanks for the tool. Everything on the pointer, window and accessibility side has been solid.

[avifenesh]

Confirmed and fixed — PR #59.

Your isolation was exactly right, and the ydotool key <numeric> contrast was the key data point. Root cause: ydotool writes raw evdev scancodes to a virtual uinput device, and X11 re-interprets those through the active XKB layout. The uinput/daemon/permission layer is healthy (which is why raw ydotool key 28:1 28:0 gave a real newline) — the breakage is the scancode→layout translation, which hits both press_key (named keys/chords become stray glyphs) and type_text (symbol/digit mangling, _→%, 1→+).

So your instinct that this belongs here rather than upstream was correct, and your suggested direction (XTEST on X11) is what the fix does:

• X11 (non-Wayland + DISPLAY + xdotool present) → xdotool key --clearmodifiers / xdotool type --clearmodifiers, which resolves keysyms against the live layout
• Key grammar is validated through the same parser both backends share, so the accepted key set is identical
• ydotool stays as fallback if xdotool is missing or errors
• Wayland unchanged
• doctor now lists an xdotool input backend on X11

Verified on a real X server (Xvfb + xterm, driven through the MCP server) before shipping: type_text "RAW_LINE1 _ 1 __ 11" landed byte-exact, and press_key "Return" produced a real newline.

One thing worth flagging for your setup: install xdotool (apt install xdotool on Mint 22.3) — it's a soft dependency, but without it X11 falls back to the old ydotool path. If you can build the fix/x11-xtest-keyboard branch and re-run your xed sequence (Return, ctrl+a, ctrl+s) I'd love the confirmation, since you have a real Cinnamon/X11 session and my verification was on Xvfb.

Also: thanks for the careful methodology note about not trusting AT-SPI/file readback and screenshotting instead — that's the right call and it saved a round trip here.

[avifenesh]

Shipped in v0.4.2 — https://github.com/agent-sh/computer-use-linux/releases/tag/v0.4.2

Available now on crates.io (computer-use-linux 0.4.2), npm (@agent-sh/computer-use-linux@0.4.2), and as prebuilt x86_64/aarch64 binaries on the release.

@martin152 — one setup note for Mint 22.3: install xdotool (sudo apt install xdotool) so the XTEST path activates. It's a soft dependency; without it X11 silently falls back to the old ydotool behaviour. After installing, doctor should list xdotool under input backends.

Then your original sequence against xed should work: press_key "Return" gives a real newline, ctrl+a selects all, ctrl+s opens save, and type_text with _ and digits lands byte-exact. If anything still misbehaves, COMPUTER_USE_LINUX_FORCE_XDOTOOL_KEYBOARD=1 forces the path on (useful if session detection guesses wrong) and COMPUTER_USE_LINUX_FORCE_YDOTOOL_KEYBOARD=1 reverts to old behaviour for comparison.

Would still value your confirmation from a real Cinnamon/X11 session — my verification was Xvfb + xterm, which validates the mechanism but not your exact WM.