diff --git a/README/WHATS_NEW_zh-CN.md b/README/WHATS_NEW_zh-CN.md index d154ac7b..df3b421b 100644 --- a/README/WHATS_NEW_zh-CN.md +++ b/README/WHATS_NEW_zh-CN.md @@ -1,5 +1,11 @@ # 本次更新 — AutoControl +## 本次更新 (2026-06-23) — Unicode 文本输入(emoji / CJK) + +输入 `write` 无法处理的任何 Unicode(emoji / CJK / 重音)。完整参考:[`docs/source/Zh/doc/new_features/v122_features_doc.rst`](../docs/source/Zh/doc/new_features/v122_features_doc.rst)。 + +- **`type_unicode` / `plan_paste` / `unicode_code_units`**(`AC_type_unicode`):`write` 通过虚拟键表输入,对 emoji/CJK/许多重音字会*抛异常*。`type_unicode` 以设定剪贴板再粘贴(`modifier` ctrl/command)可靠地输入任何文本。`unicode_code_units` 将文本拆成 UTF-16 码元(代理对)供 KEYEVENTF_UNICODE 后端使用。纯计划 + 可注入 sink、确定。 + ## 本次更新 (2026-06-23) — 等待区域颜色 阻塞直到某颜色填满(或离开)屏幕区域。完整参考:[`docs/source/Zh/doc/new_features/v121_features_doc.rst`](../docs/source/Zh/doc/new_features/v121_features_doc.rst)。 diff --git a/README/WHATS_NEW_zh-TW.md b/README/WHATS_NEW_zh-TW.md index c0513727..612bf539 100644 --- a/README/WHATS_NEW_zh-TW.md +++ b/README/WHATS_NEW_zh-TW.md @@ -1,5 +1,11 @@ # 本次更新 — AutoControl +## 本次更新 (2026-06-23) — Unicode 文字輸入(emoji / CJK) + +輸入 `write` 無法處理的任何 Unicode(emoji / CJK / 重音)。完整參考:[`docs/source/Zh/doc/new_features/v122_features_doc.rst`](../docs/source/Zh/doc/new_features/v122_features_doc.rst)。 + +- **`type_unicode` / `plan_paste` / `unicode_code_units`**(`AC_type_unicode`):`write` 透過虛擬鍵表輸入,對 emoji/CJK/許多重音字會*拋例外*。`type_unicode` 以設定剪貼簿再貼上(`modifier` ctrl/command)可靠地輸入任何文字。`unicode_code_units` 將文字拆成 UTF-16 碼元(代理對)供 KEYEVENTF_UNICODE 後端使用。純計畫 + 可注入 sink、具決定性。 + ## 本次更新 (2026-06-23) — 等待區域顏色 阻塞直到某顏色填滿(或離開)螢幕區域。完整參考:[`docs/source/Zh/doc/new_features/v121_features_doc.rst`](../docs/source/Zh/doc/new_features/v121_features_doc.rst)。 diff --git a/WHATS_NEW.md b/WHATS_NEW.md index a4373ff0..c4d1bc82 100644 --- a/WHATS_NEW.md +++ b/WHATS_NEW.md @@ -1,5 +1,11 @@ # What's New — AutoControl +## What's new (2026-06-23) — Unicode Text Entry (Emoji / CJK) + +Type any Unicode (emoji / CJK / accented) that `write` can't. Full reference: [`docs/source/Eng/doc/new_features/v122_features_doc.rst`](docs/source/Eng/doc/new_features/v122_features_doc.rst). + +- **`type_unicode` / `plan_paste` / `unicode_code_units`** (`AC_type_unicode`): `write` types through the virtual-key table and *raises* on emoji/CJK/many accented chars. `type_unicode` enters any text reliably by setting the clipboard and pasting (`modifier` ctrl/command). `unicode_code_units` splits text into UTF-16 code units (surrogate pairs) for KEYEVENTF_UNICODE backends. Pure-planning + injectable sink, deterministic. + ## What's new (2026-06-23) — Wait for Region Colour Block until a colour fills (or leaves) a screen region. Full reference: [`docs/source/Eng/doc/new_features/v121_features_doc.rst`](docs/source/Eng/doc/new_features/v121_features_doc.rst). diff --git a/docs/source/Eng/doc/new_features/v122_features_doc.rst b/docs/source/Eng/doc/new_features/v122_features_doc.rst new file mode 100644 index 00000000..5bd54665 --- /dev/null +++ b/docs/source/Eng/doc/new_features/v122_features_doc.rst @@ -0,0 +1,41 @@ +Unicode Text Entry (Emoji / CJK) +================================ + +``write`` types through the platform virtual-key table and *raises* on any +character outside it — emoji, CJK, many accented letters — so non-ASCII text entry +was impossible through the normal path. The reliable, cross-platform way to enter +arbitrary Unicode is to put it on the clipboard and paste it. + +:func:`plan_paste` builds the deterministic op-plan and :func:`unicode_code_units` +splits text into UTF-16 code units (for a backend that can do +``KEYEVENTF_UNICODE``); both are pure and unit-testable. :func:`type_unicode` +dispatches the paste plan through an injectable ``sink`` so it is tested without +touching the real clipboard. Imports no ``PySide6``. + +Headless API +------------ + +.. code-block:: python + + from je_auto_control import type_unicode, plan_paste, unicode_code_units + + type_unicode("café 🚀 値") # clipboard set + Ctrl+V + type_unicode("値", modifier="command") # macOS + + unicode_code_units("🚀") # [0xD83D, 0xDE80] (surrogate pair) + plan_paste("hi") + # [{'op': 'set_clipboard', 'text': 'hi'}, + # {'op': 'hotkey', 'keys': ['ctrl', 'v']}] + +``type_unicode`` sets the clipboard to the text and sends the paste hotkey +(``modifier`` defaults to ``"ctrl"``; use ``"command"`` on macOS), so it enters +*any* text regardless of keyboard layout — emoji, CJK, RTL, accented. It returns +the dispatched plan plus the UTF-16 code-unit count. ``unicode_code_units`` is +provided for backends that want to inject code units directly. + +Executor commands +----------------- + +``AC_type_unicode`` takes ``text`` plus an optional ``modifier`` and returns +``{ops, plan, code_units}``. It is exposed as the MCP tool ``ac_type_unicode`` and +as a Script Builder command under **Keyboard**. diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst index 12e06cca..d52d3d31 100644 --- a/docs/source/Eng/eng_index.rst +++ b/docs/source/Eng/eng_index.rst @@ -144,6 +144,7 @@ Comprehensive guides for all AutoControl features. doc/new_features/v119_features_doc doc/new_features/v120_features_doc doc/new_features/v121_features_doc + doc/new_features/v122_features_doc doc/ocr_backends/ocr_backends_doc doc/observability/observability_doc doc/operations_layer/operations_layer_doc diff --git a/docs/source/Zh/doc/new_features/v122_features_doc.rst b/docs/source/Zh/doc/new_features/v122_features_doc.rst new file mode 100644 index 00000000..20c9840c --- /dev/null +++ b/docs/source/Zh/doc/new_features/v122_features_doc.rst @@ -0,0 +1,34 @@ +Unicode 文字輸入(emoji / CJK) +============================== + +``write`` 透過平台虛擬鍵表輸入,對任何不在表內的字元會*拋例外*——emoji、CJK、許多重音字母——因此無法以正常 +途徑輸入非 ASCII 文字。可靠的跨平台輸入任意 Unicode 的方法,是將其放上剪貼簿再貼上。 + +:func:`plan_paste` 建立決定性操作計畫,:func:`unicode_code_units` 將文字拆成 UTF-16 碼元(供能做 +``KEYEVENTF_UNICODE`` 的後端使用);兩者皆為純函式、可單元測試。:func:`type_unicode` 透過可注入的 ``sink`` +派發貼上計畫,因此可在不觸碰真實剪貼簿下測試。不匯入 ``PySide6``。 + +無頭 API +-------- + +.. code-block:: python + + from je_auto_control import type_unicode, plan_paste, unicode_code_units + + type_unicode("café 🚀 値") # 設定剪貼簿 + Ctrl+V + type_unicode("値", modifier="command") # macOS + + unicode_code_units("🚀") # [0xD83D, 0xDE80](代理對) + plan_paste("hi") + # [{'op': 'set_clipboard', 'text': 'hi'}, + # {'op': 'hotkey', 'keys': ['ctrl', 'v']}] + +``type_unicode`` 將剪貼簿設為該文字並送出貼上熱鍵(``modifier`` 預設 ``"ctrl"``;macOS 用 ``"command"``), +因此無論鍵盤配置都能輸入*任何*文字——emoji、CJK、RTL、重音。它回傳派發的計畫加上 UTF-16 碼元數。 +``unicode_code_units`` 供想直接注入碼元的後端使用。 + +執行器命令 +---------- + +``AC_type_unicode`` 接受 ``text`` 與選用的 ``modifier``,並回傳 ``{ops, plan, code_units}``。它以 MCP 工具 +``ac_type_unicode`` 以及 Script Builder 中 **Keyboard** 分類下的命令提供。 diff --git a/docs/source/Zh/zh_index.rst b/docs/source/Zh/zh_index.rst index f91c3262..159dfe01 100644 --- a/docs/source/Zh/zh_index.rst +++ b/docs/source/Zh/zh_index.rst @@ -144,6 +144,7 @@ AutoControl 所有功能的完整使用指南。 doc/new_features/v119_features_doc doc/new_features/v120_features_doc doc/new_features/v121_features_doc + doc/new_features/v122_features_doc doc/ocr_backends/ocr_backends_doc doc/observability/observability_doc doc/operations_layer/operations_layer_doc diff --git a/je_auto_control/__init__.py b/je_auto_control/__init__.py index 13e9cd35..350cf419 100644 --- a/je_auto_control/__init__.py +++ b/je_auto_control/__init__.py @@ -263,6 +263,10 @@ from je_auto_control.utils.mouse_relative import ( move_mouse_relative, relative_target, ) +# Type arbitrary Unicode (emoji / CJK) via the clipboard +from je_auto_control.utils.text_unicode import ( + plan_paste, type_unicode, unicode_code_units, +) # CI workflow annotations (GitHub Actions) from je_auto_control.utils.ci_annotations import ( emit_annotations, format_annotation, @@ -1045,6 +1049,9 @@ def start_autocontrol_gui(*args, **kwargs): "hold_key", "move_mouse_relative", "relative_target", + "type_unicode", + "plan_paste", + "unicode_code_units", "emit_annotations", "format_annotation", "ClipboardHistory", "default_clipboard_history", "analyze_heal_log", "heal_stats", "scan_secrets", diff --git a/je_auto_control/gui/script_builder/command_schema.py b/je_auto_control/gui/script_builder/command_schema.py index 764f7771..f33a256c 100644 --- a/je_auto_control/gui/script_builder/command_schema.py +++ b/je_auto_control/gui/script_builder/command_schema.py @@ -185,6 +185,15 @@ def _add_keyboard_specs(specs: List[CommandSpec]) -> None: ), description="Hold a key for a duration, or auto-repeat it at rate_hz.", )) + specs.append(CommandSpec( + "AC_type_unicode", "Keyboard", "Type Unicode (emoji / CJK)", + fields=( + FieldSpec("text", FieldType.STRING, placeholder="café 🚀 値"), + FieldSpec("modifier", FieldType.STRING, optional=True, + default="ctrl", placeholder="ctrl | command"), + ), + description="Enter any Unicode text via clipboard paste (write can't).", + )) specs.append(CommandSpec( "AC_hotkey", "Keyboard", "Hotkey", fields=( diff --git a/je_auto_control/utils/executor/action_executor.py b/je_auto_control/utils/executor/action_executor.py index d805fa9a..3a7cfea9 100644 --- a/je_auto_control/utils/executor/action_executor.py +++ b/je_auto_control/utils/executor/action_executor.py @@ -3155,6 +3155,12 @@ def _move_mouse_relative(dx: Any, dy: Any) -> Dict[str, Any]: return move_mouse_relative(int(dx), int(dy)) +def _type_unicode(text: str, modifier: str = "ctrl") -> Dict[str, Any]: + """Adapter: enter arbitrary Unicode text via clipboard paste.""" + from je_auto_control.utils.text_unicode import type_unicode + return type_unicode(text, modifier=modifier) + + def _cas_put(name: str, key: str, value: Any, expected_version: Any = None) -> Dict[str, Any]: """Adapter: optimistic put into a named versioned store.""" @@ -4855,6 +4861,7 @@ def __init__(self): "AC_set_field_text": _set_field_text, "AC_hold_key": _hold_key, "AC_move_mouse_relative": _move_mouse_relative, + "AC_type_unicode": _type_unicode, "AC_detect_drift": _detect_drift, "AC_categorical_drift": _categorical_drift, "AC_diff_rows": _diff_rows, diff --git a/je_auto_control/utils/mcp_server/tools/_factories.py b/je_auto_control/utils/mcp_server/tools/_factories.py index e419914b..02fb398c 100644 --- a/je_auto_control/utils/mcp_server/tools/_factories.py +++ b/je_auto_control/utils/mcp_server/tools/_factories.py @@ -2596,6 +2596,23 @@ def tween_drag_tools() -> List[MCPTool]: ] +def text_unicode_tools() -> List[MCPTool]: + return [ + MCPTool( + name="ac_type_unicode", + description=("Enter arbitrary Unicode 'text' (emoji / CJK / accented " + "that the normal type cannot) via clipboard paste. " + "'modifier' is the paste key (ctrl / command). " + "Returns {ops, plan, code_units}."), + input_schema=schema( + {"text": {"type": "string"}, "modifier": {"type": "string"}}, + required=["text"]), + handler=h.type_unicode, + annotations=SIDE_EFFECT_ONLY, + ), + ] + + def mouse_relative_tools() -> List[MCPTool]: return [ MCPTool( @@ -5929,7 +5946,8 @@ def media_assert_tools() -> List[MCPTool]: input_macro_tools, resilience_tools, ci_annotation_tools, clipboard_history_tools, audit_analysis_tools, process_doc_tools, tween_drag_tools, mouse_path_tools, field_entry_tools, - key_hold_tools, mouse_relative_tools, plugin_sdk_tools, governance_tools, + key_hold_tools, mouse_relative_tools, text_unicode_tools, plugin_sdk_tools, + governance_tools, credential_lease_tools, egress_tools, approval_testing_tools, trajectory_eval_tools, compliance_tools, agent_trace_tools, video_report_tools, fuzzy_tools, artifact_store_tools, image_dedup_tools, diff --git a/je_auto_control/utils/mcp_server/tools/_handlers.py b/je_auto_control/utils/mcp_server/tools/_handlers.py index 83224972..66fa8eca 100644 --- a/je_auto_control/utils/mcp_server/tools/_handlers.py +++ b/je_auto_control/utils/mcp_server/tools/_handlers.py @@ -2053,6 +2053,11 @@ def move_mouse_relative(dx, dy): return _move_mouse_relative(dx, dy) +def type_unicode(text, modifier="ctrl"): + from je_auto_control.utils.executor.action_executor import _type_unicode + return _type_unicode(text, modifier) + + def detect_drift(reference, current, threshold=0.25, bins=10): from je_auto_control.utils.executor.action_executor import _detect_drift return _detect_drift(reference, current, threshold, bins) diff --git a/je_auto_control/utils/text_unicode/__init__.py b/je_auto_control/utils/text_unicode/__init__.py new file mode 100644 index 00000000..e94ea2ce --- /dev/null +++ b/je_auto_control/utils/text_unicode/__init__.py @@ -0,0 +1,6 @@ +"""Type arbitrary Unicode (emoji / CJK / accented) via the clipboard.""" +from je_auto_control.utils.text_unicode.text_unicode import ( + plan_paste, type_unicode, unicode_code_units, +) + +__all__ = ["plan_paste", "type_unicode", "unicode_code_units"] diff --git a/je_auto_control/utils/text_unicode/text_unicode.py b/je_auto_control/utils/text_unicode/text_unicode.py new file mode 100644 index 00000000..de7edf2a --- /dev/null +++ b/je_auto_control/utils/text_unicode/text_unicode.py @@ -0,0 +1,62 @@ +"""Type arbitrary Unicode (emoji / CJK / accented) via the clipboard. + +``write`` types through the platform virtual-key table and *raises* on any +character outside it — emoji, CJK, many accented letters — so non-ASCII text +entry is impossible through the normal path. The reliable, cross-platform way to +enter arbitrary Unicode is to put it on the clipboard and paste it. + +:func:`plan_paste` builds the deterministic op-plan and :func:`unicode_code_units` +splits text into UTF-16 code units (for a backend that can do +``KEYEVENTF_UNICODE``); both are pure and unit-testable. :func:`type_unicode` +dispatches the paste plan through an injectable ``sink`` so it is tested without +touching the real clipboard. Imports no ``PySide6``. +""" +from typing import Any, Callable, Dict, List, Optional + +Sink = Callable[[Dict[str, Any]], None] + + +def unicode_code_units(text: str) -> List[int]: + """Return the UTF-16 code units of ``text`` (surrogate pairs for > U+FFFF).""" + units: List[int] = [] + for char in text or "": + code = ord(char) + if code > 0xFFFF: + code -= 0x10000 + units.append(0xD800 + (code >> 10)) + units.append(0xDC00 + (code & 0x3FF)) + else: + units.append(code) + return units + + +def plan_paste(text: str, *, modifier: str = "ctrl") -> List[Dict[str, Any]]: + """Return the op-plan to enter ``text`` via clipboard paste.""" + return [{"op": "set_clipboard", "text": text}, + {"op": "hotkey", "keys": [modifier, "v"]}] + + +def _default_sink(event: Dict[str, Any]) -> None: + """Default dispatch: drive the real clipboard / keyboard backend.""" + op = event["op"] + if op == "set_clipboard": + from je_auto_control.utils.clipboard.clipboard import set_clipboard + set_clipboard(event["text"]) + elif op == "hotkey": + from je_auto_control.wrapper.auto_control_keyboard import hotkey + hotkey(list(event["keys"])) + + +def type_unicode(text: str, *, modifier: str = "ctrl", + sink: Optional[Sink] = None) -> Dict[str, Any]: + """Enter ``text`` (any Unicode) by setting the clipboard then pasting. + + ``modifier`` is the platform paste key (``"ctrl"``; use ``"command"`` on + macOS). Returns the dispatched plan plus the UTF-16 code-unit count. + """ + plan = plan_paste(text, modifier=modifier) + dispatch = sink or _default_sink + for event in plan: + dispatch(event) + return {"ops": len(plan), "plan": plan, + "code_units": len(unicode_code_units(text))} diff --git a/test/unit_test/headless/test_text_unicode_batch.py b/test/unit_test/headless/test_text_unicode_batch.py new file mode 100644 index 00000000..f2c80141 --- /dev/null +++ b/test/unit_test/headless/test_text_unicode_batch.py @@ -0,0 +1,58 @@ +"""Headless tests for Unicode text entry via clipboard. No Qt.""" +import je_auto_control as ac +from je_auto_control.utils.text_unicode import ( + plan_paste, type_unicode, unicode_code_units, +) + + +def test_code_units_ascii_and_bmp(): + assert unicode_code_units("Hi") == [72, 105] + assert unicode_code_units("値") == [0x5024] + assert unicode_code_units("") == [] + + +def test_code_units_astral_surrogate_pair(): + assert unicode_code_units("🚀") == [0xD83D, 0xDE80] # rocket > U+FFFF + assert len(unicode_code_units("a🚀b")) == 4 # 1 + 2 + 1 + + +def test_plan_paste_is_clipboard_then_hotkey(): + assert plan_paste("café 🚀") == [ + {"op": "set_clipboard", "text": "café 🚀"}, + {"op": "hotkey", "keys": ["ctrl", "v"]}, + ] + assert plan_paste("x", modifier="command")[1]["keys"] == ["command", "v"] + + +def test_type_unicode_dispatches_plan(): + events = [] + result = type_unicode("café 🚀", sink=events.append) + assert [e["op"] for e in events] == ["set_clipboard", "hotkey"] + assert events[0]["text"] == "café 🚀" + assert result["ops"] == 2 and result["code_units"] == 7 + + +# --- wiring --------------------------------------------------------------- + +def test_executor_adapter_dispatches_via_sink(): + # the executor default dispatch is device-bound; verify the planning the + # adapter delegates to instead. + events = [] + type_unicode("値", sink=events.append, modifier="ctrl") + assert events[-1] == {"op": "hotkey", "keys": ["ctrl", "v"]} + + +def test_wiring(): + known = ac.executor.known_commands() + assert "AC_type_unicode" in set(known) + from je_auto_control.utils.mcp_server.tools import build_default_tool_registry + names = {t.name for t in build_default_tool_registry()} + assert "ac_type_unicode" in names + from je_auto_control.gui.script_builder.command_schema import _build_specs + specs = {s.command for s in _build_specs()} + assert "AC_type_unicode" in specs + + +def test_facade_exports(): + for attr in ("type_unicode", "plan_paste", "unicode_code_units"): + assert hasattr(ac, attr) and attr in ac.__all__