diff --git a/README/WHATS_NEW_zh-CN.md b/README/WHATS_NEW_zh-CN.md index 1a4c438c..b492e979 100644 --- a/README/WHATS_NEW_zh-CN.md +++ b/README/WHATS_NEW_zh-CN.md @@ -1,5 +1,11 @@ # 本次更新 — AutoControl +## 本次更新 (2026-06-23) — 清空再输入字段 + +可靠地设定文本字段的值(Playwright 的 `fill` 惯用法)。完整参考:[`docs/source/Zh/doc/new_features/v117_features_doc.rst`](../docs/source/Zh/doc/new_features/v117_features_doc.rst)。 + +- **`set_field_text` / `plan_field_set`**(`AC_set_field_text`):先前没有单一的「聚焦 → 清空 → 设值」基本元件,且 `write` 对 emoji/CJK 会抛异常。本功能清空字段(全选 + 删除)后再输入文本——可选择通过剪贴板(`paste=True`),这是 `write` 无法处理之 Unicode 的安全途径。`modifier` 为平台命令键(`ctrl`/`command`)。纯计划 + 可注入 sink、确定。 + ## 本次更新 (2026-06-22) — 多路径点鼠标手势 让指针沿着路径点折线移动或拖曳。完整参考:[`docs/source/Zh/doc/new_features/v116_features_doc.rst`](../docs/source/Zh/doc/new_features/v116_features_doc.rst)。 diff --git a/README/WHATS_NEW_zh-TW.md b/README/WHATS_NEW_zh-TW.md index 1e5aeb88..c50b73dc 100644 --- a/README/WHATS_NEW_zh-TW.md +++ b/README/WHATS_NEW_zh-TW.md @@ -1,5 +1,11 @@ # 本次更新 — AutoControl +## 本次更新 (2026-06-23) — 清空再輸入欄位 + +可靠地設定文字欄位的值(Playwright 的 `fill` 慣用法)。完整參考:[`docs/source/Zh/doc/new_features/v117_features_doc.rst`](../docs/source/Zh/doc/new_features/v117_features_doc.rst)。 + +- **`set_field_text` / `plan_field_set`**(`AC_set_field_text`):先前沒有單一的「聚焦 → 清空 → 設值」基本元件,且 `write` 對 emoji/CJK 會拋例外。本功能清空欄位(全選 + 刪除)後再輸入文字——可選擇透過剪貼簿(`paste=True`),這是 `write` 無法處理之 Unicode 的安全途徑。`modifier` 為平台指令鍵(`ctrl`/`command`)。純計畫 + 可注入 sink、具決定性。 + ## 本次更新 (2026-06-22) — 多路徑點滑鼠手勢 讓指標沿著路徑點折線移動或拖曳。完整參考:[`docs/source/Zh/doc/new_features/v116_features_doc.rst`](../docs/source/Zh/doc/new_features/v116_features_doc.rst)。 diff --git a/WHATS_NEW.md b/WHATS_NEW.md index 2015723d..78991a24 100644 --- a/WHATS_NEW.md +++ b/WHATS_NEW.md @@ -1,5 +1,11 @@ # What's New — AutoControl +## What's new (2026-06-23) — Clear-Then-Type Field Entry + +Reliably set a text field's value (the Playwright `fill` idiom). Full reference: [`docs/source/Eng/doc/new_features/v117_features_doc.rst`](docs/source/Eng/doc/new_features/v117_features_doc.rst). + +- **`set_field_text` / `plan_field_set`** (`AC_set_field_text`): there was no single "focus → clear → set value" primitive, and `write` raises on emoji/CJK. This clears the field (select-all + delete) then enters the text — optionally via the clipboard (`paste=True`) which is the Unicode-safe path `write` can't do. `modifier` is the platform command key (`ctrl`/`command`). Pure-planning + injectable sink, deterministic. + ## What's new (2026-06-22) — Multi-Waypoint Mouse Gestures Move or drag the pointer through a polyline of waypoints. Full reference: [`docs/source/Eng/doc/new_features/v116_features_doc.rst`](docs/source/Eng/doc/new_features/v116_features_doc.rst). diff --git a/docs/source/Eng/doc/new_features/v117_features_doc.rst b/docs/source/Eng/doc/new_features/v117_features_doc.rst new file mode 100644 index 00000000..e36f20c5 --- /dev/null +++ b/docs/source/Eng/doc/new_features/v117_features_doc.rst @@ -0,0 +1,43 @@ +Clear-Then-Type Field Entry +=========================== + +Setting a field's value reliably means *clearing* whatever is there first, then +entering the new text — otherwise automation appends to or corrupts the existing +content. The framework has ``write`` (types, but raises on emoji / CJK / chars +outside the layout table) and ``set_clipboard`` / ``hotkey`` separately, but no +single "focus → clear → set value" primitive and no paste strategy for text that +``write`` cannot type. This adds the Playwright ``fill`` idiom. + +:func:`plan_field_set` builds the deterministic op-plan (pure, unit-testable); +:func:`set_field_text` dispatches it through an injectable ``sink`` so it is +tested without real input. Imports no ``PySide6``. + +Headless API +------------ + +.. code-block:: python + + from je_auto_control import set_field_text, plan_field_set + + set_field_text("new value") # select-all, delete, type + set_field_text("café 🚀", paste=True) # via clipboard (Unicode-safe) + set_field_text("appended", clear="none") # no clear, just type + set_field_text("値", paste=True, modifier="command") # macOS + + plan_field_set("hi") + # [{'op': 'hotkey', 'keys': ['ctrl', 'a']}, + # {'op': 'key', 'key': 'delete'}, + # {'op': 'type', 'text': 'hi'}] + +``clear`` is ``"select_all"`` (the ``modifier``+A then Delete clear) or +``"none"``. ``paste=True`` enters the text through the clipboard (``modifier``+V) +— the reliable path for Unicode / emoji / CJK that ``write`` cannot type — rather +than typing key by key. ``modifier`` is the platform command key (``"ctrl"``; use +``"command"`` on macOS). An unknown ``clear`` mode raises ``ValueError``. + +Executor commands +----------------- + +``AC_set_field_text`` takes ``text`` plus ``clear`` / ``paste`` / ``modifier`` and +returns ``{ops, plan}``. It is exposed as the MCP tool ``ac_set_field_text`` and +as a Script Builder command under **Keyboard**. diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst index 6e8a2e24..13d9a99c 100644 --- a/docs/source/Eng/eng_index.rst +++ b/docs/source/Eng/eng_index.rst @@ -139,6 +139,7 @@ Comprehensive guides for all AutoControl features. doc/new_features/v114_features_doc doc/new_features/v115_features_doc doc/new_features/v116_features_doc + doc/new_features/v117_features_doc doc/ocr_backends/ocr_backends_doc doc/observability/observability_doc doc/operations_layer/operations_layer_doc diff --git a/docs/source/Zh/doc/new_features/v117_features_doc.rst b/docs/source/Zh/doc/new_features/v117_features_doc.rst new file mode 100644 index 00000000..4c527bda --- /dev/null +++ b/docs/source/Zh/doc/new_features/v117_features_doc.rst @@ -0,0 +1,37 @@ +清空再輸入欄位 +============== + +可靠地設定欄位值,必須先*清空*既有內容再輸入新文字——否則自動化會附加到或破壞既有內容。框架分別有 ``write`` +(可輸入,但對 emoji / CJK / 不在版面表內的字元會拋例外)與 ``set_clipboard`` / ``hotkey``,但沒有單一的 +「聚焦 → 清空 → 設值」基本元件,也沒有可輸入 ``write`` 無法處理之文字的貼上策略。本功能加入 Playwright 的 +``fill`` 慣用法。 + +:func:`plan_field_set` 建立決定性的操作計畫(純函式、可單元測試);:func:`set_field_text` 透過可注入的 ``sink`` +派發,因此可在無真實輸入下測試。不匯入 ``PySide6``。 + +無頭 API +-------- + +.. code-block:: python + + from je_auto_control import set_field_text, plan_field_set + + set_field_text("new value") # 全選、刪除、輸入 + set_field_text("café 🚀", paste=True) # 透過剪貼簿(Unicode 安全) + set_field_text("appended", clear="none") # 不清空,直接輸入 + set_field_text("値", paste=True, modifier="command") # macOS + + plan_field_set("hi") + # [{'op': 'hotkey', 'keys': ['ctrl', 'a']}, + # {'op': 'key', 'key': 'delete'}, + # {'op': 'type', 'text': 'hi'}] + +``clear`` 為 ``"select_all"``(``modifier``+A 再 Delete 的清空)或 ``"none"``。``paste=True`` 透過剪貼簿 +(``modifier``+V)輸入文字——這是 ``write`` 無法輸入之 Unicode / emoji / CJK 的可靠途徑——而非逐鍵輸入。 +``modifier`` 為平台指令鍵(``"ctrl"``;macOS 用 ``"command"``)。未知的 ``clear`` 模式會拋出 ``ValueError``。 + +執行器命令 +---------- + +``AC_set_field_text`` 接受 ``text`` 以及 ``clear`` / ``paste`` / ``modifier``,並回傳 ``{ops, plan}``。它以 +MCP 工具 ``ac_set_field_text`` 以及 Script Builder 中 **Keyboard** 分類下的命令提供。 diff --git a/docs/source/Zh/zh_index.rst b/docs/source/Zh/zh_index.rst index a69899e9..705a2ca8 100644 --- a/docs/source/Zh/zh_index.rst +++ b/docs/source/Zh/zh_index.rst @@ -139,6 +139,7 @@ AutoControl 所有功能的完整使用指南。 doc/new_features/v114_features_doc doc/new_features/v115_features_doc doc/new_features/v116_features_doc + doc/new_features/v117_features_doc doc/ocr_backends/ocr_backends_doc doc/observability/observability_doc doc/operations_layer/operations_layer_doc diff --git a/je_auto_control/__init__.py b/je_auto_control/__init__.py index 38de2b16..3b5276d3 100644 --- a/je_auto_control/__init__.py +++ b/je_auto_control/__init__.py @@ -255,6 +255,8 @@ from je_auto_control.utils.mouse_path import ( drag_path, move_along_path, path_easings, plan_path, ) +# Clear-then-type a text field (Playwright `fill` idiom; paste for Unicode) +from je_auto_control.utils.field_entry import plan_field_set, set_field_text # CI workflow annotations (GitHub Actions) from je_auto_control.utils.ci_annotations import ( emit_annotations, format_annotation, @@ -1029,6 +1031,8 @@ def start_autocontrol_gui(*args, **kwargs): "move_along_path", "drag_path", "path_easings", + "plan_field_set", + "set_field_text", "emit_annotations", "format_annotation", "ClipboardHistory", "default_clipboard_history", "analyze_heal_log", "heal_stats", "scan_secrets", diff --git a/je_auto_control/gui/script_builder/command_schema.py b/je_auto_control/gui/script_builder/command_schema.py index 77c687f4..852d35a2 100644 --- a/je_auto_control/gui/script_builder/command_schema.py +++ b/je_auto_control/gui/script_builder/command_schema.py @@ -162,6 +162,18 @@ def _add_keyboard_specs(specs: List[CommandSpec]) -> None: ), description="Type text with randomized per-key delays.", )) + specs.append(CommandSpec( + "AC_set_field_text", "Keyboard", "Set Field Text", + fields=( + FieldSpec("text", FieldType.STRING, placeholder="new value"), + FieldSpec("clear", FieldType.ENUM, choices=("select_all", "none"), + optional=True, default="select_all"), + FieldSpec("paste", FieldType.BOOL, optional=True, default=False), + FieldSpec("modifier", FieldType.STRING, optional=True, + default="ctrl", placeholder="ctrl | command"), + ), + description="Clear the focused field then enter text (paste for Unicode).", + )) specs.append(CommandSpec( "AC_hotkey", "Keyboard", "Hotkey", fields=( diff --git a/je_auto_control/utils/executor/action_executor.py b/je_auto_control/utils/executor/action_executor.py index 1c88529c..87b81248 100644 --- a/je_auto_control/utils/executor/action_executor.py +++ b/je_auto_control/utils/executor/action_executor.py @@ -3092,6 +3092,14 @@ def _drag_path(waypoints: Any, button: str = "mouse_left", per_segment_steps=int(per_segment_steps)) +def _set_field_text(text: str, clear: str = "select_all", paste: Any = False, + modifier: str = "ctrl") -> Dict[str, Any]: + """Adapter: clear the focused field and enter text.""" + from je_auto_control.utils.field_entry import set_field_text + return set_field_text(text, clear=clear, paste=bool(paste), + modifier=modifier) + + def _cas_put(name: str, key: str, value: Any, expected_version: Any = None) -> Dict[str, Any]: """Adapter: optimistic put into a named versioned store.""" @@ -4789,6 +4797,7 @@ def __init__(self): "AC_checksum_digit": _checksum_digit, "AC_move_along_path": _move_along_path, "AC_drag_path": _drag_path, + "AC_set_field_text": _set_field_text, "AC_detect_drift": _detect_drift, "AC_categorical_drift": _categorical_drift, "AC_diff_rows": _diff_rows, diff --git a/je_auto_control/utils/field_entry/__init__.py b/je_auto_control/utils/field_entry/__init__.py new file mode 100644 index 00000000..be31039e --- /dev/null +++ b/je_auto_control/utils/field_entry/__init__.py @@ -0,0 +1,6 @@ +"""Clear-then-type a text field (the Playwright ``fill`` idiom).""" +from je_auto_control.utils.field_entry.field_entry import ( + plan_field_set, set_field_text, +) + +__all__ = ["plan_field_set", "set_field_text"] diff --git a/je_auto_control/utils/field_entry/field_entry.py b/je_auto_control/utils/field_entry/field_entry.py new file mode 100644 index 00000000..991c30d4 --- /dev/null +++ b/je_auto_control/utils/field_entry/field_entry.py @@ -0,0 +1,70 @@ +"""Clear-then-type a text field (the Playwright ``fill`` idiom). + +Setting a field's value reliably means *clearing* whatever is there first, then +entering the new text — otherwise automation appends to or corrupts the existing +content. The framework has ``write`` (types, but raises on emoji / CJK / chars +outside the layout table) and ``set_clipboard`` / ``hotkey`` separately, but no +single "focus → clear → set value" primitive, and no paste strategy for text +``write`` cannot type. + +:func:`plan_field_set` builds the deterministic op-plan (pure, unit-testable); +:func:`set_field_text` dispatches it through an injectable ``sink`` so it is +tested without real input. Imports no ``PySide6``. +""" +from typing import Any, Callable, Dict, List, Optional + +_CLEAR_MODES = ("select_all", "none") +Sink = Callable[[Dict[str, Any]], None] + + +def plan_field_set(text: str, *, clear: str = "select_all", paste: bool = False, + modifier: str = "ctrl") -> List[Dict[str, Any]]: + """Return the op-plan to set a focused field to ``text``. + + ``clear`` is ``"select_all"`` (``modifier``+A then Delete) or ``"none"``. + ``paste=True`` enters the text via the clipboard (``modifier``+V) — the + reliable path for Unicode / emoji / CJK that ``write`` cannot type — instead + of typing it key by key. ``modifier`` is the platform command key + (``"ctrl"``; use ``"command"`` on macOS). Raises ``ValueError`` on an unknown + ``clear`` mode. + """ + if clear not in _CLEAR_MODES: + raise ValueError(f"unknown clear mode: {clear!r}") + plan: List[Dict[str, Any]] = [] + if clear == "select_all": + plan.append({"op": "hotkey", "keys": [modifier, "a"]}) + plan.append({"op": "key", "key": "delete"}) + if paste: + plan.append({"op": "set_clipboard", "text": text}) + plan.append({"op": "hotkey", "keys": [modifier, "v"]}) + else: + plan.append({"op": "type", "text": text}) + return plan + + +def _default_sink(event: Dict[str, Any]) -> None: + """Default dispatch: drive the real keyboard / clipboard backend.""" + op = event["op"] + if op == "hotkey": + from je_auto_control.wrapper.auto_control_keyboard import hotkey + hotkey(list(event["keys"])) + elif op == "key": + from je_auto_control.wrapper.auto_control_keyboard import type_keyboard + type_keyboard(event["key"]) + elif op == "type": + from je_auto_control.wrapper.auto_control_keyboard import write + write(event["text"]) + elif op == "set_clipboard": + from je_auto_control.utils.clipboard.clipboard import set_clipboard + set_clipboard(event["text"]) + + +def set_field_text(text: str, *, clear: str = "select_all", paste: bool = False, + modifier: str = "ctrl", + sink: Optional[Sink] = None) -> Dict[str, Any]: + """Clear the focused field and enter ``text``; return the dispatched plan.""" + plan = plan_field_set(text, clear=clear, paste=paste, modifier=modifier) + dispatch = sink or _default_sink + for event in plan: + dispatch(event) + return {"ops": len(plan), "plan": plan} diff --git a/je_auto_control/utils/mcp_server/tools/_factories.py b/je_auto_control/utils/mcp_server/tools/_factories.py index cace7ba2..32a2f6d8 100644 --- a/je_auto_control/utils/mcp_server/tools/_factories.py +++ b/je_auto_control/utils/mcp_server/tools/_factories.py @@ -2550,6 +2550,24 @@ def tween_drag_tools() -> List[MCPTool]: ] +def field_entry_tools() -> List[MCPTool]: + return [ + MCPTool( + name="ac_set_field_text", + description=("Clear the focused field and enter 'text' (Playwright " + "fill). 'clear' select_all|none; 'paste' True for " + "Unicode/emoji via clipboard; 'modifier' ctrl|command. " + "Returns {ops, plan}."), + input_schema=schema({ + "text": {"type": "string"}, "clear": {"type": "string"}, + "paste": {"type": "boolean"}, "modifier": {"type": "string"}}, + required=["text"]), + handler=h.set_field_text, + annotations=SIDE_EFFECT_ONLY, + ), + ] + + def mouse_path_tools() -> List[MCPTool]: return [ MCPTool( @@ -5831,8 +5849,8 @@ def media_assert_tools() -> List[MCPTool]: checkpoint_tools, set_of_marks_tools, screen_state_tools, input_macro_tools, resilience_tools, ci_annotation_tools, clipboard_history_tools, audit_analysis_tools, - process_doc_tools, tween_drag_tools, mouse_path_tools, plugin_sdk_tools, - governance_tools, + process_doc_tools, tween_drag_tools, mouse_path_tools, field_entry_tools, + plugin_sdk_tools, governance_tools, credential_lease_tools, egress_tools, approval_testing_tools, trajectory_eval_tools, compliance_tools, agent_trace_tools, video_report_tools, fuzzy_tools, artifact_store_tools, image_dedup_tools, diff --git a/je_auto_control/utils/mcp_server/tools/_handlers.py b/je_auto_control/utils/mcp_server/tools/_handlers.py index 551a8bdf..9be960c7 100644 --- a/je_auto_control/utils/mcp_server/tools/_handlers.py +++ b/je_auto_control/utils/mcp_server/tools/_handlers.py @@ -2038,6 +2038,11 @@ def drag_path(waypoints, button="mouse_left", easing="linear", return _drag_path(waypoints, button, easing, per_segment_steps) +def set_field_text(text, clear="select_all", paste=False, modifier="ctrl"): + from je_auto_control.utils.executor.action_executor import _set_field_text + return _set_field_text(text, clear, paste, modifier) + + def detect_drift(reference, current, threshold=0.25, bins=10): from je_auto_control.utils.executor.action_executor import _detect_drift return _detect_drift(reference, current, threshold, bins) diff --git a/test/unit_test/headless/test_field_entry_batch.py b/test/unit_test/headless/test_field_entry_batch.py new file mode 100644 index 00000000..881b4786 --- /dev/null +++ b/test/unit_test/headless/test_field_entry_batch.py @@ -0,0 +1,71 @@ +"""Headless tests for clear-then-type field entry. No Qt.""" +import je_auto_control as ac +from je_auto_control.utils.field_entry import plan_field_set, set_field_text + + +def test_default_plan_clears_then_types(): + plan = plan_field_set("hello") + assert plan == [ + {"op": "hotkey", "keys": ["ctrl", "a"]}, + {"op": "key", "key": "delete"}, + {"op": "type", "text": "hello"}, + ] + + +def test_paste_plan_uses_clipboard(): + plan = plan_field_set("café 🚀", paste=True) + ops = [event["op"] for event in plan] + assert ops == ["hotkey", "key", "set_clipboard", "hotkey"] + assert plan[2] == {"op": "set_clipboard", "text": "café 🚀"} + assert plan[-1]["keys"] == ["ctrl", "v"] + + +def test_clear_none_skips_clear(): + assert plan_field_set("x", clear="none") == [{"op": "type", "text": "x"}] + + +def test_modifier_override_for_macos(): + plan = plan_field_set("y", modifier="command") + assert plan[0]["keys"] == ["command", "a"] + + +def test_unknown_clear_raises(): + try: + plan_field_set("z", clear="bogus") + except ValueError as error: + assert "clear" in str(error) + else: # pragma: no cover + raise AssertionError("expected ValueError") + + +def test_set_field_text_dispatches_plan(): + events = [] + result = set_field_text("hi", sink=events.append) + assert result["ops"] == 3 + assert [e["op"] for e in events] == ["hotkey", "key", "type"] + + +# --- wiring --------------------------------------------------------------- + +def test_executor_adapter_planning(): + # the executor's default dispatch is device-bound; exercise the adapter + # planning + JSON-free path by calling the underlying with an injected sink. + events = [] + set_field_text("v", paste=True, sink=events.append) + assert events[-1] == {"op": "hotkey", "keys": ["ctrl", "v"]} + + +def test_wiring(): + known = ac.executor.known_commands() + assert "AC_set_field_text" in set(known) + from je_auto_control.utils.mcp_server.tools import build_default_tool_registry + names = {t.name for t in build_default_tool_registry()} + assert "ac_set_field_text" in names + from je_auto_control.gui.script_builder.command_schema import _build_specs + specs = {s.command for s in _build_specs()} + assert "AC_set_field_text" in specs + + +def test_facade_exports(): + for attr in ("plan_field_set", "set_field_text"): + assert hasattr(ac, attr) and attr in ac.__all__