Add multi-waypoint mouse gestures (move/drag through a polyline)

README/WHATS_NEW_zh-CN.md

@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-22) — 多路径点鼠标手势
+
+让指针沿着路径点折线移动或拖曳。完整参考:[`docs/source/Zh/doc/new_features/v116_features_doc.rst`](../docs/source/Zh/doc/new_features/v116_features_doc.rst)。
+
+- **`plan_path` / `move_along_path` / `drag_path` / `path_easings`**(`AC_move_along_path`、`AC_drag_path`):`humanize` 与 `tween_drag` 只在单一起点→终点之间插值——先前无法驱动任意的路径点链(签名、框选、多停靠点拖曳)并在整段路径中按住按键。`plan_path` 为纯缓动点运算(重用 `tween_drag` 的缓动、交接点去重);移动/拖曳通过可注入的 sink 派发以供无头测试。纯标准库、确定。
+
 ## 本次更新 (2026-06-22) — 校验位算法
 
 计算/验证 Luhn、Verhoeff、Damm 与 ISO 7064 MOD 97-10 校验位。完整参考:[`docs/source/Zh/doc/new_features/v115_features_doc.rst`](../docs/source/Zh/doc/new_features/v115_features_doc.rst)。

README/WHATS_NEW_zh-TW.md

@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-22) — 多路徑點滑鼠手勢
+
+讓指標沿著路徑點折線移動或拖曳。完整參考:[`docs/source/Zh/doc/new_features/v116_features_doc.rst`](../docs/source/Zh/doc/new_features/v116_features_doc.rst)。
+
+- **`plan_path` / `move_along_path` / `drag_path` / `path_easings`**(`AC_move_along_path`、`AC_drag_path`):`humanize` 與 `tween_drag` 只在單一起點→終點之間插值——先前無法驅動任意的路徑點鏈(簽名、框選、多停靠點拖曳)並在整段路徑中按住按鍵。`plan_path` 為純緩動點運算(重用 `tween_drag` 的緩動、交接點去重);移動/拖曳透過可注入的 sink 派發以供無頭測試。純標準函式庫、具決定性。
+
 ## 本次更新 (2026-06-22) — 檢查碼演算法
 
 計算/驗證 Luhn、Verhoeff、Damm 與 ISO 7064 MOD 97-10 檢查碼。完整參考:[`docs/source/Zh/doc/new_features/v115_features_doc.rst`](../docs/source/Zh/doc/new_features/v115_features_doc.rst)。

WHATS_NEW.md

@@ -1,5 +1,11 @@
 # What's New — AutoControl
 
+## What's new (2026-06-22) — Multi-Waypoint Mouse Gestures
+
+Move or drag the pointer through a polyline of waypoints. Full reference: [`docs/source/Eng/doc/new_features/v116_features_doc.rst`](docs/source/Eng/doc/new_features/v116_features_doc.rst).
+
+- **`plan_path` / `move_along_path` / `drag_path` / `path_easings`** (`AC_move_along_path`, `AC_drag_path`): `humanize` and `tween_drag` only interpolate a single start→end hop — there was no way to drive an arbitrary chain of waypoints (signatures, marquee selects, multi-stop drags) with the button held across the whole path. `plan_path` is pure eased point math (reusing `tween_drag`'s easings, junctions de-duplicated); the move/drag dispatch through an injectable sink for headless testing. Pure-stdlib, deterministic.
+
 ## What's new (2026-06-22) — Check-Digit Algorithms
 
 Compute / verify Luhn, Verhoeff, Damm and ISO 7064 MOD 97-10 check digits. Full reference: [`docs/source/Eng/doc/new_features/v115_features_doc.rst`](docs/source/Eng/doc/new_features/v115_features_doc.rst).

docs/source/Eng/doc/new_features/v116_features_doc.rst

@@ -0,0 +1,43 @@
+Multi-Waypoint Mouse Gestures
+=============================
+
+``humanize.humanized_path`` and ``tween_drag`` only interpolate a *single*
+start → end hop. Real gestures — signatures, marquee / rubber-band selections,
+dragging through several drop targets, shape gestures — need an arbitrary chain
+of waypoints, with the button optionally held down across the whole path.
+
+:func:`plan_path` is pure point math (reusing the named easings from
+``tween_drag``) and is unit-testable on its own; :func:`move_along_path` and
+:func:`drag_path` dispatch through an injectable ``sink`` so the gesture is
+tested without real input. Imports no ``PySide6``.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import (
+        plan_path, move_along_path, drag_path, path_easings,
+    )
+
+    # the eased point list through every waypoint (junctions de-duplicated)
+    plan_path([(100, 100), (400, 150), (400, 500)], per_segment_steps=20)
+
+    move_along_path([(100, 100), (400, 150), (400, 500)])      # hover a polyline
+    drag_path([(50, 50), (300, 50), (300, 300)], button="mouse_left")  # L-drag
+
+``plan_path`` interpolates each consecutive pair with ``per_segment_steps`` eased
+steps (``easing`` is any name from ``path_easings()`` — ``linear`` /
+``ease_in_out_quad`` / ``ease_out_cubic`` / ``ease_in_cubic``) and does not
+duplicate the shared junction points. ``move_along_path`` emits move events
+through the path; ``drag_path`` presses at the first waypoint, moves through the
+whole path, and releases at the last — for multi-stop drags. Both take a ``sink``
+override for headless testing.
+
+Executor commands
+-----------------
+
+``AC_move_along_path`` and ``AC_drag_path`` take ``waypoints`` (a JSON
+``[[x, y], ...]`` list) plus ``easing`` / ``per_segment_steps`` (and ``button``
+for the drag). Both are exposed as MCP tools (``ac_move_along_path`` /
+``ac_drag_path``) and as Script Builder commands under **Mouse**.

docs/source/Eng/eng_index.rst

@@ -138,6 +138,7 @@ Comprehensive guides for all AutoControl features.
    doc/new_features/v113_features_doc
    doc/new_features/v114_features_doc
    doc/new_features/v115_features_doc
+   doc/new_features/v116_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

docs/source/Zh/doc/new_features/v116_features_doc.rst

@@ -0,0 +1,35 @@
+多路徑點滑鼠手勢
+================
+
+``humanize.humanized_path`` 與 ``tween_drag`` 只在*單一*起點 → 終點之間插值。真實手勢——簽名、框選
+(marquee / rubber-band)、拖曳經過多個放置目標、形狀手勢——需要任意的路徑點鏈,且可在整段路徑中持續按住按鍵。
+
+:func:`plan_path` 為純點運算(重用 ``tween_drag`` 的具名緩動),本身即可單元測試;:func:`move_along_path` 與
+:func:`drag_path` 透過可注入的 ``sink`` 派發,因此手勢可在無真實輸入下測試。不匯入 ``PySide6``。
+
+無頭 API
+--------
+
+.. code-block:: python
+
+    from je_auto_control import (
+        plan_path, move_along_path, drag_path, path_easings,
+    )
+
+    # 經過每個路徑點的緩動點列(交接點不重複)
+    plan_path([(100, 100), (400, 150), (400, 500)], per_segment_steps=20)
+
+    move_along_path([(100, 100), (400, 150), (400, 500)])      # 沿折線移動
+    drag_path([(50, 50), (300, 50), (300, 300)], button="mouse_left")  # L 形拖曳
+
+``plan_path`` 以 ``per_segment_steps`` 個緩動步驟對每個相鄰點對插值(``easing`` 為 ``path_easings()`` 中任一
+名稱——``linear`` / ``ease_in_out_quad`` / ``ease_out_cubic`` / ``ease_in_cubic``),且不重複共用的交接點。
+``move_along_path`` 沿路徑發出移動事件;``drag_path`` 在第一個路徑點按下、移動經過整段路徑、在最後一點放開——
+用於多停靠點拖曳。兩者皆可傳入 ``sink`` 以供無頭測試。
+
+執行器命令
+----------
+
+``AC_move_along_path`` 與 ``AC_drag_path`` 接受 ``waypoints``(JSON ``[[x, y], ...]`` 列表)以及 ``easing`` /
+``per_segment_steps``(拖曳另含 ``button``)。兩者皆以 MCP 工具(``ac_move_along_path`` / ``ac_drag_path``)
+以及 Script Builder 中 **Mouse** 分類下的命令提供。

docs/source/Zh/zh_index.rst

@@ -138,6 +138,7 @@ AutoControl 所有功能的完整使用指南。
    doc/new_features/v113_features_doc
    doc/new_features/v114_features_doc
    doc/new_features/v115_features_doc
+   doc/new_features/v116_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

je_auto_control/__init__.py

@@ -251,6 +251,10 @@
     mod97_10_check_digits, mod97_10_validate, verhoeff_check_digit,
     verhoeff_validate,
 )
+# Multi-waypoint mouse gestures (move / drag through a polyline of points)
+from je_auto_control.utils.mouse_path import (
+    drag_path, move_along_path, path_easings, plan_path,
+)
 # CI workflow annotations (GitHub Actions)
 from je_auto_control.utils.ci_annotations import (
     emit_annotations, format_annotation,
@@ -1021,6 +1025,10 @@ def start_autocontrol_gui(*args, **kwargs):
     "damm_check_digit",
     "mod97_10_validate",
     "mod97_10_check_digits",
+    "plan_path",
+    "move_along_path",
+    "drag_path",
+    "path_easings",
     "emit_annotations", "format_annotation",
     "ClipboardHistory", "default_clipboard_history",
     "analyze_heal_log", "heal_stats", "scan_secrets",

je_auto_control/gui/script_builder/command_schema.py

@@ -681,6 +681,36 @@ def _add_misc_specs(specs: List[CommandSpec]) -> None:
         description="Drag along an eased path; 'start'/'end' [x,y] via JSON "
                     "view.",
     ))
+    specs.append(CommandSpec(
+        "AC_move_along_path", "Mouse", "Move Along Path",
+        fields=(
+            FieldSpec("waypoints", FieldType.STRING,
+                      placeholder="[[100,100],[400,150],[400,500]]"),
+            FieldSpec("easing", FieldType.ENUM,
+                      choices=("linear", "ease_in_out_quad", "ease_out_cubic",
+                               "ease_in_cubic"),
+                      optional=True, default="linear"),
+            FieldSpec("per_segment_steps", FieldType.INT, optional=True,
+                      default=20),
+        ),
+        description="Move the pointer through a polyline of waypoints.",
+    ))
+    specs.append(CommandSpec(
+        "AC_drag_path", "Mouse", "Drag Along Path",
+        fields=(
+            FieldSpec("waypoints", FieldType.STRING,
+                      placeholder="[[50,50],[300,50],[300,300]]"),
+            FieldSpec("button", FieldType.ENUM, choices=_MOUSE_BUTTONS,
+                      optional=True, default="mouse_left"),
+            FieldSpec("easing", FieldType.ENUM,
+                      choices=("linear", "ease_in_out_quad", "ease_out_cubic",
+                               "ease_in_cubic"),
+                      optional=True, default="linear"),
+            FieldSpec("per_segment_steps", FieldType.INT, optional=True,
+                      default=20),
+        ),
+        description="Press, drag through a polyline of waypoints, release.",
+    ))
     specs.append(CommandSpec(
         "AC_list_plugins", "Tools", "List Plugin Commands",
         fields=(FieldSpec("group", FieldType.STRING, optional=True,

je_auto_control/utils/executor/action_executor.py

@@ -3069,6 +3069,29 @@ def _checksum_digit(scheme: str, partial: str) -> Dict[str, Any]:
     return {"check_digit": func(partial)}
 
 
+def _waypoints(value: Any) -> Any:
+    """Coerce a JSON string of waypoints into a list."""
+    import json
+    return json.loads(value) if isinstance(value, str) else value
+
+
+def _move_along_path(waypoints: Any, easing: str = "linear",
+                     per_segment_steps: Any = 20) -> Dict[str, Any]:
+    """Adapter: move the pointer through a polyline of waypoints."""
+    from je_auto_control.utils.mouse_path import move_along_path
+    return move_along_path(_waypoints(waypoints), easing=easing,
+                           per_segment_steps=int(per_segment_steps))
+
+
+def _drag_path(waypoints: Any, button: str = "mouse_left",
+               easing: str = "linear",
+               per_segment_steps: Any = 20) -> Dict[str, Any]:
+    """Adapter: press, drag through a polyline of waypoints, release."""
+    from je_auto_control.utils.mouse_path import drag_path
+    return drag_path(_waypoints(waypoints), button=button, easing=easing,
+                     per_segment_steps=int(per_segment_steps))
+
+
 def _cas_put(name: str, key: str, value: Any,
              expected_version: Any = None) -> Dict[str, Any]:
     """Adapter: optimistic put into a named versioned store."""
@@ -4764,6 +4787,8 @@ def __init__(self):
             "AC_gettext_ngettext": _gettext_ngettext,
             "AC_checksum_validate": _checksum_validate,
             "AC_checksum_digit": _checksum_digit,
+            "AC_move_along_path": _move_along_path,
+            "AC_drag_path": _drag_path,
             "AC_detect_drift": _detect_drift,
             "AC_categorical_drift": _categorical_drift,
             "AC_diff_rows": _diff_rows,

je_auto_control/utils/mcp_server/tools/_factories.py

@@ -2550,6 +2550,42 @@ def tween_drag_tools() -> List[MCPTool]:
     ]
 
 
+def mouse_path_tools() -> List[MCPTool]:
+    return [
+        MCPTool(
+            name="ac_move_along_path",
+            description=("Move the pointer through 'waypoints' ([[x,y],...]) as "
+                         "an eased polyline. 'per_segment_steps' + 'easing' "
+                         "(linear / ease_*). Returns {points, path}."),
+            input_schema=schema({
+                "waypoints": {"type": "array",
+                              "items": {"type": "array",
+                                        "items": {"type": "integer"}}},
+                "easing": {"type": "string"},
+                "per_segment_steps": {"type": "integer"}},
+                required=["waypoints"]),
+            handler=h.move_along_path,
+            annotations=SIDE_EFFECT_ONLY,
+        ),
+        MCPTool(
+            name="ac_drag_path",
+            description=("Press at the first of 'waypoints' ([[x,y],...]), drag "
+                         "through them, release at the last. 'button', 'easing', "
+                         "'per_segment_steps'. Returns {points, path}."),
+            input_schema=schema({
+                "waypoints": {"type": "array",
+                              "items": {"type": "array",
+                                        "items": {"type": "integer"}}},
+                "button": {"type": "string"},
+                "easing": {"type": "string"},
+                "per_segment_steps": {"type": "integer"}},
+                required=["waypoints"]),
+            handler=h.drag_path,
+            annotations=SIDE_EFFECT_ONLY,
+        ),
+    ]
+
+
 def plugin_sdk_tools() -> List[MCPTool]:
     _G = {"group": {"type": "string"}}
     return [
@@ -5795,7 +5831,8 @@ def media_assert_tools() -> List[MCPTool]:
     checkpoint_tools, set_of_marks_tools, screen_state_tools,
     input_macro_tools, resilience_tools,
     ci_annotation_tools, clipboard_history_tools, audit_analysis_tools,
-    process_doc_tools, tween_drag_tools, plugin_sdk_tools, governance_tools,
+    process_doc_tools, tween_drag_tools, mouse_path_tools, plugin_sdk_tools,
+    governance_tools,
     credential_lease_tools, egress_tools, approval_testing_tools,
     trajectory_eval_tools, compliance_tools, agent_trace_tools,
     video_report_tools, fuzzy_tools, artifact_store_tools, image_dedup_tools,

je_auto_control/utils/mcp_server/tools/_handlers.py

@@ -2027,6 +2027,17 @@ def checksum_digit(scheme, partial):
     return _checksum_digit(scheme, partial)
 
 
+def move_along_path(waypoints, easing="linear", per_segment_steps=20):
+    from je_auto_control.utils.executor.action_executor import _move_along_path
+    return _move_along_path(waypoints, easing, per_segment_steps)
+
+
+def drag_path(waypoints, button="mouse_left", easing="linear",
+              per_segment_steps=20):
+    from je_auto_control.utils.executor.action_executor import _drag_path
+    return _drag_path(waypoints, button, easing, per_segment_steps)
+
+
 def detect_drift(reference, current, threshold=0.25, bins=10):
     from je_auto_control.utils.executor.action_executor import _detect_drift
     return _detect_drift(reference, current, threshold, bins)

je_auto_control/utils/mouse_path/__init__.py

@@ -0,0 +1,6 @@
+"""Multi-waypoint mouse gestures (move or drag through a polyline of points)."""
+from je_auto_control.utils.mouse_path.mouse_path import (
+    drag_path, move_along_path, path_easings, plan_path,
+)
+
+__all__ = ["drag_path", "move_along_path", "path_easings", "plan_path"]

je_auto_control/utils/mouse_path/mouse_path.py

@@ -0,0 +1,86 @@
+"""Multi-waypoint mouse gestures: move or drag through a polyline of points.
+
+``humanize.humanized_path`` and ``tween_drag`` only interpolate a *single*
+start -> end hop. Real gestures — signatures, marquee/rubber-band selections,
+drag-through-multiple-drop-targets, shape gestures — need an arbitrary chain of
+waypoints with the button optionally held down across the whole path.
+
+:func:`plan_path` is pure point math (reusing the named easings from
+``tween_drag``) and is unit-testable on its own; :func:`move_along_path` and
+:func:`drag_path` dispatch through an injectable ``sink`` so the gesture is
+tested without real input. Imports no ``PySide6``.
+"""
+from typing import Any, Callable, Dict, List, Optional, Sequence
+
+from je_auto_control.utils.tween_drag.tween_drag import easing_names, tween_points
+
+Point = Sequence[int]
+Sink = Callable[[Dict[str, Any]], None]
+
+
+def plan_path(waypoints: Sequence[Point], *, easing: str = "linear",
+              per_segment_steps: int = 20) -> List[List[int]]:
+    """Return the eased point list passing through every waypoint in order.
+
+    Each consecutive pair is interpolated with ``per_segment_steps`` eased steps
+    (named easings from ``tween_drag``); shared junction points are not
+    duplicated. Fewer than two waypoints yields the points unchanged.
+    """
+    points: List[List[int]] = []
+    if not waypoints:
+        return points
+    if len(waypoints) == 1:
+        first = waypoints[0]
+        return [[int(first[0]), int(first[1])]]
+    for index in range(len(waypoints) - 1):
+        segment = tween_points(waypoints[index], waypoints[index + 1],
+                               per_segment_steps, easing)
+        points.extend(segment[1:] if index else segment)
+    return points
+
+
+def _default_sink(event: Dict[str, Any]) -> None:
+    """Default dispatch: drive the real mouse backend."""
+    from je_auto_control.wrapper.auto_control_mouse import (
+        press_mouse, release_mouse, set_mouse_position)
+    x, y = int(event["x"]), int(event["y"])
+    set_mouse_position(x, y)
+    op = event["op"]
+    if op == "press":
+        press_mouse(event.get("button", "mouse_left"), x, y)
+    elif op == "release":
+        release_mouse(event.get("button", "mouse_left"), x, y)
+
+
+def move_along_path(waypoints: Sequence[Point], *, easing: str = "linear",
+                    per_segment_steps: int = 20,
+                    sink: Optional[Sink] = None) -> Dict[str, Any]:
+    """Move the pointer through ``waypoints`` (no button held)."""
+    points = plan_path(waypoints, easing=easing,
+                       per_segment_steps=per_segment_steps)
+    dispatch = sink or _default_sink
+    for x, y in points:
+        dispatch({"op": "move", "x": x, "y": y})
+    return {"points": len(points), "path": points}
+
+
+def drag_path(waypoints: Sequence[Point], *, button: str = "mouse_left",
+              easing: str = "linear", per_segment_steps: int = 20,
+              sink: Optional[Sink] = None) -> Dict[str, Any]:
+    """Press at the first waypoint, move through the path, release at the last."""
+    points = plan_path(waypoints, easing=easing,
+                       per_segment_steps=per_segment_steps)
+    if not points:
+        return {"points": 0, "path": points}
+    dispatch = sink or _default_sink
+    first, last = points[0], points[-1]
+    dispatch({"op": "press", "button": button, "x": first[0], "y": first[1]})
+    for x, y in points:
+        dispatch({"op": "move", "x": x, "y": y})
+    dispatch({"op": "release", "button": button, "x": last[0], "y": last[1]})
+    return {"points": len(points), "path": points}
+
+
+def path_easings() -> List[str]:
+    """Return the available easing names (shared with ``tween_drag``)."""
+    return easing_names()