Add control_patterns: extended UIA patterns (Expand/Select/Range/Scroll)

README/WHATS_NEW_zh-CN.md

@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-24) — 扩充 UIA 控制模式(展开 / 选取 / 范围 / 滚动)
+
+以原生模式驱动树节点、列表/下拉项目、滑块与滚动,而非像素猜测。完整参考:[`docs/source/Zh/doc/new_features/v181_features_doc.rst`](../docs/source/Zh/doc/new_features/v181_features_doc.rst)。
+
+- **`expand_control` / `collapse_control` / `control_expand_state` / `select_control_item` / `control_range` / `set_control_range` / `scroll_control_into_view`**(`AC_expand_control`、`AC_select_control_item`、`AC_set_control_range` 等):无障碍后端原本只有 Value/Invoke/Toggle/Grid-read 模式,故树状视图、列表/下拉、滑块与屏幕外行都没有原生调用路径。本功能在既有后端 ABC 之上补上 ExpandCollapse / SelectionItem / RangeValue / ScrollItem 模式,通过可注入的 `accessibility.backends.get_backend()` 接缝分派(以 fake backend 无头测试;真正 UIA 调用在 Windows 后端)。不导入 `PySide6`。
+
 ## 本次更新 (2026-06-24) — 匹配前安定门 + 命中稳定性
 
 避免在动画进行中匹配,并确认命中跨帧维持稳定。完整参考:[`docs/source/Zh/doc/new_features/v180_features_doc.rst`](../docs/source/Zh/doc/new_features/v180_features_doc.rst)。

README/WHATS_NEW_zh-TW.md

@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-24) — 擴充 UIA 控制模式(展開 / 選取 / 範圍 / 捲動)
+
+以原生模式驅動樹節點、清單/下拉項目、滑桿與捲動,而非像素猜測。完整參考:[`docs/source/Zh/doc/new_features/v181_features_doc.rst`](../docs/source/Zh/doc/new_features/v181_features_doc.rst)。
+
+- **`expand_control` / `collapse_control` / `control_expand_state` / `select_control_item` / `control_range` / `set_control_range` / `scroll_control_into_view`**(`AC_expand_control`、`AC_select_control_item`、`AC_set_control_range` 等):無障礙後端原本只有 Value/Invoke/Toggle/Grid-read 模式,故樹狀檢視、清單/下拉、滑桿與螢幕外列都沒有原生呼叫路徑。本功能在既有後端 ABC 之上補上 ExpandCollapse / SelectionItem / RangeValue / ScrollItem 模式,透過可注入的 `accessibility.backends.get_backend()` 接縫分派(以 fake backend 無頭測試;真正 UIA 呼叫在 Windows 後端)。不匯入 `PySide6`。
+
 ## 本次更新 (2026-06-24) — 比對前安定閘 + 命中穩定性
 
 避免在動畫進行中比對,並確認命中跨幀維持穩定。完整參考:[`docs/source/Zh/doc/new_features/v180_features_doc.rst`](../docs/source/Zh/doc/new_features/v180_features_doc.rst)。

WHATS_NEW.md

@@ -1,5 +1,11 @@
 # What's New — AutoControl
 
+## What's new (2026-06-24) — Extended UIA Control Patterns (Expand / Select / Range / Scroll)
+
+Drive tree nodes, list/combo items, sliders and scroll natively, not by pixel guessing. Full reference: [`docs/source/Eng/doc/new_features/v181_features_doc.rst`](docs/source/Eng/doc/new_features/v181_features_doc.rst).
+
+- **`expand_control` / `collapse_control` / `control_expand_state` / `select_control_item` / `control_range` / `set_control_range` / `scroll_control_into_view`** (`AC_expand_control`, `AC_select_control_item`, `AC_set_control_range`, …): the accessibility backend had only Value/Invoke/Toggle/Grid-read patterns, so treeviews, listboxes/combos, sliders and off-screen rows had no native call path. This adds ExpandCollapse / SelectionItem / RangeValue / ScrollItem patterns on top of the existing backend ABC, dispatched through the injectable `accessibility.backends.get_backend()` seam (headless-testable via a fake backend; real UIA calls in the Windows backend). No `PySide6`.
+
 ## What's new (2026-06-24) — Pre-Match Settle Gating + Match Persistence
 
 Avoid matching mid-animation, and confirm a hit holds steady across frames. Full reference: [`docs/source/Eng/doc/new_features/v180_features_doc.rst`](docs/source/Eng/doc/new_features/v180_features_doc.rst).

docs/source/Eng/doc/new_features/v181_features_doc.rst

@@ -0,0 +1,46 @@
+Extended UIA Control Patterns (Expand / Select / Range / Scroll)
+===============================================================
+
+The accessibility backend shipped only four control patterns — Value, Invoke, Toggle and a
+read-only Grid dump. That left the controls automation hits most often undriveable by their
+*native* pattern: a treeview node could not be expanded, a listbox / combobox item could not be
+selected (SelectionItemPattern), a slider could not be set (RangeValuePattern), and a control
+could not be scrolled into view (ScrollItemPattern) — those fell back to fragile pixel guessing.
+``control_patterns`` adds those object-level actions on top of the existing accessibility
+backend ABC.
+
+Each function is a thin dispatch onto the injectable ``accessibility.backends.get_backend()``
+seam (the same seam the rest of the accessibility module uses), so the headless core is
+unit-testable on any platform by injecting a fake backend; the real UI Automation calls live in
+the Windows backend (ExpandCollapse / SelectionItem / RangeValue / ScrollItem patterns).
+Backends that don't implement a pattern raise ``AccessibilityNotAvailableError``. Imports no
+``PySide6``.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import (expand_control, collapse_control,
+                                 control_expand_state, select_control_item,
+                                 control_range, set_control_range,
+                                 scroll_control_into_view)
+
+    expand_control(name="Documents", role="treeitem")     # open a tree node
+    select_control_item(name="Option B")                  # pick a list/combo item
+    set_control_range(75, name="Volume")                  # set a slider
+    print(control_range(name="Volume"))   # {"value": 75.0, "minimum": 0, "maximum": 100}
+    scroll_control_into_view(name="Row 200")              # bring a row on-screen
+
+All locate the control by ``name`` / ``role`` / ``app_name`` / ``automation_id`` (same as the
+existing ``control_invoke`` / ``control_toggle``). The expand/select/scroll/set actions return
+``bool``; ``control_expand_state`` returns ``expanded`` / ``collapsed`` / ``partial`` / ``leaf``
+(or ``None``); ``control_range`` returns ``{value, minimum, maximum}`` (or ``None``).
+
+Executor commands
+-----------------
+
+``AC_expand_control`` / ``AC_collapse_control`` / ``AC_control_expand_state`` /
+``AC_select_control_item`` / ``AC_control_range`` / ``AC_set_control_range`` /
+``AC_scroll_control_into_view``. They are exposed as the matching ``ac_*`` MCP tools (the action
+ones destructive, the reads read-only) and as Script Builder commands under **Native UI**.

docs/source/Eng/eng_index.rst

@@ -203,6 +203,7 @@ Comprehensive guides for all AutoControl features.
    doc/new_features/v178_features_doc
    doc/new_features/v179_features_doc
    doc/new_features/v180_features_doc
+   doc/new_features/v181_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

docs/source/Zh/doc/new_features/v181_features_doc.rst

@@ -0,0 +1,42 @@
+擴充 UIA 控制模式(展開 / 選取 / 範圍 / 捲動)
+===============================================
+
+無障礙後端原本只提供四種控制模式——Value、Invoke、Toggle 與唯讀的 Grid dump。這使得自動化
+最常遇到的控制項無法以其*原生*模式驅動:樹節點無法展開、清單 / 下拉項目無法選取
+(SelectionItemPattern)、滑桿無法設定(RangeValuePattern)、控制項無法捲入視野
+(ScrollItemPattern)——這些只能退回脆弱的像素猜測。``control_patterns`` 在既有的無障礙後端
+ABC 之上補上這些物件層級動作。
+
+每個函式都是對可注入的 ``accessibility.backends.get_backend()`` 接縫的薄分派(與無障礙模組
+其餘部分相同的接縫),因此無頭核心可在任何平台透過注入 fake backend 單元測試;真正的
+UI Automation 呼叫位於 Windows 後端(ExpandCollapse / SelectionItem / RangeValue / ScrollItem
+模式)。未實作某模式的後端會拋出 ``AccessibilityNotAvailableError``。不匯入 ``PySide6``。
+
+無頭 API
+--------
+
+.. code-block:: python
+
+    from je_auto_control import (expand_control, collapse_control,
+                                 control_expand_state, select_control_item,
+                                 control_range, set_control_range,
+                                 scroll_control_into_view)
+
+    expand_control(name="Documents", role="treeitem")     # 展開樹節點
+    select_control_item(name="Option B")                  # 選取清單/下拉項目
+    set_control_range(75, name="Volume")                  # 設定滑桿
+    print(control_range(name="Volume"))   # {"value": 75.0, "minimum": 0, "maximum": 100}
+    scroll_control_into_view(name="Row 200")              # 把某列帶上螢幕
+
+全部以 ``name`` / ``role`` / ``app_name`` / ``automation_id`` 定位控制項(與既有
+``control_invoke`` / ``control_toggle`` 相同)。展開/選取/捲動/設定動作回傳 ``bool``;
+``control_expand_state`` 回傳 ``expanded`` / ``collapsed`` / ``partial`` / ``leaf``(或
+``None``);``control_range`` 回傳 ``{value, minimum, maximum}``(或 ``None``)。
+
+執行器指令
+----------
+
+``AC_expand_control`` / ``AC_collapse_control`` / ``AC_control_expand_state`` /
+``AC_select_control_item`` / ``AC_control_range`` / ``AC_set_control_range`` /
+``AC_scroll_control_into_view``。皆以對應的 ``ac_*`` MCP 工具(動作類為破壞性、讀取類為唯讀)
+及 Script Builder 指令(位於 **Native UI** 分類下)形式提供。

docs/source/Zh/zh_index.rst

@@ -203,6 +203,7 @@ AutoControl 所有功能的完整使用指南。
    doc/new_features/v178_features_doc
    doc/new_features/v179_features_doc
    doc/new_features/v180_features_doc
+   doc/new_features/v181_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

je_auto_control/__init__.py

@@ -48,6 +48,11 @@
     find_accessibility_element, list_accessibility_elements,
     read_control_table,
 )
+# Extended UIA control patterns (Expand / Select / Range / Scroll)
+from je_auto_control.utils.control_patterns import (
+    collapse_control, control_expand_state, control_range, expand_control,
+    scroll_control_into_view, select_control_item, set_control_range,
+)
 # VLM element locator (headless)
 from je_auto_control.utils.vision import (
     VLMNotAvailableError, click_by_description, locate_by_description,
@@ -1609,6 +1614,9 @@ def start_autocontrol_gui(*args, **kwargs):
     "find_accessibility_element", "list_accessibility_elements",
     "control_get_value", "control_set_value", "control_invoke",
     "control_toggle", "read_control_table",
+    "expand_control", "collapse_control", "control_expand_state",
+    "select_control_item", "control_range", "set_control_range",
+    "scroll_control_into_view",
     # VLM locator
     "VLMNotAvailableError", "locate_by_description", "click_by_description",
     "verify_description",

je_auto_control/gui/script_builder/command_schema.py

@@ -1488,6 +1488,41 @@ def _add_native_control_specs(specs: List[CommandSpec]) -> None:
         fields=fields,
         description="Read a grid/table/list control as rows of cell strings.",
     ))
+    specs.append(CommandSpec(
+        "AC_expand_control", "Native UI", "Expand Control",
+        fields=fields,
+        description="Expand a tree node / combobox (ExpandCollapsePattern).",
+    ))
+    specs.append(CommandSpec(
+        "AC_collapse_control", "Native UI", "Collapse Control",
+        fields=fields,
+        description="Collapse a tree node / combobox (ExpandCollapsePattern).",
+    ))
+    specs.append(CommandSpec(
+        "AC_control_expand_state", "Native UI", "Control Expand State",
+        fields=fields,
+        description="Read expanded/collapsed/partial/leaf state of a control.",
+    ))
+    specs.append(CommandSpec(
+        "AC_select_control_item", "Native UI", "Select Control Item",
+        fields=fields,
+        description="Select a list / tree / tab item (SelectionItemPattern).",
+    ))
+    specs.append(CommandSpec(
+        "AC_control_range", "Native UI", "Get Control Range",
+        fields=fields,
+        description="Read a slider / progress range (RangeValuePattern).",
+    ))
+    specs.append(CommandSpec(
+        "AC_set_control_range", "Native UI", "Set Control Range",
+        fields=(FieldSpec("value", FieldType.FLOAT),) + fields,
+        description="Set a slider / progress / spinner value (RangeValuePattern).",
+    ))
+    specs.append(CommandSpec(
+        "AC_scroll_control_into_view", "Native UI", "Scroll Control Into View",
+        fields=fields,
+        description="Scroll a control into view (ScrollItemPattern).",
+    ))
 
 
 def _add_misc_specs(specs: List[CommandSpec]) -> None:

je_auto_control/utils/accessibility/backends/base.py

@@ -1,5 +1,5 @@
 """Abstract accessibility backend."""
-from typing import List, Optional
+from typing import Any, Dict, List, Optional
 
 from je_auto_control.utils.accessibility.element import (
     AccessibilityElement, AccessibilityNotAvailableError,
@@ -56,6 +56,50 @@
         """Read a grid/table/list control as rows of cell strings."""
         self._unsupported("read_table")
 
+    # --- extended control patterns (Expand / Selection / Range / Scroll) ----
+
+    def expand(self, name: Optional[str] = None, role: Optional[str] = None,
+               app_name: Optional[str] = None,
+               automation_id: Optional[str] = None) -> bool:
+        """Expand the matched control (ExpandCollapsePattern); True on success."""
+        self._unsupported("expand")
+
+    def collapse(self, name: Optional[str] = None, role: Optional[str] = None,
+                 app_name: Optional[str] = None,
+                 automation_id: Optional[str] = None) -> bool:
+        """Collapse the matched control (ExpandCollapsePattern); True on success."""
+        self._unsupported("collapse")
+
+    def expand_state(self, name: Optional[str] = None, role: Optional[str] = None,
+                     app_name: Optional[str] = None,
+                     automation_id: Optional[str] = None) -> Optional[str]:
+        """Return ``expanded`` / ``collapsed`` / ``partial`` / ``leaf``, or None."""
+        self._unsupported("expand_state")
+
+    def select_item(self, name: Optional[str] = None, role: Optional[str] = None,
+                    app_name: Optional[str] = None,
+                    automation_id: Optional[str] = None) -> bool:
+        """Select the matched item (SelectionItemPattern); True on success."""
+        self._unsupported("select_item")
+
+    def get_range(self, name: Optional[str] = None, role: Optional[str] = None,
+                  app_name: Optional[str] = None,
+                  automation_id: Optional[str] = None) -> Optional[Dict[str, Any]]:
+        """Return ``{value, minimum, maximum}`` (RangeValuePattern), or None."""
+        self._unsupported("get_range")
+
+    def set_range_value(self, value: float, name: Optional[str] = None,
+                        role: Optional[str] = None, app_name: Optional[str] = None,
+                        automation_id: Optional[str] = None) -> bool:
+        """Set a slider / progress value (RangeValuePattern); True on success."""
+        self._unsupported("set_range_value")
+
+    def scroll_into_view(self, name: Optional[str] = None,
+                         role: Optional[str] = None, app_name: Optional[str] = None,
+                         automation_id: Optional[str] = None) -> bool:
+        """Scroll the matched control into view (ScrollItemPattern); True on success."""
+        self._unsupported("scroll_into_view")
+
     def _unsupported(self, operation: str):
         """Raise a clear error for an action this backend can't perform."""
         raise AccessibilityNotAvailableError(

je_auto_control/utils/accessibility/backends/windows_backend.py

@@ -8,7 +8,7 @@
 Only ``is_control_element=True`` nodes are surfaced to avoid millions of
 decorative text children.
 """
-from typing import List, Optional
+from typing import Any, Dict, List, Optional
 
 from je_auto_control.utils.accessibility.backends.base import (
     AccessibilityBackend,
@@ -25,6 +25,11 @@
 _UIA_INVOKE_PATTERN_ID = 10000
 _UIA_TOGGLE_PATTERN_ID = 10015
 _UIA_GRID_PATTERN_ID = 10006
+_UIA_EXPANDCOLLAPSE_PATTERN_ID = 10005
+_UIA_SELECTIONITEM_PATTERN_ID = 10010
+_UIA_RANGEVALUE_PATTERN_ID = 10003
+_UIA_SCROLLITEM_PATTERN_ID = 10017
+_EXPAND_STATES = {0: "collapsed", 1: "expanded", 2: "partial", 3: "leaf"}
 
 
 def _is_available() -> bool:
@@ -192,6 +197,72 @@ def read_table(self, name=None, role=None, app_name=None,
             return []
         return [self._read_row(pattern, r, cols) for r in range(rows)]
 
+    def _invoke_pattern_method(self, name, role, app_name, automation_id,
+                               pattern_id, interface_name, action):
+        """Find a control, query a pattern, run ``action(pattern)`` → bool."""
+        raw = self._find_raw(name, role, app_name, automation_id)
+        pattern = self._pattern(raw, pattern_id, interface_name) if raw else None
+        if pattern is None:
+            return False
+        try:
+            action(pattern)
+            return True
+        except (OSError, AttributeError):
+            return False
+
+    def expand(self, name=None, role=None, app_name=None, automation_id=None):
+        return self._invoke_pattern_method(
+            name, role, app_name, automation_id, _UIA_EXPANDCOLLAPSE_PATTERN_ID,
+            "IUIAutomationExpandCollapsePattern", lambda p: p.Expand())
+
+    def collapse(self, name=None, role=None, app_name=None, automation_id=None):
+        return self._invoke_pattern_method(
+            name, role, app_name, automation_id, _UIA_EXPANDCOLLAPSE_PATTERN_ID,
+            "IUIAutomationExpandCollapsePattern", lambda p: p.Collapse())
+
+    def expand_state(self, name=None, role=None, app_name=None,
+                     automation_id=None) -> Optional[str]:
+        raw = self._find_raw(name, role, app_name, automation_id)
+        pattern = self._pattern(raw, _UIA_EXPANDCOLLAPSE_PATTERN_ID,
+                                "IUIAutomationExpandCollapsePattern") if raw else None
+        if pattern is None:
+            return None
+        try:
+            return _EXPAND_STATES.get(int(pattern.CurrentExpandCollapseState))
+        except (OSError, AttributeError, ValueError, TypeError):
+            return None
+
+    def select_item(self, name=None, role=None, app_name=None, automation_id=None):
+        return self._invoke_pattern_method(
+            name, role, app_name, automation_id, _UIA_SELECTIONITEM_PATTERN_ID,
+            "IUIAutomationSelectionItemPattern", lambda p: p.Select())
+
+    def set_range_value(self, value, name=None, role=None, app_name=None,
+                        automation_id=None):
+        return self._invoke_pattern_method(
+            name, role, app_name, automation_id, _UIA_RANGEVALUE_PATTERN_ID,
+            "IUIAutomationRangeValuePattern", lambda p: p.SetValue(float(value)))
+
+    def scroll_into_view(self, name=None, role=None, app_name=None,
+                         automation_id=None):
+        return self._invoke_pattern_method(
+            name, role, app_name, automation_id, _UIA_SCROLLITEM_PATTERN_ID,
+            "IUIAutomationScrollItemPattern", lambda p: p.ScrollIntoView())
+
+    def get_range(self, name=None, role=None, app_name=None,
+                  automation_id=None) -> Optional[Dict[str, Any]]:
+        raw = self._find_raw(name, role, app_name, automation_id)
+        pattern = self._pattern(raw, _UIA_RANGEVALUE_PATTERN_ID,
+                                "IUIAutomationRangeValuePattern") if raw else None
+        if pattern is None:
+            return None
+        try:
+            return {"value": float(pattern.CurrentValue),
+                    "minimum": float(pattern.CurrentMinimum),
+                    "maximum": float(pattern.CurrentMaximum)}
+        except (OSError, AttributeError, ValueError, TypeError):
+            return None
+
     @staticmethod
     def _read_row(pattern, row: int, cols: int):
         """Read one grid row into a list of cell strings."""