Merge pull request #353 from Integration-Automation/feat/locator-chain-batch

JE-Chen · web-flow · commit a3ed5f8cc040 · 2026-06-23T08:35:53.000+08:00
Add composable / filtered candidate locators (chained locator)
diff --git a/README/WHATS_NEW_zh-CN.md b/README/WHATS_NEW_zh-CN.md
@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-23) — 可串接 / 可过滤的候选定位器
+
+用链式调用细化已定位的元素:`.within(panel).filter(has_text="Delete").nth(1)`。完整参考:[`docs/source/Zh/doc/new_features/v143_features_doc.rst`](../docs/source/Zh/doc/new_features/v143_features_doc.rst)。
+
+- **`from_boxes` / `Candidates`**(`AC_locate_chain`):`anchor_locator` 是单一关系、`grid_locator` 是单元格——两者都不支援对候选集合做可组合细化(Selenium-4 / Playwright 的链式定位惯用法)。本功能是对来自*任何*来源(模板 / OCR / a11y / `fuse_elements`)的框做纯后置过滤:`within`(区域裁切)、`filter`(`has_text` / `near` / 面积 / predicate)、`sort_reading`、`nth` / `first` / `last`、`resolve()` / `center()`。每个方法返回新的 `Candidates`(不变动)→ 完全无头可测。执行器命令套用 JSON `ops` 列表。
+
 ## 本次更新 (2026-06-23) — 重试式数值断言(expect.poll)
 
 重试*任意*值直到符合,不只限内建检查。完整参考:[`docs/source/Zh/doc/new_features/v142_features_doc.rst`](../docs/source/Zh/doc/new_features/v142_features_doc.rst)。
diff --git a/README/WHATS_NEW_zh-TW.md b/README/WHATS_NEW_zh-TW.md
@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-23) — 可串接 / 可過濾的候選定位器
+
+用鏈式呼叫細化已定位的元素:`.within(panel).filter(has_text="Delete").nth(1)`。完整參考:[`docs/source/Zh/doc/new_features/v143_features_doc.rst`](../docs/source/Zh/doc/new_features/v143_features_doc.rst)。
+
+- **`from_boxes` / `Candidates`**(`AC_locate_chain`):`anchor_locator` 是單一關係、`grid_locator` 是儲存格——兩者都不支援對候選集合做可組合細化(Selenium-4 / Playwright 的串接定位慣用法)。本功能是對來自*任何*來源(模板 / OCR / a11y / `fuse_elements`)的框做純後置過濾:`within`(區域裁切)、`filter`(`has_text` / `near` / 面積 / predicate)、`sort_reading`、`nth` / `first` / `last`、`resolve()` / `center()`。每個方法回傳新的 `Candidates`(不變動)→ 完全無頭可測。執行器命令套用 JSON `ops` 清單。
+
 ## 本次更新 (2026-06-23) — 重試式數值斷言(expect.poll)
 
 重試*任意*值直到符合,不只限內建檢查。完整參考:[`docs/source/Zh/doc/new_features/v142_features_doc.rst`](../docs/source/Zh/doc/new_features/v142_features_doc.rst)。
diff --git a/WHATS_NEW.md b/WHATS_NEW.md
@@ -1,5 +1,11 @@
 # What's New — AutoControl
 
+## What's new (2026-06-23) — Composable / Filtered Candidate Locators
+
+Refine located elements with a chain: `.within(panel).filter(has_text="Delete").nth(1)`. Full reference: [`docs/source/Eng/doc/new_features/v143_features_doc.rst`](docs/source/Eng/doc/new_features/v143_features_doc.rst).
+
+- **`from_boxes` / `Candidates`** (`AC_locate_chain`): `anchor_locator` is a single relation and `grid_locator` is cells — neither supports composable refinement of a candidate set (the Selenium-4 / Playwright chained-locator idiom). This is a pure post-filter over boxes from *any* source (template / OCR / a11y / `fuse_elements`): `within` (region clip), `filter` (`has_text` / `near` / area / predicate), `sort_reading`, `nth` / `first` / `last`, `resolve()` / `center()`. Every method returns a new `Candidates` (no mutation) → fully headless-testable. The executor command applies a JSON `ops` list.
+
 ## What's new (2026-06-23) — Retrying Value Assertions (expect.poll)
 
 Retry *any* value until it matches, not just the built-in checks. Full reference: [`docs/source/Eng/doc/new_features/v142_features_doc.rst`](docs/source/Eng/doc/new_features/v142_features_doc.rst).
diff --git a/docs/source/Eng/doc/new_features/v143_features_doc.rst b/docs/source/Eng/doc/new_features/v143_features_doc.rst
@@ -0,0 +1,44 @@
+Composable / Filtered Candidate Locators
+=========================================
+
+``anchor_locator`` resolves a single anchor→target relation and ``grid_locator``
+addresses grid cells; neither supports *composable refinement* of a candidate set —
+``.within(panel).filter(has_text="Delete").nth(2)`` — the Selenium-4 / Playwright
+chained-and-filtered locator idiom. Today refining means re-querying a backend. This is
+a pure post-filter over boxes from *any* source (template match, OCR, the a11y tree,
+:doc:`v138_features_doc`).
+
+A ``Candidates`` wraps a list of ``{x, y, width, height, …}`` boxes; every method
+returns a *new* ``Candidates`` so chains are side-effect-free and fully unit-testable.
+Pure-stdlib, imports no ``PySide6``.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import from_boxes
+
+    target = (from_boxes(boxes)                    # boxes from any locator
+              .within((0, 0, 1920, 120))           # only the toolbar
+              .filter(has_text="Delete")           # only delete buttons
+              .sort_reading()                       # left-to-right
+              .nth(1))                              # the second one
+    if target.center():
+        click(*target.center())
+
+``within(region)`` keeps boxes whose centre is inside the rectangle; ``filter`` keeps
+boxes matching every supplied criterion (``has_text`` substring, ``near`` ``(x, y,
+max_dist)`` proximity, ``min_area`` / ``max_area``, or an arbitrary ``predicate``);
+``sort_reading`` orders them; ``nth`` / ``first`` / ``last`` select; ``resolve()``
+returns the surviving list and ``center()`` the first box's centre. Chains never mutate
+the original set.
+
+Executor command
+----------------
+
+``AC_locate_chain`` applies a JSON list of ``ops`` to a ``boxes`` array in order —
+``{op:"within",region:[…]}``, ``{op:"filter",has_text:…}``, ``{op:"reading"}``,
+``{op:"nth",index:…}``, ``{op:"first"}``, ``{op:"last"}`` — returning
+``{count, boxes, center}``. It is exposed as the MCP tool ``ac_locate_chain`` and as a
+Script Builder command under **Image**.
diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst
@@ -165,6 +165,7 @@ Comprehensive guides for all AutoControl features.
    doc/new_features/v140_features_doc
    doc/new_features/v141_features_doc
    doc/new_features/v142_features_doc
+   doc/new_features/v143_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc
diff --git a/docs/source/Zh/doc/new_features/v143_features_doc.rst b/docs/source/Zh/doc/new_features/v143_features_doc.rst
@@ -0,0 +1,37 @@
+可串接 / 可過濾的候選定位器
+============================
+
+``anchor_locator`` 解析單一錨點→目標關係,``grid_locator`` 定址網格儲存格;兩者都不支援對候選集合做*可組合的細化*
+——``.within(panel).filter(has_text="Delete").nth(2)``——這是 Selenium-4 / Playwright 的串接並過濾定位慣用法。
+目前細化得重新查詢後端。本功能是對來自*任何*來源(模板比對、OCR、a11y 樹、:doc:`v138_features_doc`)的框做純
+後置過濾。
+
+``Candidates`` 包裝一串 ``{x, y, width, height, …}`` 框;每個方法都回傳*新的* ``Candidates``,因此鏈式呼叫無副作用
+且完全可單元測試。純標準函式庫,不匯入 ``PySide6``。
+
+無頭 API
+--------
+
+.. code-block:: python
+
+    from je_auto_control import from_boxes
+
+    target = (from_boxes(boxes)                    # 來自任何定位器的框
+              .within((0, 0, 1920, 120))           # 只取工具列
+              .filter(has_text="Delete")           # 只取刪除按鈕
+              .sort_reading()                       # 由左到右
+              .nth(1))                              # 第二個
+    if target.center():
+        click(*target.center())
+
+``within(region)`` 保留中心落在矩形內的框;``filter`` 保留符合所有條件的框(``has_text`` 子字串、``near``
+``(x, y, max_dist)`` 鄰近、``min_area`` / ``max_area``,或任意 ``predicate``);``sort_reading`` 排序;
+``nth`` / ``first`` / ``last`` 選取;``resolve()`` 回傳存活清單、``center()`` 回傳第一個框的中心。鏈式呼叫永不
+變動原集合。
+
+執行器命令
+----------
+
+``AC_locate_chain`` 對 ``boxes`` 陣列依序套用一串 JSON ``ops``——``{op:"within",region:[…]}``、
+``{op:"filter",has_text:…}``、``{op:"reading"}``、``{op:"nth",index:…}``、``{op:"first"}``、``{op:"last"}``——
+回傳 ``{count, boxes, center}``。它以 MCP 工具 ``ac_locate_chain`` 以及 Script Builder 中 **Image** 分類下的命令提供。
diff --git a/docs/source/Zh/zh_index.rst b/docs/source/Zh/zh_index.rst
@@ -165,6 +165,7 @@ AutoControl 所有功能的完整使用指南。
    doc/new_features/v140_features_doc
    doc/new_features/v141_features_doc
    doc/new_features/v142_features_doc
+   doc/new_features/v143_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc
diff --git a/je_auto_control/__init__.py b/je_auto_control/__init__.py
@@ -333,6 +333,8 @@
     PollResult, assert_poll, expect_poll, to_be_greater_than, to_be_stable,
     to_be_truthy, to_contain, to_equal, to_match_regex,
 )
+# Composable / filtered candidate locators (chained-locator idiom)
+from je_auto_control.utils.locator_chain import Candidates, from_boxes
 # CI workflow annotations (GitHub Actions)
 from je_auto_control.utils.ci_annotations import (
     emit_annotations, format_annotation,
@@ -1183,6 +1185,8 @@ def start_autocontrol_gui(*args, **kwargs):
     "to_match_regex",
     "to_be_truthy",
     "to_be_stable",
+    "Candidates",
+    "from_boxes",
     "emit_annotations", "format_annotation",
     "ClipboardHistory", "default_clipboard_history",
     "analyze_heal_log", "heal_stats", "scan_secrets",
diff --git a/je_auto_control/gui/script_builder/command_schema.py b/je_auto_control/gui/script_builder/command_schema.py
@@ -492,6 +492,16 @@ def _add_image_specs(specs: List[CommandSpec]) -> None:
         ),
         description="Order element boxes top-to-bottom, left-to-right (+ index).",
     ))
+    specs.append(CommandSpec(
+        "AC_locate_chain", "Image", "Locate Chain (refine boxes)",
+        fields=(
+            FieldSpec("boxes", FieldType.STRING,
+                      placeholder='[{"x":..,"y":..,"width":..,"height":..}]'),
+            FieldSpec("ops", FieldType.STRING,
+                      placeholder='[{"op":"filter","has_text":"OK"},{"op":"first"}]'),
+        ),
+        description="Refine candidate boxes (within / filter / reading / nth / …).",
+    ))
 
 
 def _add_ocr_specs(specs: List[CommandSpec]) -> None:
diff --git a/je_auto_control/utils/executor/action_executor.py b/je_auto_control/utils/executor/action_executor.py
@@ -3625,6 +3625,42 @@ def getter():
             "waited_s": result.waited_s}
 
 
+def _apply_locate_op(candidates, op: Dict[str, Any]):
+    """Apply one locate-chain op spec to a Candidates set."""
+    name = op.get("op")
+    if name == "within":
+        return candidates.within(op["region"])
+    if name == "filter":
+        return candidates.filter(has_text=op.get("has_text"), near=op.get("near"),
+                                 min_area=op.get("min_area"),
+                                 max_area=op.get("max_area"))
+    if name == "reading":
+        return candidates.sort_reading(row_tol=int(op.get("row_tol", 12)))
+    if name == "nth":
+        return candidates.nth(int(op["index"]))
+    if name == "first":
+        return candidates.first()
+    if name == "last":
+        return candidates.last()
+    raise AutoControlActionException(f"unknown locate-chain op: {name!r}")
+
+
+def _locate_chain(boxes: Any, ops: Any = None) -> Dict[str, Any]:
+    """Adapter: apply a chain of refinement ops to a set of element boxes."""
+    import json
+    from je_auto_control.utils.locator_chain import from_boxes
+    if isinstance(boxes, str):
+        boxes = json.loads(boxes)
+    if isinstance(ops, str):
+        ops = json.loads(ops) if ops.strip() else []
+    candidates = from_boxes(list(boxes))
+    for op in ops or ():
+        candidates = _apply_locate_op(candidates, op)
+    resolved = candidates.resolve()
+    return {"count": len(resolved), "boxes": resolved,
+            "center": candidates.center()}
+
+
 def _with_modifiers(modifiers: Any, actions: Any) -> Dict[str, Any]:
     """Adapter: run nested actions while modifier keys are held down."""
     import json
@@ -5365,6 +5401,7 @@ def __init__(self):
             "AC_find_grid": _find_grid,
             "AC_find_separators": _find_separators,
             "AC_expect_poll": _expect_poll,
+            "AC_locate_chain": _locate_chain,
             "AC_tile_rect": _tile_rect,
             "AC_grid_rects": _grid_rects,
             "AC_cascade_rects": _cascade_rects,
diff --git a/je_auto_control/utils/locator_chain/__init__.py b/je_auto_control/utils/locator_chain/__init__.py
@@ -0,0 +1,6 @@
+"""Composable / filtered candidate locators (chained-locator idiom)."""
+from je_auto_control.utils.locator_chain.locator_chain import (
+    Candidates, from_boxes,
+)
+
+__all__ = ["Candidates", "from_boxes"]
diff --git a/je_auto_control/utils/locator_chain/locator_chain.py b/je_auto_control/utils/locator_chain/locator_chain.py
@@ -0,0 +1,106 @@
+"""Composable / filtered candidate locators — Playwright-style chained locators.
+
+``anchor_locator`` resolves a single anchor→target relation and ``grid_locator``
+addresses grid cells; neither supports *composable refinement* of a candidate set —
+``.within(panel).filter(has_text="Delete").nth(2)`` — the Selenium-4 / Playwright
+chained-and-filtered locator idiom. Today refining means re-querying a backend; this is
+a pure post-filter over boxes from *any* source (template / OCR / a11y).
+
+A ``Candidates`` wraps a list of ``{x, y, width, height, ...}`` boxes; every method
+returns a new ``Candidates`` so chains are side-effect-free and fully unit-testable.
+Pure-stdlib, imports no ``PySide6``.
+"""
+import math
+from typing import Any, Callable, Dict, List, Optional, Sequence, Tuple
+
+Box = Dict[str, Any]
+
+
+def _center(box: Box) -> List[int]:
+    return [int(box["x"]) + int(box["width"]) // 2,
+            int(box["y"]) + int(box["height"]) // 2]
+
+
+def _area(box: Box) -> int:
+    return int(box["width"]) * int(box["height"])
+
+
+class Candidates:
+    """An immutable, chainable set of element boxes."""
+
+    def __init__(self, boxes: Sequence[Box]):
+        self._boxes: List[Box] = [dict(box) for box in boxes]
+
+    def __iter__(self):
+        return iter(self._boxes)
+
+    def __len__(self) -> int:
+        return len(self._boxes)
+
+    def resolve(self) -> List[Box]:
+        """Return the surviving boxes as a list."""
+        return list(self._boxes)
+
+    def center(self) -> Optional[List[int]]:
+        """Return the centre of the first surviving box (or ``None``)."""
+        return _center(self._boxes[0]) if self._boxes else None
+
+    def within(self, region: Sequence[int]) -> "Candidates":
+        """Keep boxes whose centre falls inside ``region`` ``(x, y, w, h)``."""
+        rx, ry, rw, rh = (int(value) for value in region[:4])
+        kept = [box for box in self._boxes
+                if rx <= _center(box)[0] <= rx + rw
+                and ry <= _center(box)[1] <= ry + rh]
+        return Candidates(kept)
+
+    def filter(self, *, has_text: Optional[str] = None,
+               near: Optional[Tuple[int, int, float]] = None,
+               min_area: Optional[int] = None, max_area: Optional[int] = None,
+               predicate: Optional[Callable[[Box], bool]] = None) -> "Candidates":
+        """Keep boxes matching every supplied criterion.
+
+        ``has_text`` substring (case-insensitive) of the box ``text``; ``near``
+        ``(x, y, max_dist)`` centre proximity; ``min_area`` / ``max_area`` size;
+        ``predicate`` an arbitrary callable.
+        """
+        checks: List[Callable[[Box], bool]] = []
+        if has_text is not None:
+            needle = str(has_text).lower()
+            checks.append(lambda b: needle in str(b.get("text", "")).lower())
+        if near is not None:
+            nx, ny, dist = near
+            checks.append(
+                lambda b: math.hypot(_center(b)[0] - nx, _center(b)[1] - ny) <= dist)
+        if min_area is not None:
+            low = int(min_area)
+            checks.append(lambda b: _area(b) >= low)
+        if max_area is not None:
+            high = int(max_area)
+            checks.append(lambda b: _area(b) <= high)
+        if predicate is not None:
+            checks.append(predicate)
+        return Candidates([b for b in self._boxes if all(c(b) for c in checks)])
+
+    def sort_reading(self, row_tol: int = 12) -> "Candidates":
+        """Order the boxes top-to-bottom, left-to-right (reuses element_parse)."""
+        from je_auto_control.utils.element_parse import reading_order
+        return Candidates(reading_order(self._boxes, row_tol=int(row_tol)))
+
+    def nth(self, index: int) -> "Candidates":
+        """Keep only the box at ``index`` (negative allowed), else empty."""
+        if -len(self._boxes) <= index < len(self._boxes):
+            return Candidates([self._boxes[index]])
+        return Candidates([])
+
+    def first(self) -> "Candidates":
+        """Keep only the first box."""
+        return self.nth(0)
+
+    def last(self) -> "Candidates":
+        """Keep only the last box."""
+        return self.nth(-1)
+
+
+def from_boxes(boxes: Sequence[Box]) -> Candidates:
+    """Wrap a list of element boxes in a :class:`Candidates` for chaining."""
+    return Candidates(boxes)
diff --git a/je_auto_control/utils/mcp_server/tools/_factories.py b/je_auto_control/utils/mcp_server/tools/_factories.py
@@ -3041,6 +3041,25 @@ def expect_poll_tools() -> List[MCPTool]:
     ]
 
 
+def locator_chain_tools() -> List[MCPTool]:
+    return [
+        MCPTool(
+            name="ac_locate_chain",
+            description=("Refine a set of element 'boxes' with a chain of 'ops' "
+                         "applied in order: {op:'within',region:[x,y,w,h]}, "
+                         "{op:'filter',has_text/near/min_area/max_area}, "
+                         "{op:'reading'}, {op:'nth',index}, {op:'first'}, "
+                         "{op:'last'}. Returns {count, boxes, center}."),
+            input_schema=schema({
+                "boxes": {"type": "array", "items": {"type": "object"}},
+                "ops": {"type": "array", "items": {"type": "object"}}},
+                required=["boxes"]),
+            handler=h.locate_chain,
+            annotations=READ_ONLY,
+        ),
+    ]
+
+
 def ssim_tools() -> List[MCPTool]:
     return [
         MCPTool(
@@ -6546,7 +6565,7 @@ def media_assert_tools() -> List[MCPTool]:
     window_layout_tools, window_arrange_tools, preprocess_tools,
     monitor_layout_tools, actionability_tools, element_parse_tools,
     hsv_segment_tools, text_regions_tools, edge_lines_tools, expect_poll_tools,
-    plugin_sdk_tools, governance_tools,
+    locator_chain_tools, plugin_sdk_tools, governance_tools,
     credential_lease_tools, egress_tools, approval_testing_tools,
     trajectory_eval_tools, compliance_tools, agent_trace_tools,
     video_report_tools, fuzzy_tools, artifact_store_tools, image_dedup_tools,
diff --git a/je_auto_control/utils/mcp_server/tools/_handlers.py b/je_auto_control/utils/mcp_server/tools/_handlers.py
@@ -2232,6 +2232,11 @@ def expect_poll(action, key=None, op="truthy", expected=None, timeout_s=5.0,
     return _expect_poll(action, key, op, expected, timeout_s, interval_s)
 
 
+def locate_chain(boxes, ops=None):
+    from je_auto_control.utils.executor.action_executor import _locate_chain
+    return _locate_chain(boxes, ops)
+
+
 def detect_drift(reference, current, threshold=0.25, bins=10):
     from je_auto_control.utils.executor.action_executor import _detect_drift
     return _detect_drift(reference, current, threshold, bins)
diff --git a/test/unit_test/headless/test_locator_chain_batch.py b/test/unit_test/headless/test_locator_chain_batch.py
