Add grid/table cell addressing by row/col

README/WHATS_NEW_zh-CN.md

@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-23) — 表格 / 网格单元格定位
+
+依(行、列)从单元格边界框定位表格单元格。完整参考:[`docs/source/Zh/doc/new_features/v125_features_doc.rst`](../docs/source/Zh/doc/new_features/v125_features_doc.rst)。
+
+- **`cluster_grid` / `locate_cell`**(`AC_grid_cell`):`anchor_locator` 处理成对关系,但无法定位二维网格。给定单元格边界框(来自 `locate_all_image` / `find_text_matches`),本功能将其分群为行(依中心 y 在 `row_tolerance` 内)与列(依中心 x),并返回 0 起算 `(row, col)` 单元格的中心——可直接点击。纯分群、完全无头可测。
+
 ## 本次更新 (2026-06-23) — 锚点序数与全部定位
 
 挑选第 N 个锚点相对匹配,或列举全部。完整参考:[`docs/source/Zh/doc/new_features/v124_features_doc.rst`](../docs/source/Zh/doc/new_features/v124_features_doc.rst)。

README/WHATS_NEW_zh-TW.md

@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-23) — 表格 / 格線儲存格定位
+
+依(列、欄)從儲存格邊界框定位表格儲存格。完整參考:[`docs/source/Zh/doc/new_features/v125_features_doc.rst`](../docs/source/Zh/doc/new_features/v125_features_doc.rst)。
+
+- **`cluster_grid` / `locate_cell`**(`AC_grid_cell`):`anchor_locator` 處理成對關係,但無法定位二維格線。給定儲存格邊界框(來自 `locate_all_image` / `find_text_matches`),本功能將其分群為列(依中心 y 在 `row_tolerance` 內)與欄(依中心 x),並回傳 0 起算 `(row, col)` 儲存格的中心——可直接點擊。純分群、完全無頭可測。
+
 ## 本次更新 (2026-06-23) — 錨點序數與全部定位
 
 挑選第 N 個錨點相對比對,或列舉全部。完整參考:[`docs/source/Zh/doc/new_features/v124_features_doc.rst`](../docs/source/Zh/doc/new_features/v124_features_doc.rst)。

WHATS_NEW.md

@@ -1,5 +1,11 @@
 # What's New — AutoControl
 
+## What's new (2026-06-23) — Grid / Table Cell Addressing
+
+Address a table cell by (row, column) from cell bounding boxes. Full reference: [`docs/source/Eng/doc/new_features/v125_features_doc.rst`](docs/source/Eng/doc/new_features/v125_features_doc.rst).
+
+- **`cluster_grid` / `locate_cell`** (`AC_grid_cell`): `anchor_locator` does pairwise relations but nothing addresses a 2-D grid. Given the cell bounding boxes (from `locate_all_image` / `find_text_matches`), this clusters them into rows (by centre-y within `row_tolerance`) and columns (by centre-x) and returns the centre of the 0-based `(row, col)` cell — ready to click. Pure clustering, fully headless-testable.
+
 ## What's new (2026-06-23) — Anchor Ordinal & Locate-All
 
 Pick the Nth anchor-relative match, or enumerate them all. Full reference: [`docs/source/Eng/doc/new_features/v124_features_doc.rst`](docs/source/Eng/doc/new_features/v124_features_doc.rst).

docs/source/Eng/doc/new_features/v125_features_doc.rst

@@ -0,0 +1,41 @@
+Grid / Table Cell Addressing
+============================
+
+``anchor_locator`` does pairwise spatial relations (target *near* / *below* an
+anchor) but nothing addresses a 2-D grid — "the cell at row 3, column 2" of a
+table. Given the bounding boxes of the cells (from an image or OCR enumeration,
+e.g. ``locate_all_image`` / ``find_text_matches``), this clusters them into rows
+and columns and returns the requested cell's centre.
+
+The clustering and lookup are pure (boxes in, grid / cell out) and fully
+unit-testable; the box enumeration stays the caller's job, so nothing here needs a
+real screen. Imports no ``PySide6``.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import cluster_grid, locate_cell
+
+    boxes = [(10, 100, 20, 10), (110, 100, 20, 10), (210, 100, 20, 10),
+             (10, 200, 20, 10), (110, 200, 20, 10), (210, 200, 20, 10)]
+
+    locate_cell(boxes, row=1, col=2)
+    # {'found': True, 'center': [220, 205], 'box': [210, 200, 20, 10],
+    #  'row': 1, 'col': 2, 'rows': 2, 'cols': 3}
+
+    cluster_grid(boxes)   # rows top-to-bottom, cells left-to-right
+
+``cluster_grid`` sorts the boxes by centre-y, starts a new row when the gap
+exceeds ``row_tolerance``, and orders each row's cells by centre-x.
+``locate_cell`` returns the centre (ready to click) of the 0-based ``(row, col)``
+cell, or ``{found: False, reason}`` when the index is out of range.
+
+Executor commands
+-----------------
+
+``AC_grid_cell`` takes ``boxes`` (a JSON ``[[x, y, w, h], ...]`` list, e.g. from a
+prior ``AC_locate_all_image`` step) plus ``row`` / ``col`` / ``row_tolerance`` and
+returns the cell dict. It is exposed as the MCP tool ``ac_grid_cell`` and as a
+Script Builder command under **Mouse**.

docs/source/Eng/eng_index.rst

@@ -147,6 +147,7 @@ Comprehensive guides for all AutoControl features.
    doc/new_features/v122_features_doc
    doc/new_features/v123_features_doc
    doc/new_features/v124_features_doc
+   doc/new_features/v125_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

docs/source/Zh/doc/new_features/v125_features_doc.rst

@@ -0,0 +1,36 @@
+表格 / 格線儲存格定位
+====================
+
+``anchor_locator`` 處理成對的空間關係(目標在錨點*附近* / *下方*),但無法定位二維格線——表格中「第 3 列、
+第 2 欄的儲存格」。給定各儲存格的邊界框(來自影像或 OCR 列舉,例如 ``locate_all_image`` / ``find_text_matches``),
+本功能將其分群為列與欄,並回傳所求儲存格的中心。
+
+分群與查詢皆為純函式(框進、格線 / 儲存格出),完全可單元測試;框的列舉仍由呼叫端負責,因此此處不需真實螢幕。
+不匯入 ``PySide6``。
+
+無頭 API
+--------
+
+.. code-block:: python
+
+    from je_auto_control import cluster_grid, locate_cell
+
+    boxes = [(10, 100, 20, 10), (110, 100, 20, 10), (210, 100, 20, 10),
+             (10, 200, 20, 10), (110, 200, 20, 10), (210, 200, 20, 10)]
+
+    locate_cell(boxes, row=1, col=2)
+    # {'found': True, 'center': [220, 205], 'box': [210, 200, 20, 10],
+    #  'row': 1, 'col': 2, 'rows': 2, 'cols': 3}
+
+    cluster_grid(boxes)   # 列由上到下、儲存格由左到右
+
+``cluster_grid`` 依中心 y 排序框,當間距超過 ``row_tolerance`` 時開始新的一列,並將每列的儲存格依中心 x 排序。
+``locate_cell`` 回傳 0 起算 ``(row, col)`` 儲存格的中心(可直接點擊),索引超出範圍時回傳
+``{found: False, reason}``。
+
+執行器命令
+----------
+
+``AC_grid_cell`` 接受 ``boxes``(JSON ``[[x, y, w, h], ...]`` 清單,例如來自前一個 ``AC_locate_all_image`` 步驟)
+以及 ``row`` / ``col`` / ``row_tolerance``,並回傳儲存格 dict。它以 MCP 工具 ``ac_grid_cell`` 以及 Script Builder
+中 **Mouse** 分類下的命令提供。

docs/source/Zh/zh_index.rst

@@ -147,6 +147,7 @@ AutoControl 所有功能的完整使用指南。
    doc/new_features/v122_features_doc
    doc/new_features/v123_features_doc
    doc/new_features/v124_features_doc
+   doc/new_features/v125_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

je_auto_control/__init__.py

@@ -271,6 +271,8 @@
 from je_auto_control.utils.modifier_state import (
     hold_modifiers, plan_with_modifiers,
 )
+# Address a table / grid cell by (row, column) from bounding boxes
+from je_auto_control.utils.grid_locator import cluster_grid, locate_cell
 # CI workflow annotations (GitHub Actions)
 from je_auto_control.utils.ci_annotations import (
     emit_annotations, format_annotation,
@@ -1058,6 +1060,8 @@ def start_autocontrol_gui(*args, **kwargs):
     "unicode_code_units",
     "hold_modifiers",
     "plan_with_modifiers",
+    "cluster_grid",
+    "locate_cell",
     "emit_annotations", "format_annotation",
     "ClipboardHistory", "default_clipboard_history",
     "analyze_heal_log", "heal_stats", "scan_secrets",

je_auto_control/gui/script_builder/command_schema.py

@@ -800,6 +800,17 @@ def _add_misc_specs(specs: List[CommandSpec]) -> None:
         ),
         description="Move the pointer by (dx, dy) from its current position.",
     ))
+    specs.append(CommandSpec(
+        "AC_grid_cell", "Mouse", "Grid Cell (row / col)",
+        fields=(
+            FieldSpec("boxes", FieldType.STRING,
+                      placeholder="[[10,100,20,10],[110,100,20,10], ...]"),
+            FieldSpec("row", FieldType.INT, placeholder="0"),
+            FieldSpec("col", FieldType.INT, placeholder="0"),
+            FieldSpec("row_tolerance", FieldType.INT, optional=True, default=10),
+        ),
+        description="Resolve a table cell centre by row/col from cell boxes.",
+    ))
     specs.append(CommandSpec(
         "AC_list_plugins", "Tools", "List Plugin Commands",
         fields=(FieldSpec("group", FieldType.STRING, optional=True,

je_auto_control/utils/executor/action_executor.py

@@ -3175,6 +3175,17 @@ def _type_unicode(text: str, modifier: str = "ctrl") -> Dict[str, Any]:
     return type_unicode(text, modifier=modifier)
 
 
+def _grid_cell(boxes: Any, row: Any, col: Any,
+               row_tolerance: Any = 10) -> Dict[str, Any]:
+    """Adapter: address a grid cell by (row, col) from a JSON list of boxes."""
+    import json
+    from je_auto_control.utils.grid_locator import locate_cell
+    if isinstance(boxes, str):
+        boxes = json.loads(boxes)
+    return locate_cell(list(boxes), int(row), int(col),
+                       row_tolerance=int(row_tolerance))
+
+
 def _with_modifiers(modifiers: Any, actions: Any) -> Dict[str, Any]:
     """Adapter: run nested actions while modifier keys are held down."""
     import json
@@ -4891,6 +4902,7 @@ def __init__(self):
             "AC_move_mouse_relative": _move_mouse_relative,
             "AC_type_unicode": _type_unicode,
             "AC_with_modifiers": _with_modifiers,
+            "AC_grid_cell": _grid_cell,
             "AC_detect_drift": _detect_drift,
             "AC_categorical_drift": _categorical_drift,
             "AC_diff_rows": _diff_rows,

je_auto_control/utils/grid_locator/__init__.py

@@ -0,0 +1,6 @@
+"""Address a table / grid cell by (row, column) from bounding boxes."""
+from je_auto_control.utils.grid_locator.grid_locator import (
+    cluster_grid, locate_cell,
+)
+
+__all__ = ["cluster_grid", "locate_cell"]

je_auto_control/utils/grid_locator/grid_locator.py

@@ -0,0 +1,65 @@
+"""Address a table / grid cell by (row, column) from a set of bounding boxes.
+
+``anchor_locator`` does pairwise spatial relations (target *near* / *below* an
+anchor) but nothing addresses a 2-D grid — "the cell at row 3, column 2" of a
+table. Given the bounding boxes of the cells (from an image or OCR enumeration —
+e.g. ``locate_all_image`` / ``find_text_matches``), this clusters them into rows
+and columns and returns the requested cell's centre.
+
+The clustering and lookup are pure (boxes in, grid / cell out) and fully
+unit-testable; the box enumeration stays the caller's job, so nothing here needs a
+real screen. Imports no ``PySide6``.
+"""
+from typing import Any, Dict, List, Sequence, Tuple
+
+Box = Sequence[int]
+
+
+def _center(box: Box) -> Tuple[int, int]:
+    """Return the integer centre ``(x, y)`` of an ``(x, y, w, h)`` box."""
+    x, y, width, height = (int(value) for value in box[:4])
+    return x + width // 2, y + height // 2
+
+
+def cluster_grid(boxes: Sequence[Box], *,
+                 row_tolerance: int = 10) -> List[List[List[int]]]:
+    """Cluster ``(x, y, w, h)`` boxes into rows (top-down), cells left-to-right.
+
+    Boxes whose centre-y values are within ``row_tolerance`` of the previous
+    box (after sorting by y) share a row; within a row the cells are ordered by
+    centre-x. Returns a list of rows, each a list of ``[x, y, w, h]`` boxes.
+    """
+    items = sorted((list(map(int, box[:4])) for box in boxes),
+                   key=lambda box: _center(box)[1])
+    rows: List[List[List[int]]] = []
+    current: List[List[int]] = []
+    last_cy = None
+    for box in items:
+        center_y = _center(box)[1]
+        if last_cy is not None and abs(center_y - last_cy) > int(row_tolerance):
+            rows.append(current)
+            current = []
+        current.append(box)
+        last_cy = center_y
+    if current:
+        rows.append(current)
+    for row in rows:
+        row.sort(key=lambda box: _center(box)[0])
+    return rows
+
+
+def locate_cell(boxes: Sequence[Box], row: int, col: int, *,
+                row_tolerance: int = 10) -> Dict[str, Any]:
+    """Return the cell at ``(row, col)`` (both 0-based) of the clustered grid."""
+    grid = cluster_grid(boxes, row_tolerance=row_tolerance)
+    if not 0 <= row < len(grid):
+        return {"found": False, "reason": "row out of range",
+                "rows": len(grid), "cols": 0}
+    line = grid[row]
+    if not 0 <= col < len(line):
+        return {"found": False, "reason": "col out of range",
+                "rows": len(grid), "cols": len(line)}
+    box = line[col]
+    center = _center(box)
+    return {"found": True, "center": [center[0], center[1]], "box": list(box),
+            "row": row, "col": col, "rows": len(grid), "cols": len(line)}

je_auto_control/utils/mcp_server/tools/_factories.py

@@ -2613,6 +2613,27 @@ def tween_drag_tools() -> List[MCPTool]:
     ]
 
 
+def grid_locator_tools() -> List[MCPTool]:
+    return [
+        MCPTool(
+            name="ac_grid_cell",
+            description=("Address a table cell: cluster 'boxes' ([[x,y,w,h],...] "
+                         "from an image/OCR enumeration) into a grid and return "
+                         "the centre of cell ('row','col') (0-based). "
+                         "Returns {found, center, row, col, rows, cols}."),
+            input_schema=schema({
+                "boxes": {"type": "array",
+                          "items": {"type": "array",
+                                    "items": {"type": "integer"}}},
+                "row": {"type": "integer"}, "col": {"type": "integer"},
+                "row_tolerance": {"type": "integer"}},
+                required=["boxes", "row", "col"]),
+            handler=h.grid_cell,
+            annotations=READ_ONLY,
+        ),
+    ]
+
+
 def modifier_state_tools() -> List[MCPTool]:
     return [
         MCPTool(
@@ -5982,7 +6003,8 @@ def media_assert_tools() -> List[MCPTool]:
     ci_annotation_tools, clipboard_history_tools, audit_analysis_tools,
     process_doc_tools, tween_drag_tools, mouse_path_tools, field_entry_tools,
     key_hold_tools, mouse_relative_tools, text_unicode_tools,
-    modifier_state_tools, plugin_sdk_tools, governance_tools,
+    modifier_state_tools, grid_locator_tools, plugin_sdk_tools,
+    governance_tools,
     credential_lease_tools, egress_tools, approval_testing_tools,
     trajectory_eval_tools, compliance_tools, agent_trace_tools,
     video_report_tools, fuzzy_tools, artifact_store_tools, image_dedup_tools,

je_auto_control/utils/mcp_server/tools/_handlers.py

@@ -2063,6 +2063,11 @@ def with_modifiers(modifiers, actions):
     return _with_modifiers(modifiers, actions)
 
 
+def grid_cell(boxes, row, col, row_tolerance=10):
+    from je_auto_control.utils.executor.action_executor import _grid_cell
+    return _grid_cell(boxes, row, col, row_tolerance)
+
+
 def detect_drift(reference, current, threshold=0.25, bins=10):
     from je_auto_control.utils.executor.action_executor import _detect_drift
     return _detect_drift(reference, current, threshold, bins)

test/unit_test/headless/test_grid_locator_batch.py

@@ -0,0 +1,69 @@
+"""Headless tests for grid / table cell addressing. No Qt."""
+import json
+
+import je_auto_control as ac
+from je_auto_control.utils.grid_locator import cluster_grid, locate_cell
+
+# a 2-row x 3-col grid, deliberately shuffled
+_BOXES = [
+    (210, 100, 20, 10), (10, 100, 20, 10), (110, 100, 20, 10),
+    (110, 200, 20, 10), (10, 200, 20, 10), (210, 200, 20, 10),
+]
+
+
+def test_cluster_groups_rows_and_orders_columns():
+    grid = cluster_grid(_BOXES)
+    assert len(grid) == 2 and [len(r) for r in grid] == [3, 3]
+    assert [b[0] for b in grid[0]] == [10, 110, 210]   # left-to-right
+    assert [b[1] for b in grid[0]] == [100, 100, 100]   # top row
+
+
+def test_locate_cell_returns_center():
+    cell = locate_cell(_BOXES, 1, 2)
+    assert cell["found"] is True
+    assert cell["center"] == [220, 205] and cell["box"] == [210, 200, 20, 10]
+    assert cell["rows"] == 2 and cell["cols"] == 3
+
+
+def test_out_of_range():
+    assert locate_cell(_BOXES, 5, 0)["found"] is False
+    assert locate_cell(_BOXES, 0, 9)["reason"] == "col out of range"
+
+
+def test_row_tolerance_merges_near_y():
+    # second row jitters by 4px in y -> still one row at tolerance 10
+    jittered = [(10, 100, 20, 10), (110, 104, 20, 10), (210, 98, 20, 10)]
+    assert len(cluster_grid(jittered, row_tolerance=10)) == 1
+    assert len(cluster_grid(jittered, row_tolerance=2)) >= 2   # splits
+
+
+def test_empty_boxes():
+    assert cluster_grid([]) == []
+    assert locate_cell([], 0, 0)["found"] is False
+
+
+# --- wiring ---------------------------------------------------------------
+
+def test_executor_round_trip():
+    rec = ac.execute_action([[
+        "AC_grid_cell", {"boxes": json.dumps([[10, 100, 20, 10],
+                                              [110, 100, 20, 10]]),
+                         "row": 0, "col": 1}]])
+    out = next(v for v in rec.values() if isinstance(v, dict))
+    assert out["found"] is True and out["center"] == [120, 105]
+
+
+def test_wiring():
+    known = ac.executor.known_commands()
+    assert "AC_grid_cell" in set(known)
+    from je_auto_control.utils.mcp_server.tools import build_default_tool_registry
+    names = {t.name for t in build_default_tool_registry()}
+    assert "ac_grid_cell" in names
+    from je_auto_control.gui.script_builder.command_schema import _build_specs
+    specs = {s.command for s in _build_specs()}
+    assert "AC_grid_cell" in specs
+
+
+def test_facade_exports():
+    for attr in ("cluster_grid", "locate_cell"):
+        assert hasattr(ac, attr) and attr in ac.__all__