diff --git a/README/WHATS_NEW_zh-CN.md b/README/WHATS_NEW_zh-CN.md index e7e3895f..d2d9607f 100644 --- a/README/WHATS_NEW_zh-CN.md +++ b/README/WHATS_NEW_zh-CN.md @@ -1,5 +1,11 @@ # 本次更新 — AutoControl +## 本次更新 (2026-06-23) — 表格 / 网格单元格定位 + +依(行、列)从单元格边界框定位表格单元格。完整参考:[`docs/source/Zh/doc/new_features/v125_features_doc.rst`](../docs/source/Zh/doc/new_features/v125_features_doc.rst)。 + +- **`cluster_grid` / `locate_cell`**(`AC_grid_cell`):`anchor_locator` 处理成对关系,但无法定位二维网格。给定单元格边界框(来自 `locate_all_image` / `find_text_matches`),本功能将其分群为行(依中心 y 在 `row_tolerance` 内)与列(依中心 x),并返回 0 起算 `(row, col)` 单元格的中心——可直接点击。纯分群、完全无头可测。 + ## 本次更新 (2026-06-23) — 锚点序数与全部定位 挑选第 N 个锚点相对匹配,或列举全部。完整参考:[`docs/source/Zh/doc/new_features/v124_features_doc.rst`](../docs/source/Zh/doc/new_features/v124_features_doc.rst)。 diff --git a/README/WHATS_NEW_zh-TW.md b/README/WHATS_NEW_zh-TW.md index 321aafd6..81837914 100644 --- a/README/WHATS_NEW_zh-TW.md +++ b/README/WHATS_NEW_zh-TW.md @@ -1,5 +1,11 @@ # 本次更新 — AutoControl +## 本次更新 (2026-06-23) — 表格 / 格線儲存格定位 + +依(列、欄)從儲存格邊界框定位表格儲存格。完整參考:[`docs/source/Zh/doc/new_features/v125_features_doc.rst`](../docs/source/Zh/doc/new_features/v125_features_doc.rst)。 + +- **`cluster_grid` / `locate_cell`**(`AC_grid_cell`):`anchor_locator` 處理成對關係,但無法定位二維格線。給定儲存格邊界框(來自 `locate_all_image` / `find_text_matches`),本功能將其分群為列(依中心 y 在 `row_tolerance` 內)與欄(依中心 x),並回傳 0 起算 `(row, col)` 儲存格的中心——可直接點擊。純分群、完全無頭可測。 + ## 本次更新 (2026-06-23) — 錨點序數與全部定位 挑選第 N 個錨點相對比對,或列舉全部。完整參考:[`docs/source/Zh/doc/new_features/v124_features_doc.rst`](../docs/source/Zh/doc/new_features/v124_features_doc.rst)。 diff --git a/WHATS_NEW.md b/WHATS_NEW.md index 10b74df8..41b21564 100644 --- a/WHATS_NEW.md +++ b/WHATS_NEW.md @@ -1,5 +1,11 @@ # What's New — AutoControl +## What's new (2026-06-23) — Grid / Table Cell Addressing + +Address a table cell by (row, column) from cell bounding boxes. Full reference: [`docs/source/Eng/doc/new_features/v125_features_doc.rst`](docs/source/Eng/doc/new_features/v125_features_doc.rst). + +- **`cluster_grid` / `locate_cell`** (`AC_grid_cell`): `anchor_locator` does pairwise relations but nothing addresses a 2-D grid. Given the cell bounding boxes (from `locate_all_image` / `find_text_matches`), this clusters them into rows (by centre-y within `row_tolerance`) and columns (by centre-x) and returns the centre of the 0-based `(row, col)` cell — ready to click. Pure clustering, fully headless-testable. + ## What's new (2026-06-23) — Anchor Ordinal & Locate-All Pick the Nth anchor-relative match, or enumerate them all. Full reference: [`docs/source/Eng/doc/new_features/v124_features_doc.rst`](docs/source/Eng/doc/new_features/v124_features_doc.rst). diff --git a/docs/source/Eng/doc/new_features/v125_features_doc.rst b/docs/source/Eng/doc/new_features/v125_features_doc.rst new file mode 100644 index 00000000..ccfa8ee9 --- /dev/null +++ b/docs/source/Eng/doc/new_features/v125_features_doc.rst @@ -0,0 +1,41 @@ +Grid / Table Cell Addressing +============================ + +``anchor_locator`` does pairwise spatial relations (target *near* / *below* an +anchor) but nothing addresses a 2-D grid — "the cell at row 3, column 2" of a +table. Given the bounding boxes of the cells (from an image or OCR enumeration, +e.g. ``locate_all_image`` / ``find_text_matches``), this clusters them into rows +and columns and returns the requested cell's centre. + +The clustering and lookup are pure (boxes in, grid / cell out) and fully +unit-testable; the box enumeration stays the caller's job, so nothing here needs a +real screen. Imports no ``PySide6``. + +Headless API +------------ + +.. code-block:: python + + from je_auto_control import cluster_grid, locate_cell + + boxes = [(10, 100, 20, 10), (110, 100, 20, 10), (210, 100, 20, 10), + (10, 200, 20, 10), (110, 200, 20, 10), (210, 200, 20, 10)] + + locate_cell(boxes, row=1, col=2) + # {'found': True, 'center': [220, 205], 'box': [210, 200, 20, 10], + # 'row': 1, 'col': 2, 'rows': 2, 'cols': 3} + + cluster_grid(boxes) # rows top-to-bottom, cells left-to-right + +``cluster_grid`` sorts the boxes by centre-y, starts a new row when the gap +exceeds ``row_tolerance``, and orders each row's cells by centre-x. +``locate_cell`` returns the centre (ready to click) of the 0-based ``(row, col)`` +cell, or ``{found: False, reason}`` when the index is out of range. + +Executor commands +----------------- + +``AC_grid_cell`` takes ``boxes`` (a JSON ``[[x, y, w, h], ...]`` list, e.g. from a +prior ``AC_locate_all_image`` step) plus ``row`` / ``col`` / ``row_tolerance`` and +returns the cell dict. It is exposed as the MCP tool ``ac_grid_cell`` and as a +Script Builder command under **Mouse**. diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst index 525ceffb..55dee8ae 100644 --- a/docs/source/Eng/eng_index.rst +++ b/docs/source/Eng/eng_index.rst @@ -147,6 +147,7 @@ Comprehensive guides for all AutoControl features. doc/new_features/v122_features_doc doc/new_features/v123_features_doc doc/new_features/v124_features_doc + doc/new_features/v125_features_doc doc/ocr_backends/ocr_backends_doc doc/observability/observability_doc doc/operations_layer/operations_layer_doc diff --git a/docs/source/Zh/doc/new_features/v125_features_doc.rst b/docs/source/Zh/doc/new_features/v125_features_doc.rst new file mode 100644 index 00000000..0be94e4c --- /dev/null +++ b/docs/source/Zh/doc/new_features/v125_features_doc.rst @@ -0,0 +1,36 @@ +表格 / 格線儲存格定位 +==================== + +``anchor_locator`` 處理成對的空間關係(目標在錨點*附近* / *下方*),但無法定位二維格線——表格中「第 3 列、 +第 2 欄的儲存格」。給定各儲存格的邊界框(來自影像或 OCR 列舉,例如 ``locate_all_image`` / ``find_text_matches``), +本功能將其分群為列與欄,並回傳所求儲存格的中心。 + +分群與查詢皆為純函式(框進、格線 / 儲存格出),完全可單元測試;框的列舉仍由呼叫端負責,因此此處不需真實螢幕。 +不匯入 ``PySide6``。 + +無頭 API +-------- + +.. code-block:: python + + from je_auto_control import cluster_grid, locate_cell + + boxes = [(10, 100, 20, 10), (110, 100, 20, 10), (210, 100, 20, 10), + (10, 200, 20, 10), (110, 200, 20, 10), (210, 200, 20, 10)] + + locate_cell(boxes, row=1, col=2) + # {'found': True, 'center': [220, 205], 'box': [210, 200, 20, 10], + # 'row': 1, 'col': 2, 'rows': 2, 'cols': 3} + + cluster_grid(boxes) # 列由上到下、儲存格由左到右 + +``cluster_grid`` 依中心 y 排序框,當間距超過 ``row_tolerance`` 時開始新的一列,並將每列的儲存格依中心 x 排序。 +``locate_cell`` 回傳 0 起算 ``(row, col)`` 儲存格的中心(可直接點擊),索引超出範圍時回傳 +``{found: False, reason}``。 + +執行器命令 +---------- + +``AC_grid_cell`` 接受 ``boxes``(JSON ``[[x, y, w, h], ...]`` 清單,例如來自前一個 ``AC_locate_all_image`` 步驟) +以及 ``row`` / ``col`` / ``row_tolerance``,並回傳儲存格 dict。它以 MCP 工具 ``ac_grid_cell`` 以及 Script Builder +中 **Mouse** 分類下的命令提供。 diff --git a/docs/source/Zh/zh_index.rst b/docs/source/Zh/zh_index.rst index 52a9f226..dedefc4a 100644 --- a/docs/source/Zh/zh_index.rst +++ b/docs/source/Zh/zh_index.rst @@ -147,6 +147,7 @@ AutoControl 所有功能的完整使用指南。 doc/new_features/v122_features_doc doc/new_features/v123_features_doc doc/new_features/v124_features_doc + doc/new_features/v125_features_doc doc/ocr_backends/ocr_backends_doc doc/observability/observability_doc doc/operations_layer/operations_layer_doc diff --git a/je_auto_control/__init__.py b/je_auto_control/__init__.py index 5f5a2ec5..8422955c 100644 --- a/je_auto_control/__init__.py +++ b/je_auto_control/__init__.py @@ -271,6 +271,8 @@ from je_auto_control.utils.modifier_state import ( hold_modifiers, plan_with_modifiers, ) +# Address a table / grid cell by (row, column) from bounding boxes +from je_auto_control.utils.grid_locator import cluster_grid, locate_cell # CI workflow annotations (GitHub Actions) from je_auto_control.utils.ci_annotations import ( emit_annotations, format_annotation, @@ -1058,6 +1060,8 @@ def start_autocontrol_gui(*args, **kwargs): "unicode_code_units", "hold_modifiers", "plan_with_modifiers", + "cluster_grid", + "locate_cell", "emit_annotations", "format_annotation", "ClipboardHistory", "default_clipboard_history", "analyze_heal_log", "heal_stats", "scan_secrets", diff --git a/je_auto_control/gui/script_builder/command_schema.py b/je_auto_control/gui/script_builder/command_schema.py index 71505ef1..53349035 100644 --- a/je_auto_control/gui/script_builder/command_schema.py +++ b/je_auto_control/gui/script_builder/command_schema.py @@ -800,6 +800,17 @@ def _add_misc_specs(specs: List[CommandSpec]) -> None: ), description="Move the pointer by (dx, dy) from its current position.", )) + specs.append(CommandSpec( + "AC_grid_cell", "Mouse", "Grid Cell (row / col)", + fields=( + FieldSpec("boxes", FieldType.STRING, + placeholder="[[10,100,20,10],[110,100,20,10], ...]"), + FieldSpec("row", FieldType.INT, placeholder="0"), + FieldSpec("col", FieldType.INT, placeholder="0"), + FieldSpec("row_tolerance", FieldType.INT, optional=True, default=10), + ), + description="Resolve a table cell centre by row/col from cell boxes.", + )) specs.append(CommandSpec( "AC_list_plugins", "Tools", "List Plugin Commands", fields=(FieldSpec("group", FieldType.STRING, optional=True, diff --git a/je_auto_control/utils/executor/action_executor.py b/je_auto_control/utils/executor/action_executor.py index 0648e6d3..9220e20f 100644 --- a/je_auto_control/utils/executor/action_executor.py +++ b/je_auto_control/utils/executor/action_executor.py @@ -3175,6 +3175,17 @@ def _type_unicode(text: str, modifier: str = "ctrl") -> Dict[str, Any]: return type_unicode(text, modifier=modifier) +def _grid_cell(boxes: Any, row: Any, col: Any, + row_tolerance: Any = 10) -> Dict[str, Any]: + """Adapter: address a grid cell by (row, col) from a JSON list of boxes.""" + import json + from je_auto_control.utils.grid_locator import locate_cell + if isinstance(boxes, str): + boxes = json.loads(boxes) + return locate_cell(list(boxes), int(row), int(col), + row_tolerance=int(row_tolerance)) + + def _with_modifiers(modifiers: Any, actions: Any) -> Dict[str, Any]: """Adapter: run nested actions while modifier keys are held down.""" import json @@ -4891,6 +4902,7 @@ def __init__(self): "AC_move_mouse_relative": _move_mouse_relative, "AC_type_unicode": _type_unicode, "AC_with_modifiers": _with_modifiers, + "AC_grid_cell": _grid_cell, "AC_detect_drift": _detect_drift, "AC_categorical_drift": _categorical_drift, "AC_diff_rows": _diff_rows, diff --git a/je_auto_control/utils/grid_locator/__init__.py b/je_auto_control/utils/grid_locator/__init__.py new file mode 100644 index 00000000..1bca3635 --- /dev/null +++ b/je_auto_control/utils/grid_locator/__init__.py @@ -0,0 +1,6 @@ +"""Address a table / grid cell by (row, column) from bounding boxes.""" +from je_auto_control.utils.grid_locator.grid_locator import ( + cluster_grid, locate_cell, +) + +__all__ = ["cluster_grid", "locate_cell"] diff --git a/je_auto_control/utils/grid_locator/grid_locator.py b/je_auto_control/utils/grid_locator/grid_locator.py new file mode 100644 index 00000000..8542ff17 --- /dev/null +++ b/je_auto_control/utils/grid_locator/grid_locator.py @@ -0,0 +1,65 @@ +"""Address a table / grid cell by (row, column) from a set of bounding boxes. + +``anchor_locator`` does pairwise spatial relations (target *near* / *below* an +anchor) but nothing addresses a 2-D grid — "the cell at row 3, column 2" of a +table. Given the bounding boxes of the cells (from an image or OCR enumeration — +e.g. ``locate_all_image`` / ``find_text_matches``), this clusters them into rows +and columns and returns the requested cell's centre. + +The clustering and lookup are pure (boxes in, grid / cell out) and fully +unit-testable; the box enumeration stays the caller's job, so nothing here needs a +real screen. Imports no ``PySide6``. +""" +from typing import Any, Dict, List, Sequence, Tuple + +Box = Sequence[int] + + +def _center(box: Box) -> Tuple[int, int]: + """Return the integer centre ``(x, y)`` of an ``(x, y, w, h)`` box.""" + x, y, width, height = (int(value) for value in box[:4]) + return x + width // 2, y + height // 2 + + +def cluster_grid(boxes: Sequence[Box], *, + row_tolerance: int = 10) -> List[List[List[int]]]: + """Cluster ``(x, y, w, h)`` boxes into rows (top-down), cells left-to-right. + + Boxes whose centre-y values are within ``row_tolerance`` of the previous + box (after sorting by y) share a row; within a row the cells are ordered by + centre-x. Returns a list of rows, each a list of ``[x, y, w, h]`` boxes. + """ + items = sorted((list(map(int, box[:4])) for box in boxes), + key=lambda box: _center(box)[1]) + rows: List[List[List[int]]] = [] + current: List[List[int]] = [] + last_cy = None + for box in items: + center_y = _center(box)[1] + if last_cy is not None and abs(center_y - last_cy) > int(row_tolerance): + rows.append(current) + current = [] + current.append(box) + last_cy = center_y + if current: + rows.append(current) + for row in rows: + row.sort(key=lambda box: _center(box)[0]) + return rows + + +def locate_cell(boxes: Sequence[Box], row: int, col: int, *, + row_tolerance: int = 10) -> Dict[str, Any]: + """Return the cell at ``(row, col)`` (both 0-based) of the clustered grid.""" + grid = cluster_grid(boxes, row_tolerance=row_tolerance) + if not 0 <= row < len(grid): + return {"found": False, "reason": "row out of range", + "rows": len(grid), "cols": 0} + line = grid[row] + if not 0 <= col < len(line): + return {"found": False, "reason": "col out of range", + "rows": len(grid), "cols": len(line)} + box = line[col] + center = _center(box) + return {"found": True, "center": [center[0], center[1]], "box": list(box), + "row": row, "col": col, "rows": len(grid), "cols": len(line)} diff --git a/je_auto_control/utils/mcp_server/tools/_factories.py b/je_auto_control/utils/mcp_server/tools/_factories.py index 8fc3e670..c22c806c 100644 --- a/je_auto_control/utils/mcp_server/tools/_factories.py +++ b/je_auto_control/utils/mcp_server/tools/_factories.py @@ -2613,6 +2613,27 @@ def tween_drag_tools() -> List[MCPTool]: ] +def grid_locator_tools() -> List[MCPTool]: + return [ + MCPTool( + name="ac_grid_cell", + description=("Address a table cell: cluster 'boxes' ([[x,y,w,h],...] " + "from an image/OCR enumeration) into a grid and return " + "the centre of cell ('row','col') (0-based). " + "Returns {found, center, row, col, rows, cols}."), + input_schema=schema({ + "boxes": {"type": "array", + "items": {"type": "array", + "items": {"type": "integer"}}}, + "row": {"type": "integer"}, "col": {"type": "integer"}, + "row_tolerance": {"type": "integer"}}, + required=["boxes", "row", "col"]), + handler=h.grid_cell, + annotations=READ_ONLY, + ), + ] + + def modifier_state_tools() -> List[MCPTool]: return [ MCPTool( @@ -5982,7 +6003,8 @@ def media_assert_tools() -> List[MCPTool]: ci_annotation_tools, clipboard_history_tools, audit_analysis_tools, process_doc_tools, tween_drag_tools, mouse_path_tools, field_entry_tools, key_hold_tools, mouse_relative_tools, text_unicode_tools, - modifier_state_tools, plugin_sdk_tools, governance_tools, + modifier_state_tools, grid_locator_tools, plugin_sdk_tools, + governance_tools, credential_lease_tools, egress_tools, approval_testing_tools, trajectory_eval_tools, compliance_tools, agent_trace_tools, video_report_tools, fuzzy_tools, artifact_store_tools, image_dedup_tools, diff --git a/je_auto_control/utils/mcp_server/tools/_handlers.py b/je_auto_control/utils/mcp_server/tools/_handlers.py index e2238fea..7350d385 100644 --- a/je_auto_control/utils/mcp_server/tools/_handlers.py +++ b/je_auto_control/utils/mcp_server/tools/_handlers.py @@ -2063,6 +2063,11 @@ def with_modifiers(modifiers, actions): return _with_modifiers(modifiers, actions) +def grid_cell(boxes, row, col, row_tolerance=10): + from je_auto_control.utils.executor.action_executor import _grid_cell + return _grid_cell(boxes, row, col, row_tolerance) + + def detect_drift(reference, current, threshold=0.25, bins=10): from je_auto_control.utils.executor.action_executor import _detect_drift return _detect_drift(reference, current, threshold, bins) diff --git a/test/unit_test/headless/test_grid_locator_batch.py b/test/unit_test/headless/test_grid_locator_batch.py new file mode 100644 index 00000000..e75ed40f --- /dev/null +++ b/test/unit_test/headless/test_grid_locator_batch.py @@ -0,0 +1,69 @@ +"""Headless tests for grid / table cell addressing. No Qt.""" +import json + +import je_auto_control as ac +from je_auto_control.utils.grid_locator import cluster_grid, locate_cell + +# a 2-row x 3-col grid, deliberately shuffled +_BOXES = [ + (210, 100, 20, 10), (10, 100, 20, 10), (110, 100, 20, 10), + (110, 200, 20, 10), (10, 200, 20, 10), (210, 200, 20, 10), +] + + +def test_cluster_groups_rows_and_orders_columns(): + grid = cluster_grid(_BOXES) + assert len(grid) == 2 and [len(r) for r in grid] == [3, 3] + assert [b[0] for b in grid[0]] == [10, 110, 210] # left-to-right + assert [b[1] for b in grid[0]] == [100, 100, 100] # top row + + +def test_locate_cell_returns_center(): + cell = locate_cell(_BOXES, 1, 2) + assert cell["found"] is True + assert cell["center"] == [220, 205] and cell["box"] == [210, 200, 20, 10] + assert cell["rows"] == 2 and cell["cols"] == 3 + + +def test_out_of_range(): + assert locate_cell(_BOXES, 5, 0)["found"] is False + assert locate_cell(_BOXES, 0, 9)["reason"] == "col out of range" + + +def test_row_tolerance_merges_near_y(): + # second row jitters by 4px in y -> still one row at tolerance 10 + jittered = [(10, 100, 20, 10), (110, 104, 20, 10), (210, 98, 20, 10)] + assert len(cluster_grid(jittered, row_tolerance=10)) == 1 + assert len(cluster_grid(jittered, row_tolerance=2)) >= 2 # splits + + +def test_empty_boxes(): + assert cluster_grid([]) == [] + assert locate_cell([], 0, 0)["found"] is False + + +# --- wiring --------------------------------------------------------------- + +def test_executor_round_trip(): + rec = ac.execute_action([[ + "AC_grid_cell", {"boxes": json.dumps([[10, 100, 20, 10], + [110, 100, 20, 10]]), + "row": 0, "col": 1}]]) + out = next(v for v in rec.values() if isinstance(v, dict)) + assert out["found"] is True and out["center"] == [120, 105] + + +def test_wiring(): + known = ac.executor.known_commands() + assert "AC_grid_cell" in set(known) + from je_auto_control.utils.mcp_server.tools import build_default_tool_registry + names = {t.name for t in build_default_tool_registry()} + assert "ac_grid_cell" in names + from je_auto_control.gui.script_builder.command_schema import _build_specs + specs = {s.command for s in _build_specs()} + assert "AC_grid_cell" in specs + + +def test_facade_exports(): + for attr in ("cluster_grid", "locate_cell"): + assert hasattr(ac, attr) and attr in ac.__all__