Add Hough line / grid / separator detection

README/WHATS_NEW_zh-CN.md

@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-23) — 线条 / 网格 / 分隔线检测(Hough)
+
+从原始像素找出表格网格线与 UI 分隔线。完整参考:[`docs/source/Zh/doc/new_features/v141_features_doc.rst`](../docs/source/Zh/doc/new_features/v141_features_doc.rst)。
+
+- **`find_lines` / `find_grid` / `find_separators`**(`AC_find_lines`、`AC_find_grid`、`AC_find_separators`):`grid_locator` 分群*已找到*的框、`shape_locator` 找封闭矩形——两者都无法从像素找出表格网格线或分隔线。Canny + 概率 Hough 检测直线段(分类水平/垂直/斜向),`find_grid` 还原 `{rows, cols, cells}` 让你定址「第 3 行、第 2 列」,`find_separators` 返回长分隔线坐标。可注入 haystack → 无头可测;OpenCV 核心(`cv2.HoughLinesP`)。
+
 ## 本次更新 (2026-06-23) — 免模型文字区检测(MSER)
 
 不跑 OCR 也能找出画面上文字的位置。完整参考:[`docs/source/Zh/doc/new_features/v140_features_doc.rst`](../docs/source/Zh/doc/new_features/v140_features_doc.rst)。

README/WHATS_NEW_zh-TW.md

@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-23) — 線條 / 網格 / 分隔線偵測(Hough)
+
+從原始像素找出表格格線與 UI 分隔線。完整參考:[`docs/source/Zh/doc/new_features/v141_features_doc.rst`](../docs/source/Zh/doc/new_features/v141_features_doc.rst)。
+
+- **`find_lines` / `find_grid` / `find_separators`**(`AC_find_lines`、`AC_find_grid`、`AC_find_separators`):`grid_locator` 分群*已找到*的框、`shape_locator` 找封閉矩形——兩者都無法從像素找出表格格線或分隔線。Canny + 機率 Hough 偵測直線段(分類水平/垂直/斜向),`find_grid` 還原 `{rows, cols, cells}` 讓你定址「第 3 列、第 2 欄」,`find_separators` 回傳長分隔線座標。可注入 haystack → 無頭可測;OpenCV 核心(`cv2.HoughLinesP`)。
+
 ## 本次更新 (2026-06-23) — 免模型文字區偵測(MSER)
 
 不跑 OCR 也能找出畫面上文字的位置。完整參考:[`docs/source/Zh/doc/new_features/v140_features_doc.rst`](../docs/source/Zh/doc/new_features/v140_features_doc.rst)。

WHATS_NEW.md

@@ -1,5 +1,11 @@
 # What's New — AutoControl
 
+## What's new (2026-06-23) — Line / Grid / Separator Detection (Hough)
+
+Find table grid lines and UI dividers from raw pixels. Full reference: [`docs/source/Eng/doc/new_features/v141_features_doc.rst`](docs/source/Eng/doc/new_features/v141_features_doc.rst).
+
+- **`find_lines` / `find_grid` / `find_separators`** (`AC_find_lines`, `AC_find_grid`, `AC_find_separators`): `grid_locator` clusters *already-found* boxes and `shape_locator` finds closed rectangles — neither finds a table's ruling lines or a divider from pixels. Canny + probabilistic Hough detects straight segments (classified horizontal/vertical/diagonal), `find_grid` recovers `{rows, cols, cells}` so you can address "row 3, col 2", and `find_separators` returns the coordinates of long dividers. Injectable haystack → headless-testable; base OpenCV (`cv2.HoughLinesP`).
+
 ## What's new (2026-06-23) — Model-Free Text-Region Detection (MSER)
 
 Find where text is on screen without running OCR. Full reference: [`docs/source/Eng/doc/new_features/v140_features_doc.rst`](docs/source/Eng/doc/new_features/v140_features_doc.rst).

docs/source/Eng/doc/new_features/v141_features_doc.rst

@@ -0,0 +1,48 @@
+Line / Grid / Separator Detection (Hough)
+=========================================
+
+``grid_locator`` clusters *already-found* element boxes into a grid; it cannot find
+the ruling lines of a table / spreadsheet or a UI divider from raw pixels, and
+``shape_locator`` only finds closed rectangles. ``find_lines``, ``find_grid`` and
+``find_separators`` detect straight line segments via Canny + the probabilistic Hough
+transform, classify them horizontal / vertical / diagonal, recover a table's row and
+column coordinates (and cells), and return the positions of long divider lines — so a
+script can address "row 3, column 2" or split a panel at its separators with no
+template.
+
+Runs on an injectable ``haystack`` (ndarray / path / PIL), so it is headless-testable
+on synthetic arrays. ``cv2.HoughLinesP`` is base OpenCV; OpenCV + NumPy come in via
+``je_open_cv``. Imports no ``PySide6``.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import find_lines, find_grid, find_separators
+
+    for seg in find_lines(min_length=80, orientation="vertical"):
+        print(seg["x1"], seg["y1"], seg["x2"], seg["y2"], seg["length"])
+
+    grid = find_grid(min_length=120)
+    cell = grid["cells"][0]                     # {x, y, width, height} of row 0, col 0
+    click(cell["x"] + cell["width"] // 2, cell["y"] + cell["height"] // 2)
+
+    dividers = find_separators(axis="horizontal")   # [y0, y1, ...] of the rules
+
+``find_lines`` returns ``{x1, y1, x2, y2, angle, length, orientation}`` per segment,
+longest first; pass ``orientation`` other than ``any`` to keep only that kind.
+``find_grid`` clusters the horizontal rules into row coordinates and the vertical rules
+into columns, returning ``{rows, cols, cells}`` (cells are the rectangles between
+consecutive rules). ``find_separators`` returns the merged coordinates of long divider
+lines along ``axis``. A blank screen yields no lines / cells.
+
+Executor commands
+-----------------
+
+``AC_find_lines`` (``min_length`` / ``max_gap`` / ``orientation`` / ``region`` →
+``{count, lines}``), ``AC_find_grid`` (``min_length`` / ``tol`` / ``region`` →
+``{rows, cols, cells}``) and ``AC_find_separators`` (``axis`` / ``min_length`` /
+``tol`` / ``region`` → ``{count, axis, coordinates}``). They are exposed as the MCP
+tools ``ac_find_lines`` / ``ac_find_grid`` / ``ac_find_separators`` and as Script
+Builder commands under **Image**.

docs/source/Eng/eng_index.rst

@@ -163,6 +163,7 @@ Comprehensive guides for all AutoControl features.
    doc/new_features/v138_features_doc
    doc/new_features/v139_features_doc
    doc/new_features/v140_features_doc
+   doc/new_features/v141_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

docs/source/Zh/doc/new_features/v141_features_doc.rst

@@ -0,0 +1,38 @@
+線條 / 網格 / 分隔線偵測(Hough)
+==================================
+
+``grid_locator`` 把*已找到的*元素框分群成網格;它無法從原始像素找出表格 / 試算表的格線或 UI 分隔線,而
+``shape_locator`` 只找封閉矩形。``find_lines``、``find_grid`` 與 ``find_separators`` 以 Canny + 機率 Hough 轉換
+偵測直線段、分類為水平 / 垂直 / 斜向、還原表格的列與欄座標(及儲存格),並回傳長分隔線的位置——讓腳本能在無模板下
+定址「第 3 列、第 2 欄」或在分隔處切分面板。
+
+在可注入的 ``haystack``(ndarray / 路徑 / PIL)上執行,因此可對合成陣列做無頭測試。``cv2.HoughLinesP`` 屬於
+OpenCV 核心;OpenCV + NumPy 透過 ``je_open_cv`` 引入。不匯入 ``PySide6``。
+
+無頭 API
+--------
+
+.. code-block:: python
+
+    from je_auto_control import find_lines, find_grid, find_separators
+
+    for seg in find_lines(min_length=80, orientation="vertical"):
+        print(seg["x1"], seg["y1"], seg["x2"], seg["y2"], seg["length"])
+
+    grid = find_grid(min_length=120)
+    cell = grid["cells"][0]                     # 第 0 列、第 0 欄的 {x, y, width, height}
+    click(cell["x"] + cell["width"] // 2, cell["y"] + cell["height"] // 2)
+
+    dividers = find_separators(axis="horizontal")   # 各格線的 [y0, y1, ...]
+
+``find_lines`` 為每段回傳 ``{x1, y1, x2, y2, angle, length, orientation}``,最長者優先;傳入非 ``any`` 的
+``orientation`` 只保留該類。``find_grid`` 將水平格線分群為列座標、垂直格線分群為欄,回傳 ``{rows, cols, cells}``
+(儲存格為相鄰格線之間的矩形)。``find_separators`` 回傳沿 ``axis`` 的長分隔線合併後座標。空白畫面不產生線條 / 儲存格。
+
+執行器命令
+----------
+
+``AC_find_lines``(``min_length`` / ``max_gap`` / ``orientation`` / ``region`` → ``{count, lines}``)、
+``AC_find_grid``(``min_length`` / ``tol`` / ``region`` → ``{rows, cols, cells}``)與 ``AC_find_separators``
+(``axis`` / ``min_length`` / ``tol`` / ``region`` → ``{count, axis, coordinates}``)。它們以 MCP 工具
+``ac_find_lines`` / ``ac_find_grid`` / ``ac_find_separators`` 以及 Script Builder 中 **Image** 分類下的命令提供。

docs/source/Zh/zh_index.rst

@@ -163,6 +163,7 @@ AutoControl 所有功能的完整使用指南。
    doc/new_features/v138_features_doc
    doc/new_features/v139_features_doc
    doc/new_features/v140_features_doc
+   doc/new_features/v141_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

je_auto_control/__init__.py

@@ -324,6 +324,10 @@
 from je_auto_control.utils.text_regions import (
     find_text_lines, find_text_regions,
 )
+# Line / grid / separator detection on raw pixels (Hough transform)
+from je_auto_control.utils.edge_lines import (
+    find_grid, find_lines, find_separators,
+)
 # CI workflow annotations (GitHub Actions)
 from je_auto_control.utils.ci_annotations import (
     emit_annotations, format_annotation,
@@ -1162,6 +1166,9 @@ def start_autocontrol_gui(*args, **kwargs):
     "dominant_hue_regions",
     "find_text_regions",
     "find_text_lines",
+    "find_lines",
+    "find_grid",
+    "find_separators",
     "emit_annotations", "format_annotation",
     "ClipboardHistory", "default_clipboard_history",
     "analyze_heal_log", "heal_stats", "scan_secrets",

je_auto_control/gui/script_builder/command_schema.py

@@ -437,6 +437,40 @@ def _add_image_specs(specs: List[CommandSpec]) -> None:
         ),
         description="Locate horizontal text lines without OCR.",
     ))
+    specs.append(CommandSpec(
+        "AC_find_lines", "Image", "Find Lines (Hough)",
+        fields=(
+            FieldSpec("min_length", FieldType.INT, optional=True, default=80),
+            FieldSpec("max_gap", FieldType.INT, optional=True, default=10),
+            FieldSpec("orientation", FieldType.ENUM, optional=True, default="any",
+                      choices=("any", "horizontal", "vertical", "diagonal")),
+            FieldSpec("region", FieldType.STRING, optional=True,
+                      placeholder=_REGION_PLACEHOLDER),
+        ),
+        description="Detect straight line segments on raw pixels.",
+    ))
+    specs.append(CommandSpec(
+        "AC_find_grid", "Image", "Find Table Grid",
+        fields=(
+            FieldSpec("min_length", FieldType.INT, optional=True, default=120),
+            FieldSpec("tol", FieldType.INT, optional=True, default=10),
+            FieldSpec("region", FieldType.STRING, optional=True,
+                      placeholder=_REGION_PLACEHOLDER),
+        ),
+        description="Recover a table's rows / columns / cells from its lines.",
+    ))
+    specs.append(CommandSpec(
+        "AC_find_separators", "Image", "Find Separator Lines",
+        fields=(
+            FieldSpec("axis", FieldType.ENUM, optional=True, default="horizontal",
+                      choices=("horizontal", "vertical")),
+            FieldSpec("min_length", FieldType.INT, optional=True, default=120),
+            FieldSpec("tol", FieldType.INT, optional=True, default=10),
+            FieldSpec("region", FieldType.STRING, optional=True,
+                      placeholder=_REGION_PLACEHOLDER),
+        ),
+        description="Coordinates of long divider lines along an axis.",
+    ))
     specs.append(CommandSpec(
         "AC_fuse_elements", "Image", "Fuse Element Boxes",
         fields=(

je_auto_control/utils/edge_lines/__init__.py

@@ -0,0 +1,6 @@
+"""Line / grid / separator detection on raw pixels (Hough transform)."""
+from je_auto_control.utils.edge_lines.edge_lines import (
+    find_grid, find_lines, find_separators,
+)
+
+__all__ = ["find_grid", "find_lines", "find_separators"]

je_auto_control/utils/edge_lines/edge_lines.py

@@ -0,0 +1,113 @@
+"""Line, grid and separator detection on raw pixels (Hough) — tables and dividers.
+
+``grid_locator`` clusters *already-found* element boxes into a grid; it cannot find
+the ruling lines of a table / spreadsheet or a UI divider from raw pixels, and
+``shape_locator`` only finds closed rectangles. This detects straight line segments
+via Canny + the probabilistic Hough transform, classifies them horizontal / vertical /
+diagonal, recovers a table's row/column coordinates and cells, and returns the
+positions of long divider lines — so a script can address "row 3, column 2" or split a
+panel at its separators without templates.
+
+Runs on an injectable ``haystack`` (ndarray / path / PIL), so it is headless-testable
+on synthetic arrays. ``cv2.HoughLinesP`` is base OpenCV; OpenCV + NumPy come in via
+``je_open_cv``. Imports no ``PySide6``.
+"""
+import math
+from typing import Any, Dict, List, Optional, Sequence
+
+from je_auto_control.utils.visual_match.visual_match import _haystack_gray
+
+ImageSource = Any
+_CANNY_LOW = 50
+_CANNY_HIGH = 150
+
+
+def _orientation(angle: float) -> str:
+    a = abs(angle) % 180
+    if a < 10 or a > 170:
+        return "horizontal"
+    if 80 < a < 100:
+        return "vertical"
+    return "diagonal"
+
+
+def _segments(gray, min_length: int, max_gap: int):
+    import cv2
+    import numpy as np
+    edges = cv2.Canny(gray, _CANNY_LOW, _CANNY_HIGH)
+    return cv2.HoughLinesP(edges, 1, np.pi / 180, threshold=50,
+                           minLineLength=int(min_length), maxLineGap=int(max_gap))
+
+
+def find_lines(haystack: Optional[ImageSource] = None, *,
+               region: Optional[Sequence[int]] = None, min_length: int = 80,
+               max_gap: int = 10, orientation: str = "any"
+               ) -> List[Dict[str, Any]]:
+    """Return straight line segments on screen, longest first.
+
+    Each is ``{x1, y1, x2, y2, angle, length, orientation}`` where ``orientation`` is
+    ``horizontal`` / ``vertical`` / ``diagonal``. Pass ``orientation`` other than
+    ``any`` to keep only that kind. ``min_length`` / ``max_gap`` tune the Hough probe.
+    """
+    segments = _segments(_haystack_gray(haystack, region), int(min_length),
+                         int(max_gap))
+    out: List[Dict[str, Any]] = []
+    if segments is None:
+        return out
+    for x1, y1, x2, y2 in segments[:, 0]:
+        angle = math.degrees(math.atan2(int(y2) - int(y1), int(x2) - int(x1)))
+        kind = _orientation(angle)
+        if orientation not in ("any", kind):
+            continue
+        out.append({"x1": int(x1), "y1": int(y1), "x2": int(x2), "y2": int(y2),
+                    "angle": round(angle, 1),
+                    "length": round(math.hypot(x2 - x1, y2 - y1), 1),
+                    "orientation": kind})
+    out.sort(key=lambda seg: seg["length"], reverse=True)
+    return out
+
+
+def _cluster(values: Sequence[int], tol: int) -> List[int]:
+    """Group near-equal coordinates and return each cluster's mean (sorted)."""
+    groups: List[List[int]] = []
+    for value in sorted(values):
+        if groups and value - groups[-1][-1] <= tol:
+            groups[-1].append(value)
+        else:
+            groups.append([value])
+    return [int(round(sum(group) / len(group))) for group in groups]
+
+
+def find_separators(haystack: Optional[ImageSource] = None, *,
+                    region: Optional[Sequence[int]] = None,
+                    axis: str = "horizontal", min_length: int = 120,
+                    tol: int = 10) -> List[int]:
+    """Return the coordinates of long divider lines along ``axis``.
+
+    ``axis="horizontal"`` returns the ``y`` of each horizontal rule (top-to-bottom);
+    ``axis="vertical"`` returns the ``x`` of each vertical rule. Near-equal lines are
+    merged within ``tol`` pixels.
+    """
+    lines = find_lines(haystack, region=region, min_length=int(min_length),
+                       orientation=axis)
+    coords = [seg["y1"] if axis == "horizontal" else seg["x1"] for seg in lines]
+    return _cluster(coords, int(tol))
+
+
+def find_grid(haystack: Optional[ImageSource] = None, *,
+              region: Optional[Sequence[int]] = None, min_length: int = 120,
+              tol: int = 10) -> Dict[str, Any]:
+    """Recover a table's grid: ``{rows: [y…], cols: [x…], cells: [{x,y,width,height}]}``.
+
+    Horizontal rules give the row coordinates, vertical rules the columns; the cells
+    are the rectangles between consecutive rules. ``min_length`` filters short edges.
+    """
+    lines = find_lines(haystack, region=region, min_length=int(min_length))
+    rows = _cluster([seg["y1"] for seg in lines
+                     if seg["orientation"] == "horizontal"], int(tol))
+    cols = _cluster([seg["x1"] for seg in lines
+                     if seg["orientation"] == "vertical"], int(tol))
+    cells = [{"x": cols[i], "y": rows[j], "width": cols[i + 1] - cols[i],
+              "height": rows[j + 1] - rows[j]}
+             for j in range(len(rows) - 1) for i in range(len(cols) - 1)]
+    return {"rows": rows, "cols": cols, "cells": cells}

je_auto_control/utils/executor/action_executor.py

@@ -3561,6 +3561,40 @@ def _find_text_lines(y_tolerance: Any = 8, region: Any = None) -> Dict[str, Any]
     return {"count": len(lines), "lines": lines}
 
 
+def _find_lines(min_length: Any = 80, max_gap: Any = 10, orientation: str = "any",
+                region: Any = None) -> Dict[str, Any]:
+    """Adapter: detect straight line segments on screen (Hough)."""
+    import json
+    from je_auto_control.utils.edge_lines import find_lines
+    if isinstance(region, str):
+        region = json.loads(region) if region.strip() else None
+    lines = find_lines(region=region, min_length=int(min_length),
+                       max_gap=int(max_gap), orientation=str(orientation))
+    return {"count": len(lines), "lines": lines}
+
+
+def _find_grid(min_length: Any = 120, tol: Any = 10,
+               region: Any = None) -> Dict[str, Any]:
+    """Adapter: recover a table grid (rows / cols / cells) from screen lines."""
+    import json
+    from je_auto_control.utils.edge_lines import find_grid
+    if isinstance(region, str):
+        region = json.loads(region) if region.strip() else None
+    return find_grid(region=region, min_length=int(min_length), tol=int(tol))
+
+
+def _find_separators(axis: str = "horizontal", min_length: Any = 120, tol: Any = 10,
+                     region: Any = None) -> Dict[str, Any]:
+    """Adapter: coordinates of long divider lines along an axis."""
+    import json
+    from je_auto_control.utils.edge_lines import find_separators
+    if isinstance(region, str):
+        region = json.loads(region) if region.strip() else None
+    coords = find_separators(region=region, axis=str(axis),
+                             min_length=int(min_length), tol=int(tol))
+    return {"count": len(coords), "axis": str(axis), "coordinates": coords}
+
+
 def _with_modifiers(modifiers: Any, actions: Any) -> Dict[str, Any]:
     """Adapter: run nested actions while modifier keys are held down."""
     import json
@@ -5297,6 +5331,9 @@ def __init__(self):
             "AC_dominant_hue_regions": _dominant_hue_regions,
             "AC_find_text_regions": _find_text_regions,
             "AC_find_text_lines": _find_text_lines,
+            "AC_find_lines": _find_lines,
+            "AC_find_grid": _find_grid,
+            "AC_find_separators": _find_separators,
             "AC_tile_rect": _tile_rect,
             "AC_grid_rects": _grid_rects,
             "AC_cascade_rects": _cascade_rects,