Add MSER text-region detection (find text without OCR)

Author: JE-Chen

README/WHATS_NEW_zh-CN.md

@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-23) — 免模型文字区检测(MSER)
+
+不跑 OCR 也能找出画面上文字的位置。完整参考:[`docs/source/Zh/doc/new_features/v140_features_doc.rst`](../docs/source/Zh/doc/new_features/v140_features_doc.rst)。
+
+- **`find_text_regions` / `find_text_lines`**(`AC_find_text_regions`、`AC_find_text_lines`):`shape_locator` 找矩形(不是文字)、`locate_text` 需要 OCR 引擎*以及*确切字串——两者都无法回答「哪里有*任何*文字?」。MSER 找出字元 / 词 / 行区块,让脚本能裁切候选框喂给 OCR(比全画面更快更准),或在未安装 OCR 相依时检测标签出现。`merge` 并集 MSER 逐字元的嵌套区域;`find_text_lines` 将字元归为逐行框;空白画面返回 `[]`。OpenCV 核心(`cv2.MSER_create`)、可注入 haystack → 无头可测。
+
 ## 本次更新 (2026-06-23) — HSV 色彩空间分割
 
 不论光照都能找出「任一色阶的红色」。完整参考:[`docs/source/Zh/doc/new_features/v139_features_doc.rst`](../docs/source/Zh/doc/new_features/v139_features_doc.rst)。

README/WHATS_NEW_zh-TW.md

@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-23) — 免模型文字區偵測(MSER)
+
+不跑 OCR 也能找出畫面上文字的位置。完整參考:[`docs/source/Zh/doc/new_features/v140_features_doc.rst`](../docs/source/Zh/doc/new_features/v140_features_doc.rst)。
+
+- **`find_text_regions` / `find_text_lines`**(`AC_find_text_regions`、`AC_find_text_lines`):`shape_locator` 找矩形(不是文字)、`locate_text` 需要 OCR 引擎*以及*確切字串——兩者都無法回答「哪裡有*任何*文字?」。MSER 找出字元 / 詞 / 行區塊,讓腳本能裁切候選框餵給 OCR(比全畫面更快更準),或在未安裝 OCR 相依時偵測標籤出現。`merge` 聯集 MSER 逐字元的巢狀區域;`find_text_lines` 將字元歸為逐行框;空白畫面回傳 `[]`。OpenCV 核心(`cv2.MSER_create`)、可注入 haystack → 無頭可測。
+
 ## 本次更新 (2026-06-23) — HSV 色彩空間分割
 
 不論光照都能找出「任一色階的紅色」。完整參考:[`docs/source/Zh/doc/new_features/v139_features_doc.rst`](../docs/source/Zh/doc/new_features/v139_features_doc.rst)。

WHATS_NEW.md

@@ -1,5 +1,11 @@
 # What's New — AutoControl
 
+## What's new (2026-06-23) — Model-Free Text-Region Detection (MSER)
+
+Find where text is on screen without running OCR. Full reference: [`docs/source/Eng/doc/new_features/v140_features_doc.rst`](docs/source/Eng/doc/new_features/v140_features_doc.rst).
+
+- **`find_text_regions` / `find_text_lines`** (`AC_find_text_regions`, `AC_find_text_lines`): `shape_locator` finds rectangles (not text) and `locate_text` needs an OCR engine *and* the exact string — neither answers "where is *any* text?". MSER finds the glyph/word/line blobs, so a script can crop candidate boxes to feed OCR (faster + more accurate than full-frame) or detect a label appeared with no OCR dependency. `merge` unions MSER's nested per-glyph regions; `find_text_lines` groups glyphs into per-line boxes; a blank screen returns `[]`. Base OpenCV (`cv2.MSER_create`), injectable haystack → headless-testable.
+
 ## What's new (2026-06-23) — HSV Colour-Space Segmentation
 
 Find "any shade of red" regardless of lighting. Full reference: [`docs/source/Eng/doc/new_features/v139_features_doc.rst`](docs/source/Eng/doc/new_features/v139_features_doc.rst).

docs/source/Eng/doc/new_features/v140_features_doc.rst

@@ -0,0 +1,46 @@
+Model-Free Text-Region Detection (MSER)
+=======================================
+
+``shape_locator`` finds rectangular contours (buttons / cards, not text) and
+``locate_text`` needs a Tesseract / Paddle engine *and* the exact string to search
+for. Neither answers "where is there *any* text on screen?" without running OCR or
+knowing the words. ``find_text_regions`` and ``find_text_lines`` use MSER (Maximally
+Stable Extremal Regions) to find the glyph / word / line blobs, so a script can crop
+the candidate text boxes to feed OCR — far faster and more accurate than full-frame
+OCR — or simply detect that a label appeared with no OCR dependency installed.
+
+Runs on an injectable ``haystack`` (ndarray / path / PIL), so it is headless-testable
+on synthetic arrays. ``cv2.MSER_create`` is base OpenCV (no contrib); OpenCV + NumPy
+come in via ``je_open_cv``. Imports no ``PySide6``.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import find_text_regions, find_text_lines
+
+    # Crop each text line and OCR just that strip.
+    for line in find_text_lines(y_tolerance=8):
+        text = locate_text  # ... feed the cropped region to your OCR of choice
+        print(line["x"], line["y"], line["width"], line["height"])
+
+    # Or per-glyph / per-word regions.
+    for box in find_text_regions(min_area=80):
+        highlight(box["x"], box["y"], box["width"], box["height"])
+
+``find_text_regions`` returns ``{x, y, width, height, area, center}`` per region,
+largest first; ``merge`` unions MSER's nested per-glyph detections, ``min_area`` /
+``max_area`` drop specks and page-sized blobs, ``max_aspect`` rejects long thin rules.
+``find_text_lines`` groups glyph boxes whose vertical centres are within
+``y_tolerance`` pixels into one box per line, top-to-bottom. A blank screen returns an
+empty list (the whole-frame extremal region is filtered out).
+
+Executor commands
+-----------------
+
+``AC_find_text_regions`` (``min_area`` / ``max_area`` / ``merge`` / ``max_aspect`` /
+``region`` → ``{count, regions}``) and ``AC_find_text_lines`` (``y_tolerance`` /
+``region`` → ``{count, lines}``). They are exposed as the MCP tools
+``ac_find_text_regions`` / ``ac_find_text_lines`` and as Script Builder commands
+under **Image**.

docs/source/Eng/eng_index.rst

@@ -162,6 +162,7 @@ Comprehensive guides for all AutoControl features.
    doc/new_features/v137_features_doc
    doc/new_features/v138_features_doc
    doc/new_features/v139_features_doc
+   doc/new_features/v140_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

docs/source/Zh/doc/new_features/v140_features_doc.rst

@@ -0,0 +1,37 @@
+免模型文字區偵測(MSER)
+========================
+
+``shape_locator`` 找的是矩形輪廓(按鈕 / 卡片,不是文字),``locate_text`` 需要 Tesseract / Paddle 引擎*以及*要
+搜尋的確切字串。兩者都無法在不跑 OCR、也不知道內容的情況下回答「畫面上哪裡有*任何*文字?」。``find_text_regions``
+與 ``find_text_lines`` 以 MSER(最大穩定極值區域)找出字元 / 詞 / 行的區塊,讓腳本能裁切候選文字框餵給 OCR
+(比全畫面 OCR 更快更準),或在未安裝 OCR 相依時單純偵測「某標籤出現了」。
+
+在可注入的 ``haystack``(ndarray / 路徑 / PIL)上執行,因此可對合成陣列做無頭測試。``cv2.MSER_create`` 屬於
+OpenCV 核心(不需 contrib);OpenCV + NumPy 透過 ``je_open_cv`` 引入。不匯入 ``PySide6``。
+
+無頭 API
+--------
+
+.. code-block:: python
+
+    from je_auto_control import find_text_regions, find_text_lines
+
+    # 裁切每一行文字,只對該長條做 OCR。
+    for line in find_text_lines(y_tolerance=8):
+        print(line["x"], line["y"], line["width"], line["height"])
+
+    # 或逐字元 / 逐詞的區域。
+    for box in find_text_regions(min_area=80):
+        highlight(box["x"], box["y"], box["width"], box["height"])
+
+``find_text_regions`` 為每個區域回傳 ``{x, y, width, height, area, center}``,由大到小;``merge`` 會聯集 MSER
+逐字元的巢狀偵測,``min_area`` / ``max_area`` 去除雜點與整頁大小的區塊,``max_aspect`` 排除長條狀的分隔線。
+``find_text_lines`` 將垂直中心在 ``y_tolerance`` 像素內的字元框歸為同一行,每行一個框、由上到下。空白畫面回傳空
+清單(整框極值區域已被濾除)。
+
+執行器命令
+----------
+
+``AC_find_text_regions``(``min_area`` / ``max_area`` / ``merge`` / ``max_aspect`` / ``region`` →
+``{count, regions}``)與 ``AC_find_text_lines``(``y_tolerance`` / ``region`` → ``{count, lines}``)。它們以
+MCP 工具 ``ac_find_text_regions`` / ``ac_find_text_lines`` 以及 Script Builder 中 **Image** 分類下的命令提供。

docs/source/Zh/zh_index.rst

@@ -162,6 +162,7 @@ AutoControl 所有功能的完整使用指南。
    doc/new_features/v137_features_doc
    doc/new_features/v138_features_doc
    doc/new_features/v139_features_doc
+   doc/new_features/v140_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

je_auto_control/__init__.py

@@ -320,6 +320,10 @@
 from je_auto_control.utils.hsv_segment import (
     color_mask, dominant_hue_regions, segment_hsv,
 )
+# Model-free on-screen text-region detection (MSER): regions and lines
+from je_auto_control.utils.text_regions import (
+    find_text_lines, find_text_regions,
+)
 # CI workflow annotations (GitHub Actions)
 from je_auto_control.utils.ci_annotations import (
     emit_annotations, format_annotation,
@@ -1156,6 +1160,8 @@ def start_autocontrol_gui(*args, **kwargs):
     "segment_hsv",
     "color_mask",
     "dominant_hue_regions",
+    "find_text_regions",
+    "find_text_lines",
     "emit_annotations", "format_annotation",
     "ClipboardHistory", "default_clipboard_history",
     "analyze_heal_log", "heal_stats", "scan_secrets",

je_auto_control/gui/script_builder/command_schema.py

@@ -416,6 +416,27 @@ def _add_image_specs(specs: List[CommandSpec]) -> None:
         ),
         description="Locate any shade of a hue, any brightness (handles red wrap).",
     ))
+    specs.append(CommandSpec(
+        "AC_find_text_regions", "Image", "Find Text Regions (MSER)",
+        fields=(
+            FieldSpec("min_area", FieldType.INT, optional=True, default=60),
+            FieldSpec("max_area", FieldType.INT, optional=True),
+            FieldSpec("merge", FieldType.BOOL, optional=True, default=True),
+            FieldSpec("max_aspect", FieldType.FLOAT, optional=True, default=12.0),
+            FieldSpec("region", FieldType.STRING, optional=True,
+                      placeholder=_REGION_PLACEHOLDER),
+        ),
+        description="Locate text regions without OCR (crop to feed an OCR engine).",
+    ))
+    specs.append(CommandSpec(
+        "AC_find_text_lines", "Image", "Find Text Lines (MSER)",
+        fields=(
+            FieldSpec("y_tolerance", FieldType.INT, optional=True, default=8),
+            FieldSpec("region", FieldType.STRING, optional=True,
+                      placeholder=_REGION_PLACEHOLDER),
+        ),
+        description="Locate horizontal text lines without OCR.",
+    ))
     specs.append(CommandSpec(
         "AC_fuse_elements", "Image", "Fuse Element Boxes",
         fields=(

je_auto_control/utils/executor/action_executor.py

@@ -3537,6 +3537,30 @@ def _dominant_hue_regions(hue: Any, hue_tol: Any = 10, sat_min: Any = 80,
             "best": boxes[0] if boxes else None}
 
 
+def _find_text_regions(min_area: Any = 60, max_area: Any = None, merge: Any = True,
+                       max_aspect: Any = 12.0, region: Any = None) -> Dict[str, Any]:
+    """Adapter: locate text/glyph regions on screen via MSER (no OCR)."""
+    import json
+    from je_auto_control.utils.text_regions import find_text_regions
+    if isinstance(region, str):
+        region = json.loads(region) if region.strip() else None
+    regions = find_text_regions(
+        region=region, min_area=int(min_area),
+        max_area=int(max_area) if max_area is not None else None,
+        merge=bool(merge), max_aspect=float(max_aspect))
+    return {"count": len(regions), "regions": regions}
+
+
+def _find_text_lines(y_tolerance: Any = 8, region: Any = None) -> Dict[str, Any]:
+    """Adapter: locate horizontal text lines on screen via MSER (no OCR)."""
+    import json
+    from je_auto_control.utils.text_regions import find_text_lines
+    if isinstance(region, str):
+        region = json.loads(region) if region.strip() else None
+    lines = find_text_lines(region=region, y_tolerance=int(y_tolerance))
+    return {"count": len(lines), "lines": lines}
+
+
 def _with_modifiers(modifiers: Any, actions: Any) -> Dict[str, Any]:
     """Adapter: run nested actions while modifier keys are held down."""
     import json
@@ -5271,6 +5295,8 @@ def __init__(self):
             "AC_reading_order": _reading_order,
             "AC_segment_hsv": _segment_hsv,
             "AC_dominant_hue_regions": _dominant_hue_regions,
+            "AC_find_text_regions": _find_text_regions,
+            "AC_find_text_lines": _find_text_lines,
             "AC_tile_rect": _tile_rect,
             "AC_grid_rects": _grid_rects,
             "AC_cascade_rects": _cascade_rects,