diff --git a/README/WHATS_NEW_zh-CN.md b/README/WHATS_NEW_zh-CN.md index 75ffabfd..96117a60 100644 --- a/README/WHATS_NEW_zh-CN.md +++ b/README/WHATS_NEW_zh-CN.md @@ -1,5 +1,11 @@ # 本次更新 — AutoControl +## 本次更新 (2026-06-23) — 免模型文字区检测(MSER) + +不跑 OCR 也能找出画面上文字的位置。完整参考:[`docs/source/Zh/doc/new_features/v140_features_doc.rst`](../docs/source/Zh/doc/new_features/v140_features_doc.rst)。 + +- **`find_text_regions` / `find_text_lines`**(`AC_find_text_regions`、`AC_find_text_lines`):`shape_locator` 找矩形(不是文字)、`locate_text` 需要 OCR 引擎*以及*确切字串——两者都无法回答「哪里有*任何*文字?」。MSER 找出字元 / 词 / 行区块,让脚本能裁切候选框喂给 OCR(比全画面更快更准),或在未安装 OCR 相依时检测标签出现。`merge` 并集 MSER 逐字元的嵌套区域;`find_text_lines` 将字元归为逐行框;空白画面返回 `[]`。OpenCV 核心(`cv2.MSER_create`)、可注入 haystack → 无头可测。 + ## 本次更新 (2026-06-23) — HSV 色彩空间分割 不论光照都能找出「任一色阶的红色」。完整参考:[`docs/source/Zh/doc/new_features/v139_features_doc.rst`](../docs/source/Zh/doc/new_features/v139_features_doc.rst)。 diff --git a/README/WHATS_NEW_zh-TW.md b/README/WHATS_NEW_zh-TW.md index 97a0d8eb..b1c0b8f3 100644 --- a/README/WHATS_NEW_zh-TW.md +++ b/README/WHATS_NEW_zh-TW.md @@ -1,5 +1,11 @@ # 本次更新 — AutoControl +## 本次更新 (2026-06-23) — 免模型文字區偵測(MSER) + +不跑 OCR 也能找出畫面上文字的位置。完整參考:[`docs/source/Zh/doc/new_features/v140_features_doc.rst`](../docs/source/Zh/doc/new_features/v140_features_doc.rst)。 + +- **`find_text_regions` / `find_text_lines`**(`AC_find_text_regions`、`AC_find_text_lines`):`shape_locator` 找矩形(不是文字)、`locate_text` 需要 OCR 引擎*以及*確切字串——兩者都無法回答「哪裡有*任何*文字?」。MSER 找出字元 / 詞 / 行區塊,讓腳本能裁切候選框餵給 OCR(比全畫面更快更準),或在未安裝 OCR 相依時偵測標籤出現。`merge` 聯集 MSER 逐字元的巢狀區域;`find_text_lines` 將字元歸為逐行框;空白畫面回傳 `[]`。OpenCV 核心(`cv2.MSER_create`)、可注入 haystack → 無頭可測。 + ## 本次更新 (2026-06-23) — HSV 色彩空間分割 不論光照都能找出「任一色階的紅色」。完整參考:[`docs/source/Zh/doc/new_features/v139_features_doc.rst`](../docs/source/Zh/doc/new_features/v139_features_doc.rst)。 diff --git a/WHATS_NEW.md b/WHATS_NEW.md index bdd52c75..464bde5d 100644 --- a/WHATS_NEW.md +++ b/WHATS_NEW.md @@ -1,5 +1,11 @@ # What's New — AutoControl +## What's new (2026-06-23) — Model-Free Text-Region Detection (MSER) + +Find where text is on screen without running OCR. Full reference: [`docs/source/Eng/doc/new_features/v140_features_doc.rst`](docs/source/Eng/doc/new_features/v140_features_doc.rst). + +- **`find_text_regions` / `find_text_lines`** (`AC_find_text_regions`, `AC_find_text_lines`): `shape_locator` finds rectangles (not text) and `locate_text` needs an OCR engine *and* the exact string — neither answers "where is *any* text?". MSER finds the glyph/word/line blobs, so a script can crop candidate boxes to feed OCR (faster + more accurate than full-frame) or detect a label appeared with no OCR dependency. `merge` unions MSER's nested per-glyph regions; `find_text_lines` groups glyphs into per-line boxes; a blank screen returns `[]`. Base OpenCV (`cv2.MSER_create`), injectable haystack → headless-testable. + ## What's new (2026-06-23) — HSV Colour-Space Segmentation Find "any shade of red" regardless of lighting. Full reference: [`docs/source/Eng/doc/new_features/v139_features_doc.rst`](docs/source/Eng/doc/new_features/v139_features_doc.rst). diff --git a/docs/source/Eng/doc/new_features/v140_features_doc.rst b/docs/source/Eng/doc/new_features/v140_features_doc.rst new file mode 100644 index 00000000..075a9ecc --- /dev/null +++ b/docs/source/Eng/doc/new_features/v140_features_doc.rst @@ -0,0 +1,46 @@ +Model-Free Text-Region Detection (MSER) +======================================= + +``shape_locator`` finds rectangular contours (buttons / cards, not text) and +``locate_text`` needs a Tesseract / Paddle engine *and* the exact string to search +for. Neither answers "where is there *any* text on screen?" without running OCR or +knowing the words. ``find_text_regions`` and ``find_text_lines`` use MSER (Maximally +Stable Extremal Regions) to find the glyph / word / line blobs, so a script can crop +the candidate text boxes to feed OCR — far faster and more accurate than full-frame +OCR — or simply detect that a label appeared with no OCR dependency installed. + +Runs on an injectable ``haystack`` (ndarray / path / PIL), so it is headless-testable +on synthetic arrays. ``cv2.MSER_create`` is base OpenCV (no contrib); OpenCV + NumPy +come in via ``je_open_cv``. Imports no ``PySide6``. + +Headless API +------------ + +.. code-block:: python + + from je_auto_control import find_text_regions, find_text_lines + + # Crop each text line and OCR just that strip. + for line in find_text_lines(y_tolerance=8): + text = locate_text # ... feed the cropped region to your OCR of choice + print(line["x"], line["y"], line["width"], line["height"]) + + # Or per-glyph / per-word regions. + for box in find_text_regions(min_area=80): + highlight(box["x"], box["y"], box["width"], box["height"]) + +``find_text_regions`` returns ``{x, y, width, height, area, center}`` per region, +largest first; ``merge`` unions MSER's nested per-glyph detections, ``min_area`` / +``max_area`` drop specks and page-sized blobs, ``max_aspect`` rejects long thin rules. +``find_text_lines`` groups glyph boxes whose vertical centres are within +``y_tolerance`` pixels into one box per line, top-to-bottom. A blank screen returns an +empty list (the whole-frame extremal region is filtered out). + +Executor commands +----------------- + +``AC_find_text_regions`` (``min_area`` / ``max_area`` / ``merge`` / ``max_aspect`` / +``region`` → ``{count, regions}``) and ``AC_find_text_lines`` (``y_tolerance`` / +``region`` → ``{count, lines}``). They are exposed as the MCP tools +``ac_find_text_regions`` / ``ac_find_text_lines`` and as Script Builder commands +under **Image**. diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst index 71f07108..ba50be0a 100644 --- a/docs/source/Eng/eng_index.rst +++ b/docs/source/Eng/eng_index.rst @@ -162,6 +162,7 @@ Comprehensive guides for all AutoControl features. doc/new_features/v137_features_doc doc/new_features/v138_features_doc doc/new_features/v139_features_doc + doc/new_features/v140_features_doc doc/ocr_backends/ocr_backends_doc doc/observability/observability_doc doc/operations_layer/operations_layer_doc diff --git a/docs/source/Zh/doc/new_features/v140_features_doc.rst b/docs/source/Zh/doc/new_features/v140_features_doc.rst new file mode 100644 index 00000000..dfc136bc --- /dev/null +++ b/docs/source/Zh/doc/new_features/v140_features_doc.rst @@ -0,0 +1,37 @@ +免模型文字區偵測(MSER) +======================== + +``shape_locator`` 找的是矩形輪廓(按鈕 / 卡片,不是文字),``locate_text`` 需要 Tesseract / Paddle 引擎*以及*要 +搜尋的確切字串。兩者都無法在不跑 OCR、也不知道內容的情況下回答「畫面上哪裡有*任何*文字?」。``find_text_regions`` +與 ``find_text_lines`` 以 MSER(最大穩定極值區域)找出字元 / 詞 / 行的區塊,讓腳本能裁切候選文字框餵給 OCR +(比全畫面 OCR 更快更準),或在未安裝 OCR 相依時單純偵測「某標籤出現了」。 + +在可注入的 ``haystack``(ndarray / 路徑 / PIL)上執行,因此可對合成陣列做無頭測試。``cv2.MSER_create`` 屬於 +OpenCV 核心(不需 contrib);OpenCV + NumPy 透過 ``je_open_cv`` 引入。不匯入 ``PySide6``。 + +無頭 API +-------- + +.. code-block:: python + + from je_auto_control import find_text_regions, find_text_lines + + # 裁切每一行文字,只對該長條做 OCR。 + for line in find_text_lines(y_tolerance=8): + print(line["x"], line["y"], line["width"], line["height"]) + + # 或逐字元 / 逐詞的區域。 + for box in find_text_regions(min_area=80): + highlight(box["x"], box["y"], box["width"], box["height"]) + +``find_text_regions`` 為每個區域回傳 ``{x, y, width, height, area, center}``,由大到小;``merge`` 會聯集 MSER +逐字元的巢狀偵測,``min_area`` / ``max_area`` 去除雜點與整頁大小的區塊,``max_aspect`` 排除長條狀的分隔線。 +``find_text_lines`` 將垂直中心在 ``y_tolerance`` 像素內的字元框歸為同一行,每行一個框、由上到下。空白畫面回傳空 +清單(整框極值區域已被濾除)。 + +執行器命令 +---------- + +``AC_find_text_regions``(``min_area`` / ``max_area`` / ``merge`` / ``max_aspect`` / ``region`` → +``{count, regions}``)與 ``AC_find_text_lines``(``y_tolerance`` / ``region`` → ``{count, lines}``)。它們以 +MCP 工具 ``ac_find_text_regions`` / ``ac_find_text_lines`` 以及 Script Builder 中 **Image** 分類下的命令提供。 diff --git a/docs/source/Zh/zh_index.rst b/docs/source/Zh/zh_index.rst index 011e0a2e..fd580f6f 100644 --- a/docs/source/Zh/zh_index.rst +++ b/docs/source/Zh/zh_index.rst @@ -162,6 +162,7 @@ AutoControl 所有功能的完整使用指南。 doc/new_features/v137_features_doc doc/new_features/v138_features_doc doc/new_features/v139_features_doc + doc/new_features/v140_features_doc doc/ocr_backends/ocr_backends_doc doc/observability/observability_doc doc/operations_layer/operations_layer_doc diff --git a/je_auto_control/__init__.py b/je_auto_control/__init__.py index e375fba4..c21890fc 100644 --- a/je_auto_control/__init__.py +++ b/je_auto_control/__init__.py @@ -320,6 +320,10 @@ from je_auto_control.utils.hsv_segment import ( color_mask, dominant_hue_regions, segment_hsv, ) +# Model-free on-screen text-region detection (MSER): regions and lines +from je_auto_control.utils.text_regions import ( + find_text_lines, find_text_regions, +) # CI workflow annotations (GitHub Actions) from je_auto_control.utils.ci_annotations import ( emit_annotations, format_annotation, @@ -1156,6 +1160,8 @@ def start_autocontrol_gui(*args, **kwargs): "segment_hsv", "color_mask", "dominant_hue_regions", + "find_text_regions", + "find_text_lines", "emit_annotations", "format_annotation", "ClipboardHistory", "default_clipboard_history", "analyze_heal_log", "heal_stats", "scan_secrets", diff --git a/je_auto_control/gui/script_builder/command_schema.py b/je_auto_control/gui/script_builder/command_schema.py index 3fb97a93..7d0b26e1 100644 --- a/je_auto_control/gui/script_builder/command_schema.py +++ b/je_auto_control/gui/script_builder/command_schema.py @@ -416,6 +416,27 @@ def _add_image_specs(specs: List[CommandSpec]) -> None: ), description="Locate any shade of a hue, any brightness (handles red wrap).", )) + specs.append(CommandSpec( + "AC_find_text_regions", "Image", "Find Text Regions (MSER)", + fields=( + FieldSpec("min_area", FieldType.INT, optional=True, default=60), + FieldSpec("max_area", FieldType.INT, optional=True), + FieldSpec("merge", FieldType.BOOL, optional=True, default=True), + FieldSpec("max_aspect", FieldType.FLOAT, optional=True, default=12.0), + FieldSpec("region", FieldType.STRING, optional=True, + placeholder=_REGION_PLACEHOLDER), + ), + description="Locate text regions without OCR (crop to feed an OCR engine).", + )) + specs.append(CommandSpec( + "AC_find_text_lines", "Image", "Find Text Lines (MSER)", + fields=( + FieldSpec("y_tolerance", FieldType.INT, optional=True, default=8), + FieldSpec("region", FieldType.STRING, optional=True, + placeholder=_REGION_PLACEHOLDER), + ), + description="Locate horizontal text lines without OCR.", + )) specs.append(CommandSpec( "AC_fuse_elements", "Image", "Fuse Element Boxes", fields=( diff --git a/je_auto_control/utils/executor/action_executor.py b/je_auto_control/utils/executor/action_executor.py index 21d1c95e..d25a0559 100644 --- a/je_auto_control/utils/executor/action_executor.py +++ b/je_auto_control/utils/executor/action_executor.py @@ -3537,6 +3537,30 @@ def _dominant_hue_regions(hue: Any, hue_tol: Any = 10, sat_min: Any = 80, "best": boxes[0] if boxes else None} +def _find_text_regions(min_area: Any = 60, max_area: Any = None, merge: Any = True, + max_aspect: Any = 12.0, region: Any = None) -> Dict[str, Any]: + """Adapter: locate text/glyph regions on screen via MSER (no OCR).""" + import json + from je_auto_control.utils.text_regions import find_text_regions + if isinstance(region, str): + region = json.loads(region) if region.strip() else None + regions = find_text_regions( + region=region, min_area=int(min_area), + max_area=int(max_area) if max_area is not None else None, + merge=bool(merge), max_aspect=float(max_aspect)) + return {"count": len(regions), "regions": regions} + + +def _find_text_lines(y_tolerance: Any = 8, region: Any = None) -> Dict[str, Any]: + """Adapter: locate horizontal text lines on screen via MSER (no OCR).""" + import json + from je_auto_control.utils.text_regions import find_text_lines + if isinstance(region, str): + region = json.loads(region) if region.strip() else None + lines = find_text_lines(region=region, y_tolerance=int(y_tolerance)) + return {"count": len(lines), "lines": lines} + + def _with_modifiers(modifiers: Any, actions: Any) -> Dict[str, Any]: """Adapter: run nested actions while modifier keys are held down.""" import json @@ -5271,6 +5295,8 @@ def __init__(self): "AC_reading_order": _reading_order, "AC_segment_hsv": _segment_hsv, "AC_dominant_hue_regions": _dominant_hue_regions, + "AC_find_text_regions": _find_text_regions, + "AC_find_text_lines": _find_text_lines, "AC_tile_rect": _tile_rect, "AC_grid_rects": _grid_rects, "AC_cascade_rects": _cascade_rects, diff --git a/je_auto_control/utils/mcp_server/tools/_factories.py b/je_auto_control/utils/mcp_server/tools/_factories.py index 476bb2f9..b7477f98 100644 --- a/je_auto_control/utils/mcp_server/tools/_factories.py +++ b/je_auto_control/utils/mcp_server/tools/_factories.py @@ -2936,6 +2936,40 @@ def hsv_segment_tools() -> List[MCPTool]: ] +def text_regions_tools() -> List[MCPTool]: + return [ + MCPTool( + name="ac_find_text_regions", + description=("Locate text/glyph regions on screen via MSER — NO OCR " + "engine or known string needed. Returns {count, regions:" + "[{x,y,width,height,area,center}]} (largest first). " + "'merge' unions nested glyph boxes; 'min_area'/'max_area'/" + "'max_aspect' filter. Crop these to feed OCR."), + input_schema=schema({ + "min_area": {"type": "integer"}, + "max_area": {"type": "integer"}, + "merge": {"type": "boolean"}, + "max_aspect": {"type": "number"}, + "region": {"type": "array", "items": {"type": "integer"}}}, + required=[]), + handler=h.find_text_regions, + annotations=READ_ONLY, + ), + MCPTool( + name="ac_find_text_lines", + description=("Locate horizontal lines of text on screen via MSER: one " + "box per line (glyphs within 'y_tolerance' px grouped). " + "Returns {count, lines}. No OCR needed."), + input_schema=schema({ + "y_tolerance": {"type": "integer"}, + "region": {"type": "array", "items": {"type": "integer"}}}, + required=[]), + handler=h.find_text_lines, + annotations=READ_ONLY, + ), + ] + + def ssim_tools() -> List[MCPTool]: return [ MCPTool( @@ -6440,7 +6474,7 @@ def media_assert_tools() -> List[MCPTool]: color_region_tools, ssim_tools, feature_match_tools, shape_locator_tools, window_layout_tools, window_arrange_tools, preprocess_tools, monitor_layout_tools, actionability_tools, element_parse_tools, - hsv_segment_tools, plugin_sdk_tools, governance_tools, + hsv_segment_tools, text_regions_tools, plugin_sdk_tools, governance_tools, credential_lease_tools, egress_tools, approval_testing_tools, trajectory_eval_tools, compliance_tools, agent_trace_tools, video_report_tools, fuzzy_tools, artifact_store_tools, image_dedup_tools, diff --git a/je_auto_control/utils/mcp_server/tools/_handlers.py b/je_auto_control/utils/mcp_server/tools/_handlers.py index cafe8b49..880ae7d0 100644 --- a/je_auto_control/utils/mcp_server/tools/_handlers.py +++ b/je_auto_control/utils/mcp_server/tools/_handlers.py @@ -2200,6 +2200,17 @@ def dominant_hue_regions(hue, hue_tol=10, sat_min=80, val_min=80, min_area=50, return _dominant_hue_regions(hue, hue_tol, sat_min, val_min, min_area, region) +def find_text_regions(min_area=60, max_area=None, merge=True, max_aspect=12.0, + region=None): + from je_auto_control.utils.executor.action_executor import _find_text_regions + return _find_text_regions(min_area, max_area, merge, max_aspect, region) + + +def find_text_lines(y_tolerance=8, region=None): + from je_auto_control.utils.executor.action_executor import _find_text_lines + return _find_text_lines(y_tolerance, region) + + def detect_drift(reference, current, threshold=0.25, bins=10): from je_auto_control.utils.executor.action_executor import _detect_drift return _detect_drift(reference, current, threshold, bins) diff --git a/je_auto_control/utils/text_regions/__init__.py b/je_auto_control/utils/text_regions/__init__.py new file mode 100644 index 00000000..a27f316c --- /dev/null +++ b/je_auto_control/utils/text_regions/__init__.py @@ -0,0 +1,6 @@ +"""Model-free on-screen text-region detection (MSER): regions and lines.""" +from je_auto_control.utils.text_regions.text_regions import ( + find_text_lines, find_text_regions, +) + +__all__ = ["find_text_lines", "find_text_regions"] diff --git a/je_auto_control/utils/text_regions/text_regions.py b/je_auto_control/utils/text_regions/text_regions.py new file mode 100644 index 00000000..fe78af2e --- /dev/null +++ b/je_auto_control/utils/text_regions/text_regions.py @@ -0,0 +1,132 @@ +"""Model-free on-screen text-region detection (MSER) — where is text, without OCR. + +``shape_locator`` finds rectangular contours (buttons / cards, not text) and +``locate_text`` needs a Tesseract / Paddle engine *and* the exact string to search +for. Neither answers "where is there *any* text on screen" without running OCR or +knowing the words. This uses MSER (Maximally Stable Extremal Regions) to find the +glyph/word blobs, so a script can crop candidate text boxes to feed OCR (much faster +and more accurate than full-frame OCR) or detect that a label appeared with no OCR +dependency installed. + +Runs on an injectable ``haystack`` (ndarray / path / PIL), so it is headless-testable +on synthetic arrays. OpenCV + NumPy come in via ``je_open_cv`` (``cv2.MSER_create`` is +base OpenCV, no contrib). Imports no ``PySide6``. +""" +from typing import Any, Dict, List, Optional, Sequence, Tuple + +from je_auto_control.utils.visual_match.visual_match import _haystack_gray + +ImageSource = Any +Rect = Tuple[int, int, int, int] + + +def _box_dict(rect: Rect) -> Dict[str, Any]: + x, y, width, height = rect + return {"x": x, "y": y, "width": width, "height": height, + "area": width * height, "center": [x + width // 2, y + height // 2]} + + +def _accept(rect: Rect, shape, min_area: int, max_area: Optional[int], + max_aspect: float) -> bool: + """Whether an MSER box looks like text (size / aspect, not the whole frame).""" + _x, _y, width, height = rect + frame_h, frame_w = shape[:2] + if width >= 0.95 * frame_w and height >= 0.95 * frame_h: + return False # whole-frame extremal region + area = width * height + if area < min_area or (max_area is not None and area > max_area): + return False + aspect = width / height if height else 0.0 + return aspect <= max_aspect + + +def _filtered_boxes(gray, min_area: int, max_area: Optional[int], + max_aspect: float) -> List[Rect]: + """Return de-duplicated MSER bounding boxes passing the size / aspect filters.""" + import cv2 + regions, _bboxes = cv2.MSER_create().detectRegions(gray) + out: List[Rect] = [] + seen = set() + for points in regions: + rect = cv2.boundingRect(points.reshape(-1, 1, 2)) + if rect not in seen and _accept(rect, gray.shape, min_area, max_area, + max_aspect): + out.append(rect) + seen.add(rect) + return out + + +def _overlaps(a: Rect, b: Rect) -> bool: + return (a[0] < b[0] + b[2] and b[0] < a[0] + a[2] + and a[1] < b[1] + b[3] and b[1] < a[1] + a[3]) + + +def _union(a: Rect, b: Rect) -> Rect: + x, y = min(a[0], b[0]), min(a[1], b[1]) + return (x, y, max(a[0] + a[2], b[0] + b[2]) - x, + max(a[1] + a[3], b[1] + b[3]) - y) + + +def _merge_pass(rects: Sequence[Rect]) -> Tuple[bool, List[Rect]]: + out: List[Rect] = [] + changed = False + for rect in rects: + index = next((i for i, kept in enumerate(out) if _overlaps(rect, kept)), + None) + if index is None: + out.append(rect) + else: + out[index] = _union(rect, out[index]) + changed = True + return changed, out + + +def _merge_overlapping(rects: Sequence[Rect]) -> List[Rect]: + """Union all overlapping rectangles (collapses MSER's nested glyph regions).""" + result = list(rects) + changed = True + while changed: + changed, result = _merge_pass(result) + return result + + +def find_text_regions(haystack: Optional[ImageSource] = None, *, + region: Optional[Sequence[int]] = None, min_area: int = 60, + max_area: Optional[int] = None, merge: bool = True, + max_aspect: float = 12.0) -> List[Dict[str, Any]]: + """Return boxes of text/glyph regions on screen, largest first (no OCR needed). + + ``merge`` unions overlapping detections (MSER reports nested regions per glyph). + ``min_area`` / ``max_area`` drop specks and page-sized blobs; ``max_aspect`` + rejects long thin lines that are rules rather than text. Each result is + ``{x, y, width, height, area, center}``. + """ + rects = _filtered_boxes(_haystack_gray(haystack, region), int(min_area), + int(max_area) if max_area is not None else None, + float(max_aspect)) + if merge: + rects = _merge_overlapping(rects) + rects.sort(key=lambda rect: rect[2] * rect[3], reverse=True) + return [_box_dict(rect) for rect in rects] + + +def find_text_lines(haystack: Optional[ImageSource] = None, *, + region: Optional[Sequence[int]] = None, + y_tolerance: int = 8) -> List[Dict[str, Any]]: + """Return one box per horizontal line of text, top-to-bottom. + + Glyph boxes whose vertical centres are within ``y_tolerance`` pixels are grouped + into a line spanning their combined extent — ready to crop and feed to OCR. + """ + boxes = _filtered_boxes(_haystack_gray(haystack, region), 40, None, 20.0) + rows: List[Dict[str, Any]] = [] + for rect in sorted(boxes, key=lambda item: item[1]): + center_y = rect[1] + rect[3] // 2 + row = next((candidate for candidate in rows + if abs(center_y - candidate["cy"]) <= int(y_tolerance)), None) + if row is None: + rows.append({"cy": center_y, "rect": rect}) + else: + row["rect"] = _union(row["rect"], rect) + return [_box_dict(row["rect"]) + for row in sorted(rows, key=lambda item: item["cy"])] diff --git a/test/unit_test/headless/test_text_regions_batch.py b/test/unit_test/headless/test_text_regions_batch.py new file mode 100644 index 00000000..df7abff9 --- /dev/null +++ b/test/unit_test/headless/test_text_regions_batch.py @@ -0,0 +1,70 @@ +"""Headless tests for MSER text-region detection. No Qt.""" +import pytest + +import je_auto_control as ac + +np = pytest.importorskip("numpy") +cv2 = pytest.importorskip("cv2") + +from je_auto_control.utils.text_regions import ( # noqa: E402 + find_text_lines, find_text_regions, +) + + +def _two_rows(): + img = np.full((120, 300), 235, dtype=np.uint8) + for i, ch in enumerate("ABCD"): + cv2.putText(img, ch, (20 + i * 30, 40), cv2.FONT_HERSHEY_SIMPLEX, 1.0, 0, 2) + for i, ch in enumerate("XY"): + cv2.putText(img, ch, (20 + i * 30, 90), cv2.FONT_HERSHEY_SIMPLEX, 1.0, 0, 2) + return img + + +def test_finds_glyph_regions(): + regions = find_text_regions(_two_rows(), min_area=60) + assert len(regions) >= 4 # at least the four top-row glyphs + assert all("center" in r and r["area"] >= 60 for r in regions) + + +def test_groups_into_two_lines(): + lines = find_text_lines(_two_rows(), y_tolerance=8) + assert len(lines) == 2 + top, bottom = sorted(lines, key=lambda line: line["y"]) + assert top["width"] > bottom["width"] # ABCD wider than XY + assert bottom["y"] > top["y"] + + +def test_blank_image_has_no_text(): + blank = np.full((100, 100), 235, dtype=np.uint8) + assert find_text_regions(blank) == [] + assert find_text_lines(blank) == [] + + +def test_min_area_filters_small(): + big = find_text_regions(_two_rows(), min_area=100000) + assert big == [] + + +def test_merge_reduces_region_count(): + scene = _two_rows() + unmerged = find_text_regions(scene, min_area=40, merge=False) + merged = find_text_regions(scene, min_area=40, merge=True) + assert len(merged) <= len(unmerged) + + +# --- wiring --------------------------------------------------------------- + +def test_wiring(): + known = set(ac.executor.known_commands()) + assert {"AC_find_text_regions", "AC_find_text_lines"} <= known + from je_auto_control.utils.mcp_server.tools import build_default_tool_registry + names = {t.name for t in build_default_tool_registry()} + assert {"ac_find_text_regions", "ac_find_text_lines"} <= names + from je_auto_control.gui.script_builder.command_schema import _build_specs + specs = {s.command for s in _build_specs()} + assert {"AC_find_text_regions", "AC_find_text_lines"} <= specs + + +def test_facade_exports(): + for attr in ("find_text_regions", "find_text_lines"): + assert hasattr(ac, attr) and attr in ac.__all__