diff --git a/README/WHATS_NEW_zh-CN.md b/README/WHATS_NEW_zh-CN.md index 1c049791..eef34573 100644 --- a/README/WHATS_NEW_zh-CN.md +++ b/README/WHATS_NEW_zh-CN.md @@ -1,5 +1,11 @@ # 本次更新 — AutoControl +## 本次更新 (2026-06-23) — 具信心分数的模板匹配 + +返回分数、搜索多尺度、找出所有出现处的模板匹配。完整参考:[`docs/source/Zh/doc/new_features/v127_features_doc.rst`](../docs/source/Zh/doc/new_features/v127_features_doc.rst)。 + +- **`match_template` / `match_template_all` / `best_matches` / `TemplateMatch`**(`AC_match_template`、`AC_match_template_all`):既有匹配器(`find_object`)为单一尺度且*丢弃分数*。本功能返回带 `score`/`scale`/`center` 的 `Match`、搜索 `scales` 容忍 DPI/缩放,并以非极大值抑制列举每个出现处。可注入 `haystack`(ndarray/路径/PIL)→ 无头可测;OpenCV/NumPy 透过 `je_open_cv` 依赖。 + ## 本次更新 (2026-06-23) — 等待窗口标题(正则) 阻塞直到窗口标题符合正则(或消失)。完整参考:[`docs/source/Zh/doc/new_features/v126_features_doc.rst`](../docs/source/Zh/doc/new_features/v126_features_doc.rst)。 diff --git a/README/WHATS_NEW_zh-TW.md b/README/WHATS_NEW_zh-TW.md index 4f13f6e8..d1397e79 100644 --- a/README/WHATS_NEW_zh-TW.md +++ b/README/WHATS_NEW_zh-TW.md @@ -1,5 +1,11 @@ # 本次更新 — AutoControl +## 本次更新 (2026-06-23) — 具信心分數的模板比對 + +回傳分數、搜尋多尺度、找出所有出現處的模板比對。完整參考:[`docs/source/Zh/doc/new_features/v127_features_doc.rst`](../docs/source/Zh/doc/new_features/v127_features_doc.rst)。 + +- **`match_template` / `match_template_all` / `best_matches` / `TemplateMatch`**(`AC_match_template`、`AC_match_template_all`):既有比對器(`find_object`)為單一尺度且*丟棄分數*。本功能回傳帶 `score`/`scale`/`center` 的 `Match`、搜尋 `scales` 容忍 DPI/縮放,並以非極大值抑制列舉每個出現處。可注入 `haystack`(ndarray/路徑/PIL)→ 無頭可測;OpenCV/NumPy 透過 `je_open_cv` 相依。 + ## 本次更新 (2026-06-23) — 等待視窗標題(正則) 阻塞直到視窗標題符合正則(或消失)。完整參考:[`docs/source/Zh/doc/new_features/v126_features_doc.rst`](../docs/source/Zh/doc/new_features/v126_features_doc.rst)。 diff --git a/WHATS_NEW.md b/WHATS_NEW.md index 9d63c13d..6ef10d69 100644 --- a/WHATS_NEW.md +++ b/WHATS_NEW.md @@ -1,5 +1,11 @@ # What's New — AutoControl +## What's new (2026-06-23) — Confidence-Returning Template Matching + +Template matching that returns the score, searches multiple scales, and finds all occurrences. Full reference: [`docs/source/Eng/doc/new_features/v127_features_doc.rst`](docs/source/Eng/doc/new_features/v127_features_doc.rst). + +- **`match_template` / `match_template_all` / `best_matches` / `TemplateMatch`** (`AC_match_template`, `AC_match_template_all`): the existing matcher (`find_object`) is single-scale and *discards the score*. This returns a `Match` with `score`/`scale`/`center`, searches `scales` for DPI/zoom tolerance, and enumerates every occurrence with non-maximum suppression. Injectable `haystack` (ndarray/path/PIL) → headless-testable on synthetic arrays; OpenCV/NumPy via the `je_open_cv` dependency. + ## What's new (2026-06-23) — Wait for Window Title (Regex) Block until a window title matches a regex (or vanishes). Full reference: [`docs/source/Eng/doc/new_features/v126_features_doc.rst`](docs/source/Eng/doc/new_features/v126_features_doc.rst). diff --git a/docs/source/Eng/doc/new_features/v127_features_doc.rst b/docs/source/Eng/doc/new_features/v127_features_doc.rst new file mode 100644 index 00000000..ff6718a6 --- /dev/null +++ b/docs/source/Eng/doc/new_features/v127_features_doc.rst @@ -0,0 +1,43 @@ +Confidence-Returning Template Matching +====================================== + +The project's template matcher (``je_open_cv.find_object`` via ``cv2_utils``) is +single-scale and returns only bounding boxes — the correlation *score* it +computes internally is discarded. So there was no way to rank candidates, set a +confidence threshold and read back *how well* it matched, find a control when the +UI is DPI / zoom-scaled, or enumerate *every* occurrence. This adds those, like +PyAutoGUI ``confidence`` / ``locateAll`` and SikuliX ``similarity`` / ``findAll``. + +The matching takes an injectable ``haystack`` image (ndarray / path / PIL), so it +is unit-testable on synthetic arrays without a real screen — only the default +(grab the screen) is device-bound. OpenCV + NumPy come in via the project's +``je_open_cv`` dependency. Imports no ``PySide6``. + +Headless API +------------ + +.. code-block:: python + + from je_auto_control import match_template, match_template_all, best_matches + + m = match_template("button.png", min_score=0.85, scales=(0.9, 1.0, 1.1)) + if m: + print(m.score, m.scale, m.center) # confidence + DPI scale + click point + + for hit in match_template_all("row_handle.png", min_score=0.8): + click(*hit.center) # every occurrence, overlaps removed + +``match_template`` returns the single best :class:`Match` (``x`` / ``y`` / +``width`` / ``height`` / ``score`` / ``scale`` / ``center``) at or above +``min_score``, searching each entry in ``scales`` for DPI / zoom tolerance. +``match_template_all`` returns every hit, merging overlapping detections by +non-maximum suppression (``nms_iou``) and capping at ``max_results``. +``best_matches`` returns the top-N by score regardless of threshold (for tuning). + +Executor commands +----------------- + +``AC_match_template`` returns ``{found, match}`` (the match dict carries the +score); ``AC_match_template_all`` returns ``{count, matches}``. Both are exposed +as MCP tools (``ac_match_template`` / ``ac_match_template_all``) and as Script +Builder commands under **Image**. diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst index b0ebbbc7..97c2fe05 100644 --- a/docs/source/Eng/eng_index.rst +++ b/docs/source/Eng/eng_index.rst @@ -149,6 +149,7 @@ Comprehensive guides for all AutoControl features. doc/new_features/v124_features_doc doc/new_features/v125_features_doc doc/new_features/v126_features_doc + doc/new_features/v127_features_doc doc/ocr_backends/ocr_backends_doc doc/observability/observability_doc doc/operations_layer/operations_layer_doc diff --git a/docs/source/Zh/doc/new_features/v127_features_doc.rst b/docs/source/Zh/doc/new_features/v127_features_doc.rst new file mode 100644 index 00000000..95fbc9a2 --- /dev/null +++ b/docs/source/Zh/doc/new_features/v127_features_doc.rst @@ -0,0 +1,36 @@ +具信心分數的模板比對 +==================== + +本專案的模板比對器(``cv2_utils`` 經由 ``je_open_cv.find_object``)為單一尺度且僅回傳邊界框——其內部計算的 +相關性*分數*被丟棄。因此先前無法為候選排名、設定信心門檻並讀回*比對程度*、在 UI 經 DPI / 縮放時找到控制項, +或列舉*每一個*出現處。本功能加入這些,類似 PyAutoGUI 的 ``confidence`` / ``locateAll`` 與 SikuliX 的 +``similarity`` / ``findAll``。 + +比對接受可注入的 ``haystack`` 影像(ndarray / 路徑 / PIL),因此可在無真實螢幕下對合成陣列做單元測試——僅預設 +(擷取螢幕)為裝置相依。OpenCV + NumPy 透過專案的 ``je_open_cv`` 相依引入。不匯入 ``PySide6``。 + +無頭 API +-------- + +.. code-block:: python + + from je_auto_control import match_template, match_template_all, best_matches + + m = match_template("button.png", min_score=0.85, scales=(0.9, 1.0, 1.1)) + if m: + print(m.score, m.scale, m.center) # 信心 + DPI 尺度 + 點擊點 + + for hit in match_template_all("row_handle.png", min_score=0.8): + click(*hit.center) # 每個出現處,重疊已移除 + +``match_template`` 回傳達到 ``min_score`` 的單一最佳 :class:`Match`(``x`` / ``y`` / ``width`` / ``height`` / +``score`` / ``scale`` / ``center``),並搜尋 ``scales`` 中每個尺度以容忍 DPI / 縮放。``match_template_all`` +回傳所有命中,以非極大值抑制(``nms_iou``)合併重疊偵測並以 ``max_results`` 設上限。``best_matches`` 回傳依 +分數排序的前 N 個(不論門檻,供調校)。 + +執行器命令 +---------- + +``AC_match_template`` 回傳 ``{found, match}``(match dict 帶有分數);``AC_match_template_all`` 回傳 +``{count, matches}``。兩者皆以 MCP 工具(``ac_match_template`` / ``ac_match_template_all``)以及 Script Builder +中 **Image** 分類下的命令提供。 diff --git a/docs/source/Zh/zh_index.rst b/docs/source/Zh/zh_index.rst index cc6842f4..0fc69a0e 100644 --- a/docs/source/Zh/zh_index.rst +++ b/docs/source/Zh/zh_index.rst @@ -149,6 +149,7 @@ AutoControl 所有功能的完整使用指南。 doc/new_features/v124_features_doc doc/new_features/v125_features_doc doc/new_features/v126_features_doc + doc/new_features/v127_features_doc doc/ocr_backends/ocr_backends_doc doc/observability/observability_doc doc/operations_layer/operations_layer_doc diff --git a/je_auto_control/__init__.py b/je_auto_control/__init__.py index 9959357b..83441a97 100644 --- a/je_auto_control/__init__.py +++ b/je_auto_control/__init__.py @@ -273,6 +273,11 @@ ) # Address a table / grid cell by (row, column) from bounding boxes from je_auto_control.utils.grid_locator import cluster_grid, locate_cell +# Confidence-returning template matching (score / multi-scale / find-all + NMS) +from je_auto_control.utils.visual_match import ( + best_matches, match_template, match_template_all, +) +from je_auto_control.utils.visual_match import Match as TemplateMatch # CI workflow annotations (GitHub Actions) from je_auto_control.utils.ci_annotations import ( emit_annotations, format_annotation, @@ -1062,6 +1067,10 @@ def start_autocontrol_gui(*args, **kwargs): "plan_with_modifiers", "cluster_grid", "locate_cell", + "TemplateMatch", + "match_template", + "match_template_all", + "best_matches", "emit_annotations", "format_annotation", "ClipboardHistory", "default_clipboard_history", "analyze_heal_log", "heal_stats", "scan_secrets", diff --git a/je_auto_control/gui/script_builder/command_schema.py b/je_auto_control/gui/script_builder/command_schema.py index 28443251..2536dd85 100644 --- a/je_auto_control/gui/script_builder/command_schema.py +++ b/je_auto_control/gui/script_builder/command_schema.py @@ -253,6 +253,31 @@ def _add_image_specs(specs: List[CommandSpec]) -> None: default=0.8, min_value=0.0, max_value=1.0), ), )) + specs.append(CommandSpec( + "AC_match_template", "Image", "Match Template (scored)", + fields=( + FieldSpec("template", FieldType.FILE_PATH), + FieldSpec("min_score", FieldType.FLOAT, optional=True, default=0.8, + min_value=0.0, max_value=1.0), + FieldSpec("scales", FieldType.STRING, optional=True, + placeholder="[0.9, 1.0, 1.1]"), + FieldSpec("region", FieldType.STRING, optional=True, + placeholder="[left, top, right, bottom]"), + ), + description="Locate a template and return its confidence score + scale.", + )) + specs.append(CommandSpec( + "AC_match_template_all", "Image", "Match Template All (scored)", + fields=( + FieldSpec("template", FieldType.FILE_PATH), + FieldSpec("min_score", FieldType.FLOAT, optional=True, default=0.8, + min_value=0.0, max_value=1.0), + FieldSpec("max_results", FieldType.INT, optional=True, default=20), + FieldSpec("nms_iou", FieldType.FLOAT, optional=True, default=0.3, + min_value=0.0, max_value=1.0), + ), + description="Find every occurrence of a template (scored, NMS-deduped).", + )) def _add_ocr_specs(specs: List[CommandSpec]) -> None: diff --git a/je_auto_control/utils/executor/action_executor.py b/je_auto_control/utils/executor/action_executor.py index 9d5e800e..d4575367 100644 --- a/je_auto_control/utils/executor/action_executor.py +++ b/je_auto_control/utils/executor/action_executor.py @@ -3197,6 +3197,38 @@ def _grid_cell(boxes: Any, row: Any, col: Any, row_tolerance=int(row_tolerance)) +def _match_template(template: str, min_score: Any = 0.8, scales: Any = None, + region: Any = None, + method: str = "ccoeff_normed") -> Dict[str, Any]: + """Adapter: best confidence-scored template match on the screen.""" + import json + from je_auto_control.utils.visual_match import match_template + if isinstance(scales, str): + scales = json.loads(scales) if scales.strip() else None + if isinstance(region, str): + region = json.loads(region) if region.strip() else None + match = match_template(template, region=region, + scales=tuple(scales) if scales else (1.0,), + min_score=float(min_score), method=method) + return {"found": match is not None, + "match": match.to_dict() if match else None} + + +def _match_template_all(template: str, min_score: Any = 0.8, + max_results: Any = 20, nms_iou: Any = 0.3, + region: Any = None) -> Dict[str, Any]: + """Adapter: every confidence-scored template match on the screen (NMS).""" + import json + from je_auto_control.utils.visual_match import match_template_all + if isinstance(region, str): + region = json.loads(region) if region.strip() else None + matches = match_template_all(template, region=region, + min_score=float(min_score), + max_results=int(max_results), + nms_iou=float(nms_iou)) + return {"count": len(matches), "matches": [m.to_dict() for m in matches]} + + def _with_modifiers(modifiers: Any, actions: Any) -> Dict[str, Any]: """Adapter: run nested actions while modifier keys are held down.""" import json @@ -4914,6 +4946,8 @@ def __init__(self): "AC_type_unicode": _type_unicode, "AC_with_modifiers": _with_modifiers, "AC_grid_cell": _grid_cell, + "AC_match_template": _match_template, + "AC_match_template_all": _match_template_all, "AC_detect_drift": _detect_drift, "AC_categorical_drift": _categorical_drift, "AC_diff_rows": _diff_rows, diff --git a/je_auto_control/utils/mcp_server/tools/_factories.py b/je_auto_control/utils/mcp_server/tools/_factories.py index eefd80c2..de40c5be 100644 --- a/je_auto_control/utils/mcp_server/tools/_factories.py +++ b/je_auto_control/utils/mcp_server/tools/_factories.py @@ -2628,6 +2628,42 @@ def tween_drag_tools() -> List[MCPTool]: ] +def visual_match_tools() -> List[MCPTool]: + return [ + MCPTool( + name="ac_match_template", + description=("Find 'template' (image path) on screen and return the " + "match WITH its score: {found, match:{x,y,width,height," + "score,scale,center}}. 'scales' [..] for DPI/zoom, " + "'min_score', 'region', 'method'."), + input_schema=schema({ + "template": {"type": "string"}, + "min_score": {"type": "number"}, + "scales": {"type": "array", "items": {"type": "number"}}, + "region": {"type": "array", "items": {"type": "integer"}}, + "method": {"type": "string"}}, + required=["template"]), + handler=h.match_template, + annotations=READ_ONLY, + ), + MCPTool( + name="ac_match_template_all", + description=("Find EVERY occurrence of 'template' on screen " + ">= 'min_score', overlaps removed by NMS. " + "Returns {count, matches}."), + input_schema=schema({ + "template": {"type": "string"}, + "min_score": {"type": "number"}, + "max_results": {"type": "integer"}, + "nms_iou": {"type": "number"}, + "region": {"type": "array", "items": {"type": "integer"}}}, + required=["template"]), + handler=h.match_template_all, + annotations=READ_ONLY, + ), + ] + + def grid_locator_tools() -> List[MCPTool]: return [ MCPTool( @@ -6018,8 +6054,8 @@ def media_assert_tools() -> List[MCPTool]: ci_annotation_tools, clipboard_history_tools, audit_analysis_tools, process_doc_tools, tween_drag_tools, mouse_path_tools, field_entry_tools, key_hold_tools, mouse_relative_tools, text_unicode_tools, - modifier_state_tools, grid_locator_tools, plugin_sdk_tools, - governance_tools, + modifier_state_tools, grid_locator_tools, visual_match_tools, + plugin_sdk_tools, governance_tools, credential_lease_tools, egress_tools, approval_testing_tools, trajectory_eval_tools, compliance_tools, agent_trace_tools, video_report_tools, fuzzy_tools, artifact_store_tools, image_dedup_tools, diff --git a/je_auto_control/utils/mcp_server/tools/_handlers.py b/je_auto_control/utils/mcp_server/tools/_handlers.py index a46b281c..5014fd8c 100644 --- a/je_auto_control/utils/mcp_server/tools/_handlers.py +++ b/je_auto_control/utils/mcp_server/tools/_handlers.py @@ -2068,6 +2068,19 @@ def grid_cell(boxes, row, col, row_tolerance=10): return _grid_cell(boxes, row, col, row_tolerance) +def match_template(template, min_score=0.8, scales=None, region=None, + method="ccoeff_normed"): + from je_auto_control.utils.executor.action_executor import _match_template + return _match_template(template, min_score, scales, region, method) + + +def match_template_all(template, min_score=0.8, max_results=20, nms_iou=0.3, + region=None): + from je_auto_control.utils.executor.action_executor import ( + _match_template_all) + return _match_template_all(template, min_score, max_results, nms_iou, region) + + def detect_drift(reference, current, threshold=0.25, bins=10): from je_auto_control.utils.executor.action_executor import _detect_drift return _detect_drift(reference, current, threshold, bins) diff --git a/je_auto_control/utils/visual_match/__init__.py b/je_auto_control/utils/visual_match/__init__.py new file mode 100644 index 00000000..8bfc9746 --- /dev/null +++ b/je_auto_control/utils/visual_match/__init__.py @@ -0,0 +1,6 @@ +"""Confidence-returning template matching (score, multi-scale, find-all + NMS).""" +from je_auto_control.utils.visual_match.visual_match import ( + Match, best_matches, match_template, match_template_all, +) + +__all__ = ["Match", "best_matches", "match_template", "match_template_all"] diff --git a/je_auto_control/utils/visual_match/visual_match.py b/je_auto_control/utils/visual_match/visual_match.py new file mode 100644 index 00000000..1b7e81d6 --- /dev/null +++ b/je_auto_control/utils/visual_match/visual_match.py @@ -0,0 +1,172 @@ +"""Confidence-returning template matching: score, multi-scale, find-all + NMS. + +The project's template matcher (``je_open_cv.find_object`` via ``cv2_utils``) is +single-scale and returns only bounding boxes — the correlation *score* it computes +internally is discarded, so there is no way to rank candidates, set a confidence +threshold and read back how well it matched, find a button when the UI is +DPI/zoom-scaled, or enumerate *every* occurrence. This adds those, in the style of +PyAutoGUI ``confidence`` / ``locateAll`` and SikuliX ``similarity`` / ``findAll``. + +The matching takes an injectable ``haystack`` image (ndarray / path / PIL), so it +is unit-testable on synthetic arrays without a real screen; only the default +(grab the screen) is device-bound. OpenCV + NumPy come in via the project's +``je_open_cv`` dependency and are imported lazily. Imports no ``PySide6``. +""" +from dataclasses import asdict, dataclass +from typing import Any, Dict, List, Optional, Sequence + +# cv2 method name -> the OpenCV constant is resolved lazily in _method(). +_METHOD_NAMES = ("ccoeff_normed", "ccorr_normed", "sqdiff_normed") +ImageSource = Any + + +@dataclass(frozen=True) +class Match: + """One template match: top-left (x, y), size, correlation score, scale.""" + + x: int + y: int + width: int + height: int + score: float + scale: float + + @property + def center(self) -> List[int]: + """The match's centre point ``[x, y]`` (ready to click).""" + return [self.x + self.width // 2, self.y + self.height // 2] + + def to_dict(self) -> Dict[str, Any]: + """Return the match as a plain dict including the centre point.""" + data = asdict(self) + data["center"] = self.center + return data + + +def _method(name: str) -> int: + import cv2 + table = {"ccoeff_normed": cv2.TM_CCOEFF_NORMED, + "ccorr_normed": cv2.TM_CCORR_NORMED, + "sqdiff_normed": cv2.TM_SQDIFF_NORMED} + if name not in table: + raise ValueError(f"unknown method: {name!r}") + return table[name] + + +def _to_gray(source: ImageSource): + """Load a path / ndarray / PIL image as a 2-D grayscale ndarray.""" + import cv2 + import numpy as np + if hasattr(source, "shape"): + array = np.asarray(source) + elif isinstance(source, (str, bytes)) or hasattr(source, "__fspath__"): + array = cv2.imread(str(source), cv2.IMREAD_COLOR) + if array is None: + raise ValueError(f"could not read image: {source!r}") + else: + array = np.asarray(source) + if array.ndim == 2: + return array + channels = array.shape[2] + code = cv2.COLOR_BGRA2GRAY if channels == 4 else cv2.COLOR_BGR2GRAY + return cv2.cvtColor(array, code) + + +def _grab_gray(region: Optional[Sequence[int]]): + from je_auto_control.utils.cv2_utils.screenshot import pil_screenshot + image = pil_screenshot(screen_region=list(region) if region else None) + return _to_gray(image) + + +def _haystack_gray(haystack: Optional[ImageSource], + region: Optional[Sequence[int]]): + return _to_gray(haystack) if haystack is not None else _grab_gray(region) + + +def _resize(template, scale: float): + import cv2 + if abs(scale - 1.0) < 1e-9: + return template + height, width = template.shape[:2] + new_size = (max(1, round(width * scale)), max(1, round(height * scale))) + return cv2.resize(template, new_size) + + +def match_template(template: ImageSource, *, haystack: Optional[ImageSource] = None, + region: Optional[Sequence[int]] = None, + scales: Sequence[float] = (1.0,), min_score: float = 0.8, + method: str = "ccoeff_normed") -> Optional[Match]: + """Return the single best match of ``template`` at or above ``min_score``. + + Searches each scale in ``scales`` (e.g. ``(0.9, 1.0, 1.1)`` for DPI / zoom + tolerance) and keeps the highest-scoring hit, or ``None`` if none clear the + threshold. + """ + import cv2 + tmpl = _to_gray(template) + hay = _haystack_gray(haystack, region) + metric = _method(method) + best: Optional[Match] = None + for scale in scales: + scaled = _resize(tmpl, float(scale)) + if scaled.shape[0] > hay.shape[0] or scaled.shape[1] > hay.shape[1]: + continue + _, max_val, _, max_loc = cv2.minMaxLoc(cv2.matchTemplate(hay, scaled, + metric)) + if max_val >= min_score and (best is None or max_val > best.score): + best = Match(int(max_loc[0]), int(max_loc[1]), scaled.shape[1], + scaled.shape[0], round(float(max_val), 4), float(scale)) + return best + + +def _iou(a: Match, b: Match) -> float: + left = max(a.x, b.x) + top = max(a.y, b.y) + right = min(a.x + a.width, b.x + b.width) + bottom = min(a.y + a.height, b.y + b.height) + inter = max(0, right - left) * max(0, bottom - top) + if inter == 0: + return 0.0 + union = a.width * a.height + b.width * b.height - inter + return inter / union + + +def _nms(matches: List[Match], iou_threshold: float) -> List[Match]: + kept: List[Match] = [] + for candidate in sorted(matches, key=lambda m: m.score, reverse=True): + if all(_iou(candidate, k) <= iou_threshold for k in kept): + kept.append(candidate) + return kept + + +def match_template_all(template: ImageSource, *, + haystack: Optional[ImageSource] = None, + region: Optional[Sequence[int]] = None, + min_score: float = 0.8, max_results: int = 20, + nms_iou: float = 0.3) -> List[Match]: + """Return every match of ``template`` >= ``min_score``, overlaps removed. + + Overlapping detections (the matcher fires on neighbouring pixels) are merged + by non-maximum suppression on the intersection-over-union, highest score + kept. Results are ordered by score, capped at ``max_results``. + """ + import cv2 + import numpy as np + tmpl = _to_gray(template) + hay = _haystack_gray(haystack, region) + height, width = tmpl.shape[:2] + result = cv2.matchTemplate(hay, tmpl, cv2.TM_CCOEFF_NORMED) + ys, xs = np.nonzero(result >= float(min_score)) + candidates = [Match(int(x), int(y), width, height, + round(float(result[y, x]), 4), 1.0) + for y, x in zip(ys, xs)] + return _nms(candidates, float(nms_iou))[:int(max_results)] + + +def best_matches(template: ImageSource, *, + haystack: Optional[ImageSource] = None, + region: Optional[Sequence[int]] = None, + top_n: int = 5) -> List[Match]: + """Return the top ``top_n`` matches by score (any score), nearest-best first.""" + return match_template_all(template, haystack=haystack, region=region, + min_score=-1.0, max_results=int(top_n)) diff --git a/test/unit_test/headless/test_visual_match_batch.py b/test/unit_test/headless/test_visual_match_batch.py new file mode 100644 index 00000000..d47988ac --- /dev/null +++ b/test/unit_test/headless/test_visual_match_batch.py @@ -0,0 +1,87 @@ +"""Headless tests for confidence-returning template matching. No Qt.""" +import pytest + +import je_auto_control as ac + +np = pytest.importorskip("numpy") +pytest.importorskip("cv2") + +from je_auto_control.utils.visual_match import ( # noqa: E402 + Match, best_matches, match_template, match_template_all, +) + + +def _patch(): + """A 20x20 textured patch (non-constant, so CCOEFF_NORMED is well-defined).""" + return np.tile(np.arange(0, 200, 10, dtype=np.uint8), (20, 1)) + + +def _haystack(): + hay = np.zeros((100, 200), dtype=np.uint8) + patch = _patch() + hay[30:50, 50:70] = patch + hay[30:50, 120:140] = patch + return hay + + +def test_match_returns_score_and_box(): + match = match_template(_patch(), haystack=_haystack(), min_score=0.9) + assert isinstance(match, Match) + assert (match.x, match.y, match.width, match.height) == (50, 30, 20, 20) + assert match.score == pytest.approx(1.0) + assert match.center == [60, 40] + + +def test_no_match_below_threshold(): + other = np.tile(np.arange(200, 0, -10, dtype=np.uint8), (20, 1))[:8, :8] + assert match_template(other, haystack=_haystack(), min_score=0.99) is None + + +def test_match_all_with_nms(): + matches = match_template_all(_patch(), haystack=_haystack(), min_score=0.9) + assert len(matches) == 2 # two patches, neighbours merged + xs = sorted(m.x for m in matches) + assert xs == [50, 120] + assert all(m.score == pytest.approx(1.0) for m in matches) + + +def test_multiscale_finds_scaled_template(): + import cv2 + yy, xx = np.mgrid[0:20, 0:20] # non-self-similar texture + tmpl = ((yy * 53 + xx * 97 + yy * xx * 17) % 256).astype(np.uint8) + big = cv2.resize(tmpl, (40, 40)) # == _resize(tmpl, 2.0) + hay = np.zeros((120, 200), dtype=np.uint8) + hay[10:50, 60:100] = big # embed only the 2x version + match = match_template(tmpl, haystack=hay, scales=(1.0, 2.0), min_score=0.9) + assert match is not None and match.scale == pytest.approx(2.0) + + +def test_unknown_method_raises(): + with pytest.raises(ValueError): + match_template(_patch(), haystack=_haystack(), method="bogus") + + +def test_to_dict_has_center(): + match = match_template(_patch(), haystack=_haystack(), min_score=0.9) + data = match.to_dict() + assert data["center"] == [60, 40] and data["score"] == pytest.approx(1.0) + + +# --- wiring --------------------------------------------------------------- + +def test_wiring(): + known = ac.executor.known_commands() + assert {"AC_match_template", "AC_match_template_all"} <= set(known) + from je_auto_control.utils.mcp_server.tools import build_default_tool_registry + names = {t.name for t in build_default_tool_registry()} + assert {"ac_match_template", "ac_match_template_all"} <= names + from je_auto_control.gui.script_builder.command_schema import _build_specs + specs = {s.command for s in _build_specs()} + assert {"AC_match_template", "AC_match_template_all"} <= specs + + +def test_facade_exports(): + for attr in ("TemplateMatch", "match_template", "match_template_all", + "best_matches"): + assert hasattr(ac, attr) and attr in ac.__all__ + assert callable(best_matches)