Add confidence-returning template matching (score / multi-scale / find-all)

README/WHATS_NEW_zh-CN.md

@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-23) — 具信心分数的模板匹配
+
+返回分数、搜索多尺度、找出所有出现处的模板匹配。完整参考:[`docs/source/Zh/doc/new_features/v127_features_doc.rst`](../docs/source/Zh/doc/new_features/v127_features_doc.rst)。
+
+- **`match_template` / `match_template_all` / `best_matches` / `TemplateMatch`**(`AC_match_template`、`AC_match_template_all`):既有匹配器(`find_object`)为单一尺度且*丢弃分数*。本功能返回带 `score`/`scale`/`center` 的 `Match`、搜索 `scales` 容忍 DPI/缩放,并以非极大值抑制列举每个出现处。可注入 `haystack`(ndarray/路径/PIL)→ 无头可测;OpenCV/NumPy 透过 `je_open_cv` 依赖。
+
 ## 本次更新 (2026-06-23) — 等待窗口标题(正则)
 
 阻塞直到窗口标题符合正则(或消失)。完整参考:[`docs/source/Zh/doc/new_features/v126_features_doc.rst`](../docs/source/Zh/doc/new_features/v126_features_doc.rst)。

README/WHATS_NEW_zh-TW.md

@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-23) — 具信心分數的模板比對
+
+回傳分數、搜尋多尺度、找出所有出現處的模板比對。完整參考:[`docs/source/Zh/doc/new_features/v127_features_doc.rst`](../docs/source/Zh/doc/new_features/v127_features_doc.rst)。
+
+- **`match_template` / `match_template_all` / `best_matches` / `TemplateMatch`**(`AC_match_template`、`AC_match_template_all`):既有比對器(`find_object`)為單一尺度且*丟棄分數*。本功能回傳帶 `score`/`scale`/`center` 的 `Match`、搜尋 `scales` 容忍 DPI/縮放,並以非極大值抑制列舉每個出現處。可注入 `haystack`(ndarray/路徑/PIL)→ 無頭可測;OpenCV/NumPy 透過 `je_open_cv` 相依。
+
 ## 本次更新 (2026-06-23) — 等待視窗標題(正則)
 
 阻塞直到視窗標題符合正則(或消失)。完整參考:[`docs/source/Zh/doc/new_features/v126_features_doc.rst`](../docs/source/Zh/doc/new_features/v126_features_doc.rst)。

WHATS_NEW.md

@@ -1,5 +1,11 @@
 # What's New — AutoControl
 
+## What's new (2026-06-23) — Confidence-Returning Template Matching
+
+Template matching that returns the score, searches multiple scales, and finds all occurrences. Full reference: [`docs/source/Eng/doc/new_features/v127_features_doc.rst`](docs/source/Eng/doc/new_features/v127_features_doc.rst).
+
+- **`match_template` / `match_template_all` / `best_matches` / `TemplateMatch`** (`AC_match_template`, `AC_match_template_all`): the existing matcher (`find_object`) is single-scale and *discards the score*. This returns a `Match` with `score`/`scale`/`center`, searches `scales` for DPI/zoom tolerance, and enumerates every occurrence with non-maximum suppression. Injectable `haystack` (ndarray/path/PIL) → headless-testable on synthetic arrays; OpenCV/NumPy via the `je_open_cv` dependency.
+
 ## What's new (2026-06-23) — Wait for Window Title (Regex)
 
 Block until a window title matches a regex (or vanishes). Full reference: [`docs/source/Eng/doc/new_features/v126_features_doc.rst`](docs/source/Eng/doc/new_features/v126_features_doc.rst).

docs/source/Eng/doc/new_features/v127_features_doc.rst

@@ -0,0 +1,43 @@
+Confidence-Returning Template Matching
+======================================
+
+The project's template matcher (``je_open_cv.find_object`` via ``cv2_utils``) is
+single-scale and returns only bounding boxes — the correlation *score* it
+computes internally is discarded. So there was no way to rank candidates, set a
+confidence threshold and read back *how well* it matched, find a control when the
+UI is DPI / zoom-scaled, or enumerate *every* occurrence. This adds those, like
+PyAutoGUI ``confidence`` / ``locateAll`` and SikuliX ``similarity`` / ``findAll``.
+
+The matching takes an injectable ``haystack`` image (ndarray / path / PIL), so it
+is unit-testable on synthetic arrays without a real screen — only the default
+(grab the screen) is device-bound. OpenCV + NumPy come in via the project's
+``je_open_cv`` dependency. Imports no ``PySide6``.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import match_template, match_template_all, best_matches
+
+    m = match_template("button.png", min_score=0.85, scales=(0.9, 1.0, 1.1))
+    if m:
+        print(m.score, m.scale, m.center)     # confidence + DPI scale + click point
+
+    for hit in match_template_all("row_handle.png", min_score=0.8):
+        click(*hit.center)                     # every occurrence, overlaps removed
+
+``match_template`` returns the single best :class:`Match` (``x`` / ``y`` /
+``width`` / ``height`` / ``score`` / ``scale`` / ``center``) at or above
+``min_score``, searching each entry in ``scales`` for DPI / zoom tolerance.
+``match_template_all`` returns every hit, merging overlapping detections by
+non-maximum suppression (``nms_iou``) and capping at ``max_results``.
+``best_matches`` returns the top-N by score regardless of threshold (for tuning).
+
+Executor commands
+-----------------
+
+``AC_match_template`` returns ``{found, match}`` (the match dict carries the
+score); ``AC_match_template_all`` returns ``{count, matches}``. Both are exposed
+as MCP tools (``ac_match_template`` / ``ac_match_template_all``) and as Script
+Builder commands under **Image**.

docs/source/Eng/eng_index.rst

@@ -149,6 +149,7 @@ Comprehensive guides for all AutoControl features.
    doc/new_features/v124_features_doc
    doc/new_features/v125_features_doc
    doc/new_features/v126_features_doc
+   doc/new_features/v127_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

docs/source/Zh/doc/new_features/v127_features_doc.rst

@@ -0,0 +1,36 @@
+具信心分數的模板比對
+====================
+
+本專案的模板比對器(``cv2_utils`` 經由 ``je_open_cv.find_object``)為單一尺度且僅回傳邊界框——其內部計算的
+相關性*分數*被丟棄。因此先前無法為候選排名、設定信心門檻並讀回*比對程度*、在 UI 經 DPI / 縮放時找到控制項,
+或列舉*每一個*出現處。本功能加入這些,類似 PyAutoGUI 的 ``confidence`` / ``locateAll`` 與 SikuliX 的
+``similarity`` / ``findAll``。
+
+比對接受可注入的 ``haystack`` 影像(ndarray / 路徑 / PIL),因此可在無真實螢幕下對合成陣列做單元測試——僅預設
+(擷取螢幕)為裝置相依。OpenCV + NumPy 透過專案的 ``je_open_cv`` 相依引入。不匯入 ``PySide6``。
+
+無頭 API
+--------
+
+.. code-block:: python
+
+    from je_auto_control import match_template, match_template_all, best_matches
+
+    m = match_template("button.png", min_score=0.85, scales=(0.9, 1.0, 1.1))
+    if m:
+        print(m.score, m.scale, m.center)     # 信心 + DPI 尺度 + 點擊點
+
+    for hit in match_template_all("row_handle.png", min_score=0.8):
+        click(*hit.center)                     # 每個出現處,重疊已移除
+
+``match_template`` 回傳達到 ``min_score`` 的單一最佳 :class:`Match`(``x`` / ``y`` / ``width`` / ``height`` /
+``score`` / ``scale`` / ``center``),並搜尋 ``scales`` 中每個尺度以容忍 DPI / 縮放。``match_template_all``
+回傳所有命中,以非極大值抑制(``nms_iou``)合併重疊偵測並以 ``max_results`` 設上限。``best_matches`` 回傳依
+分數排序的前 N 個(不論門檻,供調校)。
+
+執行器命令
+----------
+
+``AC_match_template`` 回傳 ``{found, match}``(match dict 帶有分數);``AC_match_template_all`` 回傳
+``{count, matches}``。兩者皆以 MCP 工具(``ac_match_template`` / ``ac_match_template_all``)以及 Script Builder
+中 **Image** 分類下的命令提供。

docs/source/Zh/zh_index.rst

@@ -149,6 +149,7 @@ AutoControl 所有功能的完整使用指南。
    doc/new_features/v124_features_doc
    doc/new_features/v125_features_doc
    doc/new_features/v126_features_doc
+   doc/new_features/v127_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

je_auto_control/__init__.py

@@ -273,6 +273,11 @@
 )
 # Address a table / grid cell by (row, column) from bounding boxes
 from je_auto_control.utils.grid_locator import cluster_grid, locate_cell
+# Confidence-returning template matching (score / multi-scale / find-all + NMS)
+from je_auto_control.utils.visual_match import (
+    best_matches, match_template, match_template_all,
+)
+from je_auto_control.utils.visual_match import Match as TemplateMatch
 # CI workflow annotations (GitHub Actions)
 from je_auto_control.utils.ci_annotations import (
     emit_annotations, format_annotation,
@@ -1062,6 +1067,10 @@ def start_autocontrol_gui(*args, **kwargs):
     "plan_with_modifiers",
     "cluster_grid",
     "locate_cell",
+    "TemplateMatch",
+    "match_template",
+    "match_template_all",
+    "best_matches",
     "emit_annotations", "format_annotation",
     "ClipboardHistory", "default_clipboard_history",
     "analyze_heal_log", "heal_stats", "scan_secrets",

je_auto_control/gui/script_builder/command_schema.py

@@ -253,6 +253,31 @@ def _add_image_specs(specs: List[CommandSpec]) -> None:
                       default=0.8, min_value=0.0, max_value=1.0),
         ),
     ))
+    specs.append(CommandSpec(
+        "AC_match_template", "Image", "Match Template (scored)",
+        fields=(
+            FieldSpec("template", FieldType.FILE_PATH),
+            FieldSpec("min_score", FieldType.FLOAT, optional=True, default=0.8,
+                      min_value=0.0, max_value=1.0),
+            FieldSpec("scales", FieldType.STRING, optional=True,
+                      placeholder="[0.9, 1.0, 1.1]"),
+            FieldSpec("region", FieldType.STRING, optional=True,
+                      placeholder="[left, top, right, bottom]"),
+        ),
+        description="Locate a template and return its confidence score + scale.",
+    ))
+    specs.append(CommandSpec(
+        "AC_match_template_all", "Image", "Match Template All (scored)",
+        fields=(
+            FieldSpec("template", FieldType.FILE_PATH),
+            FieldSpec("min_score", FieldType.FLOAT, optional=True, default=0.8,
+                      min_value=0.0, max_value=1.0),
+            FieldSpec("max_results", FieldType.INT, optional=True, default=20),
+            FieldSpec("nms_iou", FieldType.FLOAT, optional=True, default=0.3,
+                      min_value=0.0, max_value=1.0),
+        ),
+        description="Find every occurrence of a template (scored, NMS-deduped).",
+    ))
 
 
 def _add_ocr_specs(specs: List[CommandSpec]) -> None:

je_auto_control/utils/executor/action_executor.py

@@ -3197,6 +3197,38 @@ def _grid_cell(boxes: Any, row: Any, col: Any,
                        row_tolerance=int(row_tolerance))
 
 
+def _match_template(template: str, min_score: Any = 0.8, scales: Any = None,
+                    region: Any = None,
+                    method: str = "ccoeff_normed") -> Dict[str, Any]:
+    """Adapter: best confidence-scored template match on the screen."""
+    import json
+    from je_auto_control.utils.visual_match import match_template
+    if isinstance(scales, str):
+        scales = json.loads(scales) if scales.strip() else None
+    if isinstance(region, str):
+        region = json.loads(region) if region.strip() else None
+    match = match_template(template, region=region,
+                           scales=tuple(scales) if scales else (1.0,),
+                           min_score=float(min_score), method=method)
+    return {"found": match is not None,
+            "match": match.to_dict() if match else None}
+
+
+def _match_template_all(template: str, min_score: Any = 0.8,
+                        max_results: Any = 20, nms_iou: Any = 0.3,
+                        region: Any = None) -> Dict[str, Any]:
+    """Adapter: every confidence-scored template match on the screen (NMS)."""
+    import json
+    from je_auto_control.utils.visual_match import match_template_all
+    if isinstance(region, str):
+        region = json.loads(region) if region.strip() else None
+    matches = match_template_all(template, region=region,
+                                 min_score=float(min_score),
+                                 max_results=int(max_results),
+                                 nms_iou=float(nms_iou))
+    return {"count": len(matches), "matches": [m.to_dict() for m in matches]}
+
+
 def _with_modifiers(modifiers: Any, actions: Any) -> Dict[str, Any]:
     """Adapter: run nested actions while modifier keys are held down."""
     import json
@@ -4914,6 +4946,8 @@ def __init__(self):
             "AC_type_unicode": _type_unicode,
             "AC_with_modifiers": _with_modifiers,
             "AC_grid_cell": _grid_cell,
+            "AC_match_template": _match_template,
+            "AC_match_template_all": _match_template_all,
             "AC_detect_drift": _detect_drift,
             "AC_categorical_drift": _categorical_drift,
             "AC_diff_rows": _diff_rows,

je_auto_control/utils/mcp_server/tools/_factories.py

@@ -2628,6 +2628,42 @@ def tween_drag_tools() -> List[MCPTool]:
     ]
 
 
+def visual_match_tools() -> List[MCPTool]:
+    return [
+        MCPTool(
+            name="ac_match_template",
+            description=("Find 'template' (image path) on screen and return the "
+                         "match WITH its score: {found, match:{x,y,width,height,"
+                         "score,scale,center}}. 'scales' [..] for DPI/zoom, "
+                         "'min_score', 'region', 'method'."),
+            input_schema=schema({
+                "template": {"type": "string"},
+                "min_score": {"type": "number"},
+                "scales": {"type": "array", "items": {"type": "number"}},
+                "region": {"type": "array", "items": {"type": "integer"}},
+                "method": {"type": "string"}},
+                required=["template"]),
+            handler=h.match_template,
+            annotations=READ_ONLY,
+        ),
+        MCPTool(
+            name="ac_match_template_all",
+            description=("Find EVERY occurrence of 'template' on screen "
+                         ">= 'min_score', overlaps removed by NMS. "
+                         "Returns {count, matches}."),
+            input_schema=schema({
+                "template": {"type": "string"},
+                "min_score": {"type": "number"},
+                "max_results": {"type": "integer"},
+                "nms_iou": {"type": "number"},
+                "region": {"type": "array", "items": {"type": "integer"}}},
+                required=["template"]),
+            handler=h.match_template_all,
+            annotations=READ_ONLY,
+        ),
+    ]
+
+
 def grid_locator_tools() -> List[MCPTool]:
     return [
         MCPTool(
@@ -6018,8 +6054,8 @@ def media_assert_tools() -> List[MCPTool]:
     ci_annotation_tools, clipboard_history_tools, audit_analysis_tools,
     process_doc_tools, tween_drag_tools, mouse_path_tools, field_entry_tools,
     key_hold_tools, mouse_relative_tools, text_unicode_tools,
-    modifier_state_tools, grid_locator_tools, plugin_sdk_tools,
-    governance_tools,
+    modifier_state_tools, grid_locator_tools, visual_match_tools,
+    plugin_sdk_tools, governance_tools,
     credential_lease_tools, egress_tools, approval_testing_tools,
     trajectory_eval_tools, compliance_tools, agent_trace_tools,
     video_report_tools, fuzzy_tools, artifact_store_tools, image_dedup_tools,

je_auto_control/utils/mcp_server/tools/_handlers.py

@@ -2068,6 +2068,19 @@ def grid_cell(boxes, row, col, row_tolerance=10):
     return _grid_cell(boxes, row, col, row_tolerance)
 
 
+def match_template(template, min_score=0.8, scales=None, region=None,
+                   method="ccoeff_normed"):
+    from je_auto_control.utils.executor.action_executor import _match_template
+    return _match_template(template, min_score, scales, region, method)
+
+
+def match_template_all(template, min_score=0.8, max_results=20, nms_iou=0.3,
+                       region=None):
+    from je_auto_control.utils.executor.action_executor import (
+        _match_template_all)
+    return _match_template_all(template, min_score, max_results, nms_iou, region)
+
+
 def detect_drift(reference, current, threshold=0.25, bins=10):
     from je_auto_control.utils.executor.action_executor import _detect_drift
     return _detect_drift(reference, current, threshold, bins)

je_auto_control/utils/visual_match/__init__.py

@@ -0,0 +1,6 @@
+"""Confidence-returning template matching (score, multi-scale, find-all + NMS)."""
+from je_auto_control.utils.visual_match.visual_match import (
+    Match, best_matches, match_template, match_template_all,
+)
+
+__all__ = ["Match", "best_matches", "match_template", "match_template_all"]