Add masked template matching (ignore background via mask/alpha)

README/WHATS_NEW_zh-CN.md

@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-23) — 遮罩模板匹配
+
+不论背景如何都能匹配图标。完整参考:[`docs/source/Zh/doc/new_features/v129_features_doc.rst`](../docs/source/Zh/doc/new_features/v129_features_doc.rst)。
+
+- **`match_masked` / `match_masked_all`**(`AC_match_masked`、`AC_match_masked_all`):一般模板匹配会计分*每个*像素,因此从某背景裁切出的图标在不同背景上会匹配失败。本功能只计算你标记为相关的像素——明确的灰阶 `mask`,或 RGBA 模板的 alpha 通道——让透明 /「不在乎」的像素不再拉低分数。返回与计分模板匹配相同的 `Match`(score/center);使用 OpenCV 遮罩 `TM_CCORR_NORMED`,NaN 归零。可注入 haystack → 无头可测。
+
 ## 本次更新 (2026-06-23) — 依颜色定位屏幕区域
 
 依颜色找出绿色状态药丸 / 红色横幅。完整参考:[`docs/source/Zh/doc/new_features/v128_features_doc.rst`](../docs/source/Zh/doc/new_features/v128_features_doc.rst)。

README/WHATS_NEW_zh-TW.md

@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-23) — 遮罩模板比對
+
+不論背景如何都能比對圖示。完整參考:[`docs/source/Zh/doc/new_features/v129_features_doc.rst`](../docs/source/Zh/doc/new_features/v129_features_doc.rst)。
+
+- **`match_masked` / `match_masked_all`**(`AC_match_masked`、`AC_match_masked_all`):一般模板比對會計分*每個*像素,因此從某背景裁切出的圖示在不同背景上會比對失敗。本功能只計算你標記為相關的像素——明確的灰階 `mask`,或 RGBA 模板的 alpha 通道——讓透明 /「不在乎」的像素不再拉低分數。回傳與計分模板比對相同的 `Match`(score/center);使用 OpenCV 遮罩 `TM_CCORR_NORMED`,NaN 歸零。可注入 haystack → 無頭可測。
+
 ## 本次更新 (2026-06-23) — 依顏色定位螢幕區域
 
 依顏色找出綠色狀態藥丸 / 紅色橫幅。完整參考:[`docs/source/Zh/doc/new_features/v128_features_doc.rst`](../docs/source/Zh/doc/new_features/v128_features_doc.rst)。

WHATS_NEW.md

@@ -1,5 +1,11 @@
 # What's New — AutoControl
 
+## What's new (2026-06-23) — Masked Template Matching
+
+Match icons regardless of their background. Full reference: [`docs/source/Eng/doc/new_features/v129_features_doc.rst`](docs/source/Eng/doc/new_features/v129_features_doc.rst).
+
+- **`match_masked` / `match_masked_all`** (`AC_match_masked`, `AC_match_masked_all`): plain template matching scores *every* pixel, so an icon clipped from one background fails over a different one. These count only the pixels you mark relevant — an explicit grayscale `mask`, or an RGBA template's alpha channel — so transparent / "don't care" pixels stop dragging the score down. Returns the same `Match` (score/center) as scored template matching; OpenCV masked `TM_CCORR_NORMED`, NaNs zeroed. Injectable haystack → headless-testable.
+
 ## What's new (2026-06-23) — Locate On-Screen Regions by Colour
 
 Find the green status pill / red banner by colour. Full reference: [`docs/source/Eng/doc/new_features/v128_features_doc.rst`](docs/source/Eng/doc/new_features/v128_features_doc.rst).

docs/source/Eng/doc/new_features/v129_features_doc.rst

@@ -0,0 +1,47 @@
+Masked Template Matching (Ignore the Background)
+================================================
+
+Plain template matching scores *every* pixel of the template, so an icon clipped
+from one background fails to match the same icon over a different one — a toolbar
+glyph on a hovered vs. idle button, a cursor over arbitrary content, a logo on a
+themed surface. ``match_masked`` counts only the pixels you mark as relevant: an
+explicit grayscale ``mask`` (non-zero = use), or — if you pass an RGBA template —
+its alpha channel. The transparent / "don't care" pixels stop dragging the score
+down.
+
+It builds on the same ``Match`` result as :doc:`v127_features_doc` (top-left,
+size, ``score``, ``center``) and runs on an injectable ``haystack`` (ndarray /
+path / PIL), so it is unit-testable on synthetic arrays. Matching uses OpenCV's
+masked ``TM_CCORR_NORMED`` (the only normed metric that accepts a mask without
+producing NaNs); non-finite cells are zeroed. OpenCV + NumPy come in via
+``je_open_cv``; imports no ``PySide6``.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import match_masked, match_masked_all
+
+    # A PNG icon with transparency — its alpha is the mask automatically.
+    hit = match_masked("save_icon.png", min_score=0.9)
+    if hit:
+        click(*hit.center)
+
+    # An explicit mask: only the white pixels of mask.png are compared.
+    for hit in match_masked_all("cursor.png", mask="cursor_mask.png",
+                                min_score=0.95):
+        print(hit.x, hit.y, hit.score)
+
+``match_masked`` returns the single best ``Match`` at or above ``min_score`` (or
+``None``); ``match_masked_all`` returns every match with overlaps removed by
+non-maximum suppression, highest score first, capped at ``max_results``. A mask
+whose shape does not match the template raises ``ValueError``.
+
+Executor commands
+-----------------
+
+``AC_match_masked`` / ``AC_match_masked_all`` take ``template`` (and optional
+``mask``) plus ``min_score`` / ``region`` (and ``max_results`` / ``nms_iou`` for
+the *all* form). They are exposed as the MCP tools ``ac_match_masked`` /
+``ac_match_masked_all`` and as Script Builder commands under **Image**.

docs/source/Eng/eng_index.rst

@@ -151,6 +151,7 @@ Comprehensive guides for all AutoControl features.
    doc/new_features/v126_features_doc
    doc/new_features/v127_features_doc
    doc/new_features/v128_features_doc
+   doc/new_features/v129_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

docs/source/Zh/doc/new_features/v129_features_doc.rst

@@ -0,0 +1,39 @@
+遮罩模板比對(忽略背景)
+========================
+
+一般模板比對會計分模板的*每個*像素,因此從某背景裁切出的圖示無法比對到同一圖示在不同背景上的情形——
+工具列圖示在 hover 與閒置按鈕上、游標疊在任意內容上、Logo 在主題化表面上。``match_masked`` 只計算你標記為
+相關的像素:明確的灰階 ``mask``(非零 = 使用),或——若傳入 RGBA 模板——其 alpha 通道。透明 /「不在乎」的
+像素就不會再把分數拉低。
+
+它沿用與 :doc:`v127_features_doc` 相同的 ``Match`` 結果(左上角、尺寸、``score``、``center``),並在可注入的
+``haystack``(ndarray / 路徑 / PIL)上執行,因此可對合成陣列做單元測試。比對使用 OpenCV 的遮罩
+``TM_CCORR_NORMED``(唯一能接受遮罩且不產生 NaN 的正規化度量);非有限值會被歸零。OpenCV + NumPy 透過
+``je_open_cv`` 引入;不匯入 ``PySide6``。
+
+無頭 API
+--------
+
+.. code-block:: python
+
+    from je_auto_control import match_masked, match_masked_all
+
+    # 帶透明度的 PNG 圖示——其 alpha 自動作為遮罩。
+    hit = match_masked("save_icon.png", min_score=0.9)
+    if hit:
+        click(*hit.center)
+
+    # 明確遮罩:只比對 mask.png 的白色像素。
+    for hit in match_masked_all("cursor.png", mask="cursor_mask.png",
+                                min_score=0.95):
+        print(hit.x, hit.y, hit.score)
+
+``match_masked`` 回傳達到 ``min_score`` 的單一最佳 ``Match``(或 ``None``);``match_masked_all`` 回傳每個
+比對,以非極大值抑制移除重疊,分數由高到低,上限 ``max_results``。遮罩形狀與模板不符會丟出 ``ValueError``。
+
+執行器命令
+----------
+
+``AC_match_masked`` / ``AC_match_masked_all`` 接受 ``template``(及選用 ``mask``)以及
+``min_score`` / ``region``(*all* 形式另有 ``max_results`` / ``nms_iou``)。它們以 MCP 工具
+``ac_match_masked`` / ``ac_match_masked_all`` 以及 Script Builder 中 **Image** 分類下的命令提供。

docs/source/Zh/zh_index.rst

@@ -151,6 +151,7 @@ AutoControl 所有功能的完整使用指南。
    doc/new_features/v126_features_doc
    doc/new_features/v127_features_doc
    doc/new_features/v128_features_doc
+   doc/new_features/v129_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

je_auto_control/__init__.py

@@ -275,7 +275,8 @@
 from je_auto_control.utils.grid_locator import cluster_grid, locate_cell
 # Confidence-returning template matching (score / multi-scale / find-all + NMS)
 from je_auto_control.utils.visual_match import (
-    best_matches, match_template, match_template_all,
+    best_matches, match_masked, match_masked_all, match_template,
+    match_template_all,
 )
 from je_auto_control.utils.visual_match import Match as TemplateMatch
 # Locate on-screen regions by colour (mask + connected components)
@@ -1074,6 +1075,8 @@ def start_autocontrol_gui(*args, **kwargs):
     "TemplateMatch",
     "match_template",
     "match_template_all",
+    "match_masked",
+    "match_masked_all",
     "best_matches",
     "find_color_region",
     "find_color_regions",

je_auto_control/gui/script_builder/command_schema.py

@@ -262,7 +262,7 @@
             FieldSpec("scales", FieldType.STRING, optional=True,
                       placeholder="[0.9, 1.0, 1.1]"),
             FieldSpec("region", FieldType.STRING, optional=True,
                       placeholder="[left, top, right, bottom]"),
         ),
         description="Locate a template and return its confidence score + scale.",
     ))
@@ -278,6 +278,31 @@
         ),
         description="Find every occurrence of a template (scored, NMS-deduped).",
     ))
+    specs.append(CommandSpec(
+        "AC_match_masked", "Image", "Match Masked Template",
+        fields=(
+            FieldSpec("template", FieldType.FILE_PATH),
+            FieldSpec("mask", FieldType.FILE_PATH, optional=True),
+            FieldSpec("min_score", FieldType.FLOAT, optional=True, default=0.9,
+                      min_value=0.0, max_value=1.0),
+            FieldSpec("region", FieldType.STRING, optional=True,
+                      placeholder="[left, top, right, bottom]"),
+        ),
+        description="Match counting only opaque/masked pixels (alpha or mask).",
+    ))
+    specs.append(CommandSpec(
+        "AC_match_masked_all", "Image", "Match Masked Template All",
+        fields=(
+            FieldSpec("template", FieldType.FILE_PATH),
+            FieldSpec("mask", FieldType.FILE_PATH, optional=True),
+            FieldSpec("min_score", FieldType.FLOAT, optional=True, default=0.9,
+                      min_value=0.0, max_value=1.0),
+            FieldSpec("max_results", FieldType.INT, optional=True, default=20),
+            FieldSpec("nms_iou", FieldType.FLOAT, optional=True, default=0.3,
+                      min_value=0.0, max_value=1.0),
+        ),
+        description="Find every masked match of a template (NMS-deduped).",
+    ))
     specs.append(CommandSpec(
         "AC_find_color_region", "Image", "Find Colour Region",
         fields=(

je_auto_control/utils/executor/action_executor.py

@@ -3229,6 +3229,34 @@ def _match_template_all(template: str, min_score: Any = 0.8,
     return {"count": len(matches), "matches": [m.to_dict() for m in matches]}
 
 
+def _match_masked(template: str, mask: Any = None, min_score: Any = 0.9,
+                  region: Any = None) -> Dict[str, Any]:
+    """Adapter: best masked template match (alpha / mask ignores background)."""
+    import json
+    from je_auto_control.utils.visual_match import match_masked
+    if isinstance(region, str):
+        region = json.loads(region) if region.strip() else None
+    match = match_masked(template, mask=mask, region=region,
+                         min_score=float(min_score))
+    return {"found": match is not None,
+            "match": match.to_dict() if match else None}
+
+
+def _match_masked_all(template: str, mask: Any = None, min_score: Any = 0.9,
+                      max_results: Any = 20, nms_iou: Any = 0.3,
+                      region: Any = None) -> Dict[str, Any]:
+    """Adapter: every masked template match on the screen (NMS)."""
+    import json
+    from je_auto_control.utils.visual_match import match_masked_all
+    if isinstance(region, str):
+        region = json.loads(region) if region.strip() else None
+    matches = match_masked_all(template, mask=mask, region=region,
+                               min_score=float(min_score),
+                               max_results=int(max_results),
+                               nms_iou=float(nms_iou))
+    return {"count": len(matches), "matches": [m.to_dict() for m in matches]}
+
+
 def _find_color_region(rgb: Any, tolerance: Any = 20, min_area: Any = 50,
                        region: Any = None) -> Dict[str, Any]:
     """Adapter: locate coloured regions on the screen, largest first."""
@@ -4964,6 +4992,8 @@ def __init__(self):
             "AC_grid_cell": _grid_cell,
             "AC_match_template": _match_template,
             "AC_match_template_all": _match_template_all,
+            "AC_match_masked": _match_masked,
+            "AC_match_masked_all": _match_masked_all,
             "AC_find_color_region": _find_color_region,
             "AC_detect_drift": _detect_drift,
             "AC_categorical_drift": _categorical_drift,

je_auto_control/utils/mcp_server/tools/_factories.py

@@ -2681,6 +2681,37 @@ def visual_match_tools() -> List[MCPTool]:
             handler=h.match_template_all,
             annotations=READ_ONLY,
         ),
+        MCPTool(
+            name="ac_match_masked",
+            description=("Find 'template' on screen counting only masked/opaque "
+                         "pixels: a grayscale 'mask' (non-zero = use) or, if "
+                         "omitted, the template's RGBA alpha. For glyphs/icons "
+                         "over a transparent or varying background. Returns "
+                         "{found, match}. 'min_score', 'region'."),
+            input_schema=schema({
+                "template": {"type": "string"},
+                "mask": {"type": "string"},
+                "min_score": {"type": "number"},
+                "region": {"type": "array", "items": {"type": "integer"}}},
+                required=["template"]),
+            handler=h.match_masked,
+            annotations=READ_ONLY,
+        ),
+        MCPTool(
+            name="ac_match_masked_all",
+            description=("Find EVERY masked match of 'template' >= 'min_score', "
+                         "overlaps removed by NMS. Returns {count, matches}."),
+            input_schema=schema({
+                "template": {"type": "string"},
+                "mask": {"type": "string"},
+                "min_score": {"type": "number"},
+                "max_results": {"type": "integer"},
+                "nms_iou": {"type": "number"},
+                "region": {"type": "array", "items": {"type": "integer"}}},
+                required=["template"]),
+            handler=h.match_masked_all,
+            annotations=READ_ONLY,
+        ),
     ]
 
 

je_auto_control/utils/mcp_server/tools/_handlers.py

@@ -2081,6 +2081,19 @@ def match_template_all(template, min_score=0.8, max_results=20, nms_iou=0.3,
     return _match_template_all(template, min_score, max_results, nms_iou, region)
 
 
+def match_masked(template, mask=None, min_score=0.9, region=None):
+    from je_auto_control.utils.executor.action_executor import _match_masked
+    return _match_masked(template, mask, min_score, region)
+
+
+def match_masked_all(template, mask=None, min_score=0.9, max_results=20,
+                     nms_iou=0.3, region=None):
+    from je_auto_control.utils.executor.action_executor import (
+        _match_masked_all)
+    return _match_masked_all(template, mask, min_score, max_results, nms_iou,
+                             region)
+
+
 def find_color_region(rgb, tolerance=20, min_area=50, region=None):
     from je_auto_control.utils.executor.action_executor import (
         _find_color_region)

je_auto_control/utils/visual_match/__init__.py

@@ -1,6 +1,8 @@
 """Confidence-returning template matching (score, multi-scale, find-all + NMS)."""
 from je_auto_control.utils.visual_match.visual_match import (
-    Match, best_matches, match_template, match_template_all,
+    Match, best_matches, match_masked, match_masked_all, match_template,
+    match_template_all,
 )
 
-__all__ = ["Match", "best_matches", "match_template", "match_template_all"]
+__all__ = ["Match", "best_matches", "match_masked", "match_masked_all",
+           "match_template", "match_template_all"]

je_auto_control/utils/visual_match/visual_match.py

@@ -170,3 +170,92 @@ def best_matches(template: ImageSource, *,
     """Return the top ``top_n`` matches by score (any score), nearest-best first."""
     return match_template_all(template, haystack=haystack, region=region,
                               min_score=-1.0, max_results=int(top_n))
+
+
+def _load_unchanged(source: ImageSource):
+    """Load a path / ndarray / PIL image keeping an alpha channel if present."""
+    import cv2
+    import numpy as np
+    if hasattr(source, "shape"):
+        return np.asarray(source)
+    if isinstance(source, (str, bytes)) or hasattr(source, "__fspath__"):
+        array = cv2.imread(str(source), cv2.IMREAD_UNCHANGED)
+        if array is None:
+            raise ValueError(f"could not read image: {source!r}")
+        return array
+    return np.asarray(source)
+
+
+def _template_and_mask(template: ImageSource, mask: Optional[ImageSource]):
+    """Return (gray_template, uint8_mask_or_None); alpha is the implicit mask."""
+    import cv2
+    import numpy as np
+    array = _load_unchanged(template)
+    if array.ndim == 3 and array.shape[2] == 4:
+        gray = cv2.cvtColor(array, cv2.COLOR_BGRA2GRAY)
+        implicit = array[:, :, 3]
+    else:
+        gray = _to_gray(array)
+        implicit = None
+    chosen = _to_gray(mask) if mask is not None else implicit
+    if chosen is None:
+        return gray, None
+    chosen = np.ascontiguousarray(chosen, dtype=np.uint8)
+    if chosen.shape[:2] != gray.shape[:2]:
+        raise ValueError("mask shape must match template shape")
+    return gray, chosen
+
+
+def _masked_scores(template: ImageSource, mask: Optional[ImageSource],
+                   haystack: Optional[ImageSource],
+                   region: Optional[Sequence[int]]):
+    """Return (score_map, gray_template) for masked correlation, NaNs zeroed."""
+    import cv2
+    import numpy as np
+    tmpl, msk = _template_and_mask(template, mask)
+    hay = _haystack_gray(haystack, region)
+    if tmpl.shape[0] > hay.shape[0] or tmpl.shape[1] > hay.shape[1]:
+        return None, tmpl
+    result = cv2.matchTemplate(hay, tmpl, cv2.TM_CCORR_NORMED, mask=msk)
+    return np.nan_to_num(result, nan=0.0, posinf=0.0, neginf=0.0), tmpl
+
+
+def match_masked(template: ImageSource, *, mask: Optional[ImageSource] = None,
+                 haystack: Optional[ImageSource] = None,
+                 region: Optional[Sequence[int]] = None,
+                 min_score: float = 0.9) -> Optional[Match]:
+    """Return the best match counting only masked (opaque) template pixels.
+
+    ``mask`` is a grayscale image where non-zero pixels participate; if omitted
+    and ``template`` is RGBA, its alpha channel is the mask. This finds icons /
+    buttons whose background is transparent or varies (a glyph over any colour)
+    where a plain template would be dragged down by the irrelevant pixels.
+    Returns ``None`` when nothing clears ``min_score``.
+    """
+    import cv2
+    scores, tmpl = _masked_scores(template, mask, haystack, region)
+    if scores is None:
+        return None
+    _, max_val, _, max_loc = cv2.minMaxLoc(scores)
+    if max_val < min_score:
+        return None
+    return Match(int(max_loc[0]), int(max_loc[1]), tmpl.shape[1], tmpl.shape[0],
+                 round(float(max_val), 4), 1.0)
+
+
+def match_masked_all(template: ImageSource, *, mask: Optional[ImageSource] = None,
+                     haystack: Optional[ImageSource] = None,
+                     region: Optional[Sequence[int]] = None,
+                     min_score: float = 0.9, max_results: int = 20,
+                     nms_iou: float = 0.3) -> List[Match]:
+    """Return every masked match >= ``min_score`` with overlaps removed (NMS)."""
+    import numpy as np
+    scores, tmpl = _masked_scores(template, mask, haystack, region)
+    if scores is None:
+        return []
+    height, width = tmpl.shape[:2]
+    ys, xs = np.nonzero(scores >= float(min_score))
+    candidates = [Match(int(x), int(y), width, height,
+                        round(float(scores[y, x]), 4), 1.0)
+                  for y, x in zip(ys, xs)]
+    return _nms(candidates, float(nms_iou))[:int(max_results)]