Add match_ensemble: multi-template consensus matching

README/WHATS_NEW_zh-CN.md

@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-24) — 多模板共识匹配
+
+把同一目标的多个参考裁切投票成单一可信位置。完整参考:[`docs/source/Zh/doc/new_features/v178_features_doc.rst`](../docs/source/Zh/doc/new_features/v178_features_doc.rst)。
+
+- **`match_ensemble` / `vote_centers`**(`AC_match_ensemble`、`AC_vote_centers`):一个按钮以多种状态呈现(默认/悬停/按下)但是单一逻辑目标;`ab_locator` 只选一个策略、`match_template(scales=...)` 只扫一个模板——两者都不融合多参考。本功能匹配每个参考,聚类命中中心,只有在 ≥ `min_votes` 个于 `agree_px` 内一致时才接受,返回 `{point, votes, n_candidates, spread}`——减少换肤/动画 UI 的误判。重用 `visual_match.match_template` + `grounding_consensus`;`vote_centers` 为纯投票核心。不导入 `PySide6`。
+
 ## 本次更新 (2026-06-24) — 逐步评审特征 + 规则式步骤评分
 
 把为代理步骤评分所需的证据打包,并内建规则式评分器。完整参考:[`docs/source/Zh/doc/new_features/v177_features_doc.rst`](../docs/source/Zh/doc/new_features/v177_features_doc.rst)。

README/WHATS_NEW_zh-TW.md

@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-24) — 多模板共識比對
+
+把同一目標的多個參考裁切投票成單一可信位置。完整參考:[`docs/source/Zh/doc/new_features/v178_features_doc.rst`](../docs/source/Zh/doc/new_features/v178_features_doc.rst)。
+
+- **`match_ensemble` / `vote_centers`**(`AC_match_ensemble`、`AC_vote_centers`):一個按鈕以多種狀態呈現(預設/懸停/按下)但是單一邏輯目標;`ab_locator` 只選一個策略、`match_template(scales=...)` 只掃一個模板——兩者都不融合多參考。本功能比對每個參考,聚類命中中心,只有在 ≥ `min_votes` 個於 `agree_px` 內一致時才接受,回傳 `{point, votes, n_candidates, spread}`——減少換膚/動畫 UI 的誤判。重用 `visual_match.match_template` + `grounding_consensus`;`vote_centers` 為純投票核心。不匯入 `PySide6`。
+
 ## 本次更新 (2026-06-24) — 逐步評審特徵 + 規則式步驟評分
 
 把為代理步驟評分所需的證據打包,並內建規則式評分器。完整參考:[`docs/source/Zh/doc/new_features/v177_features_doc.rst`](../docs/source/Zh/doc/new_features/v177_features_doc.rst)。

WHATS_NEW.md

@@ -1,5 +1,11 @@
 # What's New — AutoControl
 
+## What's new (2026-06-24) — Multi-Template Consensus Matching
+
+Vote several reference crops of one target into a single trustworthy location. Full reference: [`docs/source/Eng/doc/new_features/v178_features_doc.rst`](docs/source/Eng/doc/new_features/v178_features_doc.rst).
+
+- **`match_ensemble` / `vote_centers`** (`AC_match_ensemble`, `AC_vote_centers`): a button renders in several states (default/hover/pressed) but is one logical target; `ab_locator` picks one strategy and `match_template(scales=...)` sweeps one template — neither fuses multiple references. This matches each reference, clusters the hit centres, and accepts a target only when ≥ `min_votes` agree within `agree_px`, returning `{point, votes, n_candidates, spread}` — cutting false positives on themed/animated UI. Reuses `visual_match.match_template` + `grounding_consensus`; `vote_centers` is the pure voting core. No `PySide6`.
+
 ## What's new (2026-06-24) — Per-Step Critic Features + Rule-Based Step Scorer
 
 Bundle the evidence to score an agent step, with a built-in rule-based scorer. Full reference: [`docs/source/Eng/doc/new_features/v177_features_doc.rst`](docs/source/Eng/doc/new_features/v177_features_doc.rst).

docs/source/Eng/doc/new_features/v178_features_doc.rst

@@ -0,0 +1,43 @@
+Multi-Template Consensus Matching
+=================================
+
+A button often renders in several states — default / hover / pressed / disabled — yet it is a
+single logical target. ``ab_locator`` A/B-tests *which single strategy wins* and
+``match_template(scales=...)`` sweeps *one* template across scales — neither fuses *multiple
+reference crops* into one vote. ``match_ensemble`` matches each reference, clusters the hit
+centres and accepts a target only when at least ``min_votes`` references agree within
+``agree_px`` — sharply cutting false positives on themed / animated UI.
+
+The voting core (``vote_centers``) is pure-stdlib and reuses ``grounding_consensus`` for the
+clustering, so it is unit-testable with no image; only ``match_ensemble`` itself calls
+``visual_match.match_template`` (testable on a synthetic injected ``haystack``). Imports no
+``PySide6``.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import match_ensemble, vote_centers
+
+    # three reference crops of the same button (default / hover / pressed)
+    result = match_ensemble(["btn_default.png", "btn_hover.png", "btn_pressed.png"],
+                            min_score=0.8, agree_px=10, min_votes=2)
+    if result:
+        click(*result["point"])     # {point, votes, n_candidates, spread}
+
+    # or vote hit centres you already have
+    vote_centers([[100, 100], [102, 98], [400, 400]], agree_px=10, min_votes=2)
+
+``match_ensemble`` returns ``{point, votes, n_candidates, spread}`` for the consensus location,
+or ``None`` if fewer than ``min_votes`` references agree. ``vote_centers`` is the pure voting
+step over candidate centres you supply.
+
+Executor commands
+-----------------
+
+``AC_match_ensemble`` (``templates`` / ``min_score`` / ``agree_px`` / ``min_votes`` /
+``region`` → ``{found, result}``) and ``AC_vote_centers`` (``centers`` / ``agree_px`` /
+``min_votes`` → ``{found, result}``). They are exposed as the MCP tools ``ac_match_ensemble`` /
+``ac_vote_centers`` (read-only) and as the Script Builder commands **Match Ensemble (vote
+references)** / **Vote Centers (consensus)** under **Image**.

docs/source/Eng/eng_index.rst

@@ -200,6 +200,7 @@ Comprehensive guides for all AutoControl features.
    doc/new_features/v175_features_doc
    doc/new_features/v176_features_doc
    doc/new_features/v177_features_doc
+   doc/new_features/v178_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

docs/source/Zh/doc/new_features/v178_features_doc.rst

@@ -0,0 +1,39 @@
+多模板共識比對
+==============
+
+一個按鈕常以多種狀態呈現——預設 / 懸停 / 按下 / 停用——但它是單一邏輯目標。``ab_locator``
+A/B 測試*哪個單一策略勝出*,``match_template(scales=...)`` 跨縮放掃描*一個*模板——兩者都不把
+*多個參考裁切*融合成一次投票。``match_ensemble`` 比對每個參考,聚類命中中心,只有在至少
+``min_votes`` 個參考於 ``agree_px`` 內一致時才接受目標——大幅減少換膚 / 動畫 UI 上的誤判。
+
+投票核心(``vote_centers``)為純標準函式庫,並重用 ``grounding_consensus`` 做聚類,因此可在
+無影像下單元測試;只有 ``match_ensemble`` 本身呼叫 ``visual_match.match_template``(可於注入的
+合成 ``haystack`` 上測試)。不匯入 ``PySide6``。
+
+無頭 API
+--------
+
+.. code-block:: python
+
+    from je_auto_control import match_ensemble, vote_centers
+
+    # 同一按鈕的三個參考裁切(預設 / 懸停 / 按下)
+    result = match_ensemble(["btn_default.png", "btn_hover.png", "btn_pressed.png"],
+                            min_score=0.8, agree_px=10, min_votes=2)
+    if result:
+        click(*result["point"])     # {point, votes, n_candidates, spread}
+
+    # 或對你已有的命中中心投票
+    vote_centers([[100, 100], [102, 98], [400, 400]], agree_px=10, min_votes=2)
+
+``match_ensemble`` 回傳共識位置的 ``{point, votes, n_candidates, spread}``,或在少於
+``min_votes`` 個參考一致時回傳 ``None``。``vote_centers`` 是對你提供的候選中心的純投票步驟。
+
+執行器指令
+----------
+
+``AC_match_ensemble``(``templates`` / ``min_score`` / ``agree_px`` / ``min_votes`` /
+``region`` → ``{found, result}``)與 ``AC_vote_centers``(``centers`` / ``agree_px`` /
+``min_votes`` → ``{found, result}``)。兩者以 MCP 工具 ``ac_match_ensemble`` /
+``ac_vote_centers``(唯讀)及 Script Builder 指令 **Match Ensemble (vote references)** /
+**Vote Centers (consensus)**(位於 **Image** 分類下)形式提供。

docs/source/Zh/zh_index.rst

@@ -200,6 +200,7 @@ AutoControl 所有功能的完整使用指南。
    doc/new_features/v175_features_doc
    doc/new_features/v176_features_doc
    doc/new_features/v177_features_doc
+   doc/new_features/v178_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

je_auto_control/__init__.py

@@ -291,6 +291,10 @@
 from je_auto_control.utils.edge_match import (
     chamfer_distance, edge_match, edge_match_all,
 )
+# Multi-template consensus matching (vote references onto one location)
+from je_auto_control.utils.match_ensemble import (
+    match_ensemble, vote_centers,
+)
 # Otsu auto-thresholding for template matching (no hand-tuned min_score)
 from je_auto_control.utils.match_autothresh import (
     auto_threshold, match_auto,
@@ -1274,6 +1278,8 @@ def start_autocontrol_gui(*args, **kwargs):
     "edge_match",
     "edge_match_all",
     "chamfer_distance",
+    "match_ensemble",
+    "vote_centers",
     "SubPixelMatch",
     "match_subpixel",
     "refine_peak",

je_auto_control/gui/script_builder/command_schema.py

@@ -407,6 +407,30 @@ def _add_image_specs(specs: List[CommandSpec]) -> None:
         ),
         description="Match with a sub-pixel-refined centre (drag / slider precision).",
     ))
+    specs.append(CommandSpec(
+        "AC_match_ensemble", "Image", "Match Ensemble (vote references)",
+        fields=(
+            FieldSpec("templates", FieldType.STRING,
+                      placeholder='["default.png", "hover.png", "pressed.png"]'),
+            FieldSpec("min_score", FieldType.FLOAT, optional=True, default=0.8,
+                      min_value=0.0, max_value=1.0),
+            FieldSpec("agree_px", FieldType.INT, optional=True, default=10),
+            FieldSpec("min_votes", FieldType.INT, optional=True, default=2),
+            FieldSpec("region", FieldType.STRING, optional=True,
+                      placeholder=_REGION_PLACEHOLDER),
+        ),
+        description="Vote several reference crops onto one consensus location.",
+    ))
+    specs.append(CommandSpec(
+        "AC_vote_centers", "Image", "Vote Centers (consensus)",
+        fields=(
+            FieldSpec("centers", FieldType.STRING,
+                      placeholder="[[100, 100], [102, 98]]"),
+            FieldSpec("agree_px", FieldType.INT, optional=True, default=10),
+            FieldSpec("min_votes", FieldType.INT, optional=True, default=2),
+        ),
+        description="Vote candidate hit centres into one consensus target.",
+    ))
     specs.append(CommandSpec(
         "AC_grid_cells", "Image", "Grid Cells (coarse grounding)",
         fields=(

je_auto_control/utils/executor/action_executor.py

@@ -3400,6 +3400,32 @@ def _match_subpixel(template: str, min_score: Any = 0.0, region: Any = None,
             "match": match.to_dict() if match else None}
 
 
+def _vote_centers(centers: Any, agree_px: Any = 10,
+                  min_votes: Any = 2) -> Dict[str, Any]:
+    """Adapter: vote candidate hit centres into a consensus target."""
+    import json
+    from je_auto_control.utils.match_ensemble import vote_centers
+    if isinstance(centers, str):
+        centers = json.loads(centers)
+    result = vote_centers(centers, agree_px=float(agree_px),
+                          min_votes=int(min_votes))
+    return {"found": result is not None, "result": result}
+
+
+def _match_ensemble(templates: Any, min_score: Any = 0.8, agree_px: Any = 10,
+                    min_votes: Any = 2, region: Any = None) -> Dict[str, Any]:
+    """Adapter: vote several template references onto one consensus location."""
+    import json
+    from je_auto_control.utils.match_ensemble import match_ensemble
+    if isinstance(templates, str):
+        templates = json.loads(templates)
+    if isinstance(region, str):
+        region = json.loads(region) if region.strip() else None
+    result = match_ensemble(templates, region=region, min_score=float(min_score),
+                            agree_px=float(agree_px), min_votes=int(min_votes))
+    return {"found": result is not None, "result": result}
+
+
 def _region_arg(value: Any) -> Optional[List[int]]:
     """Coerce a JSON-string / list region arg into a list of ints, or None."""
     import json
@@ -6123,6 +6149,8 @@ def __init__(self):
             "AC_edge_match": _edge_match,
             "AC_edge_match_all": _edge_match_all,
             "AC_match_subpixel": _match_subpixel,
+            "AC_match_ensemble": _match_ensemble,
+            "AC_vote_centers": _vote_centers,
             "AC_grid_cells": _grid_cells,
             "AC_cell_for_point": _cell_for_point,
             "AC_point_for_cell": _point_for_cell,

je_auto_control/utils/match_ensemble/__init__.py

@@ -0,0 +1,6 @@
+"""Multi-template consensus matching (vote several references onto one location)."""
+from je_auto_control.utils.match_ensemble.match_ensemble import (
+    match_ensemble, vote_centers,
+)
+
+__all__ = ["match_ensemble", "vote_centers"]

je_auto_control/utils/match_ensemble/match_ensemble.py

@@ -0,0 +1,57 @@
+"""Multi-template consensus matching (vote several references onto one location).
+
+A button often renders in several states — default / hover / pressed / disabled — yet it is a
+single logical target. ``ab_locator`` A/B-tests *which single strategy wins* and
+``match_template(scales=...)`` sweeps *one* template across scales — neither fuses *multiple
+reference crops* into one vote. ``match_ensemble`` matches each reference, clusters the hit
+centres and accepts a target only when at least ``min_votes`` references agree within
+``agree_px`` — sharply cutting false positives on themed / animated UI.
+
+The voting core (``vote_centers``) is pure-stdlib and reuses ``grounding_consensus`` for the
+clustering, so it is unit-testable with no image; only ``match_ensemble`` itself calls
+``visual_match.match_template`` (and is testable on a synthetic injected ``haystack``). Imports
+no ``PySide6``.
+"""
+from typing import Any, Dict, List, Optional, Sequence
+
+ImageSource = Any
+
+
+def vote_centers(centers: Sequence[Sequence[int]], *, agree_px: float = 10,
+                 min_votes: int = 2) -> Optional[Dict[str, Any]]:
+    """Cluster candidate hit centres and return the agreed target if enough agree.
+
+    Returns ``{point, votes, n_candidates, spread}`` for the largest cluster, or ``None``
+    when fewer than ``min_votes`` candidates fall within ``agree_px`` of each other.
+    """
+    from je_auto_control.utils.grounding_consensus import consensus_point
+    result = consensus_point(centers, cluster_radius=float(agree_px))
+    if result is None:
+        return None
+    votes = round(result.agreement * len(centers))
+    if votes < int(min_votes):
+        return None
+    return {"point": result.point, "votes": votes,
+            "n_candidates": len(centers), "spread": result.spread}
+
+
+def match_ensemble(templates: Sequence[ImageSource], *,
+                   haystack: Optional[ImageSource] = None,
+                   region: Optional[Sequence[int]] = None,
+                   scales: Sequence[float] = (1.0,), min_score: float = 0.8,
+                   agree_px: float = 10, min_votes: int = 2
+                   ) -> Optional[Dict[str, Any]]:
+    """Match each reference in ``templates`` and return their voted consensus target.
+
+    Each template is located with ``visual_match.match_template``; the hit centres are then
+    fused by :func:`vote_centers`. Returns ``{point, votes, n_candidates, spread}`` or
+    ``None`` if too few references agree.
+    """
+    from je_auto_control.utils.visual_match import match_template
+    centers: List[List[int]] = []
+    for template in templates:
+        match = match_template(template, haystack=haystack, region=region,
+                               scales=tuple(scales), min_score=float(min_score))
+        if match is not None:
+            centers.append(match.center)
+    return vote_centers(centers, agree_px=agree_px, min_votes=min_votes)

je_auto_control/utils/mcp_server/tools/_factories.py

@@ -3847,6 +3847,38 @@ def rotated_match_tools() -> List[MCPTool]:
             handler=h.match_subpixel,
             annotations=READ_ONLY,
         ),
+        MCPTool(
+            name="ac_match_ensemble",
+            description=("Match several reference crops of one target ('templates': "
+                         "image paths — e.g. default / hover / pressed states) and "
+                         "return their VOTED consensus location: {found, result:"
+                         "{point, votes, n_candidates, spread}}. Accepts only when "
+                         ">= 'min_votes' references agree within 'agree_px'. Cuts false "
+                         "positives on themed / animated UI. 'min_score', 'region'."),
+            input_schema=schema({
+                "templates": {"type": "array", "items": {"type": "string"}},
+                "min_score": {"type": "number"},
+                "agree_px": {"type": "number"},
+                "min_votes": {"type": "integer"},
+                "region": {"type": "array", "items": {"type": "integer"}}},
+                required=["templates"]),
+            handler=h.match_ensemble,
+            annotations=READ_ONLY,
+        ),
+        MCPTool(
+            name="ac_vote_centers",
+            description=("Vote candidate hit 'centers' ([[x,y],...]) into one consensus "
+                         "target: {found, result:{point, votes, n_candidates, spread}}. "
+                         "Accepts only when >= 'min_votes' agree within 'agree_px'."),
+            input_schema=schema({
+                "centers": {"type": "array",
+                            "items": {"type": "array", "items": {"type": "integer"}}},
+                "agree_px": {"type": "number"},
+                "min_votes": {"type": "integer"}},
+                required=["centers"]),
+            handler=h.vote_centers,
+            annotations=READ_ONLY,
+        ),
     ]
 
 

je_auto_control/utils/mcp_server/tools/_handlers.py

@@ -2142,6 +2142,16 @@ def match_subpixel(template, min_score=0.0, region=None, method="ccoeff_normed")
     return _match_subpixel(template, min_score, region, method)
 
 
+def vote_centers(centers, agree_px=10, min_votes=2):
+    from je_auto_control.utils.executor.action_executor import _vote_centers
+    return _vote_centers(centers, agree_px, min_votes)
+
+
+def match_ensemble(templates, min_score=0.8, agree_px=10, min_votes=2, region=None):
+    from je_auto_control.utils.executor.action_executor import _match_ensemble
+    return _match_ensemble(templates, min_score, agree_px, min_votes, region)
+
+
 def grid_cells(rows, cols, region=None):
     from je_auto_control.utils.executor.action_executor import _grid_cells
     return _grid_cells(rows, cols, region)