Add grounding_consensus: self-consistency over grounding proposals

README/WHATS_NEW_zh-CN.md

@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-24) — Grounding 自我一致性(提案共识)
+
+把多个 grounding 提案融合成单一一致目标,并附 agreement 分数。完整参考:[`docs/source/Zh/doc/new_features/v172_features_doc.rst`](../docs/source/Zh/doc/new_features/v172_features_doc.rst)。
+
+- **`consensus_point` / `consensus_element` / `is_confident`**(`AC_consensus_point`、`AC_consensus_element`):一个目标可同时以多种方式 grounding(set-of-marks / OCR / 模板 / a11y / 模型 N 次采样)而未必一致。`ab_locator`/`element_scoring` 依历史排序*策略*;`snap_to_element` 只贴*单一*坐标——两者都不融合*同时*的提案。本功能将候选点聚类(或对候选元素投票),返回一致的 `point` + `agreement` 比例 + `spread`,而 `is_confident` 标记低一致度目标,让代理改为放大 / 询问而非盲目点击。纯标准库;不导入 `PySide6`。
+
 ## 本次更新 (2026-06-24) — 亚像素模板匹配精修
 
 把匹配中心精修到像素的分数位,供拖曳 / 滑块 / 高 DPI 精度。完整参考:[`docs/source/Zh/doc/new_features/v171_features_doc.rst`](../docs/source/Zh/doc/new_features/v171_features_doc.rst)。

README/WHATS_NEW_zh-TW.md

@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-24) — Grounding 自我一致性(提案共識)
+
+把多個 grounding 提案融合成單一一致目標,並附 agreement 分數。完整參考:[`docs/source/Zh/doc/new_features/v172_features_doc.rst`](../docs/source/Zh/doc/new_features/v172_features_doc.rst)。
+
+- **`consensus_point` / `consensus_element` / `is_confident`**(`AC_consensus_point`、`AC_consensus_element`):一個目標可同時以多種方式 grounding(set-of-marks / OCR / 樣板 / a11y / 模型 N 次抽樣)而未必一致。`ab_locator`/`element_scoring` 依歷史排序*策略*;`snap_to_element` 只貼*單一*座標——兩者都不融合*同時*的提案。本功能將候選點聚類(或對候選元素投票),回傳一致的 `point` + `agreement` 比例 + `spread`,而 `is_confident` 標記低一致度目標,讓代理改為放大 / 詢問而非盲目點擊。純標準函式庫;不匯入 `PySide6`。
+
 ## 本次更新 (2026-06-24) — 次像素樣板比對精修
 
 把比對中心精修到像素的分數位,供拖曳 / 滑桿 / 高 DPI 精度。完整參考:[`docs/source/Zh/doc/new_features/v171_features_doc.rst`](../docs/source/Zh/doc/new_features/v171_features_doc.rst)。

WHATS_NEW.md

@@ -1,5 +1,11 @@
 # What's New — AutoControl
 
+## What's new (2026-06-24) — Grounding Self-Consistency (Consensus Over Proposals)
+
+Fuse several grounding proposals into one agreed target with an agreement score. Full reference: [`docs/source/Eng/doc/new_features/v172_features_doc.rst`](docs/source/Eng/doc/new_features/v172_features_doc.rst).
+
+- **`consensus_point` / `consensus_element` / `is_confident`** (`AC_consensus_point`, `AC_consensus_element`): a target can be grounded several ways at once (set-of-marks / OCR / template / a11y / N model samples) and they don't always agree. `ab_locator`/`element_scoring` rank *strategies* by history; `snap_to_element` snaps a *single* coordinate — neither fuses *simultaneous* proposals. This clusters candidate points (or votes candidate elements), returns the agreed `point` + an `agreement` fraction + `spread`, and `is_confident` flags low-agreement targets so the agent zooms / asks instead of clicking blind. Pure-stdlib; no `PySide6`.
+
 ## What's new (2026-06-24) — Sub-Pixel Template-Match Refinement
 
 Refine a match's centre to a fraction of a pixel for drag / slider / high-DPI precision. Full reference: [`docs/source/Eng/doc/new_features/v171_features_doc.rst`](docs/source/Eng/doc/new_features/v171_features_doc.rst).

docs/source/Eng/doc/new_features/v172_features_doc.rst

@@ -0,0 +1,44 @@
+Grounding Self-Consistency (Consensus Over Proposals)
+=====================================================
+
+A target can be grounded several ways at once — set-of-marks, OCR, template match, the a11y
+tree, or N samples from a model — and they don't always agree. ``ab_locator`` /
+``element_scoring`` rank locator *strategies* by historical reliability but never fuse
+*simultaneous* proposals into one consensus point with a dispersion metric, and
+``action_grounding.snap_to_element`` snaps a *single* coordinate to the nearest element with no
+notion of disagreement. ``grounding_consensus`` clusters the candidate points (or votes
+candidate elements), returns the agreed target plus an *agreement* fraction and *spread*, and
+flags low-agreement targets so the caller can zoom / ask a human instead of clicking blind.
+
+Pure-stdlib geometry; deterministic and unit-testable with no device. Imports no ``PySide6``.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import consensus_point, consensus_element, is_confident
+
+    proposals = [[480, 260], [484, 258], [477, 263], [900, 100]]  # one outlier
+    result = consensus_point(proposals, cluster_radius=24)
+    if is_confident(result, min_agreement=0.6):
+        click(*result.point)
+    else:
+        print("proposals disagree:", result.agreement, result.spread)
+
+    # or vote proposals onto a known element list
+    element, agreement = consensus_element(proposals, elements)
+
+``consensus_point`` clusters ``[x, y]`` or ``[x, y, weight]`` candidates and returns a
+``ConsensusResult`` (``point`` / ``agreement`` — winning cluster weight over total / ``spread``
+/ ``n_clusters``), or ``None`` when empty. ``consensus_element`` votes each proposal to its
+nearest element and returns ``(element, agreement)``. ``is_confident`` thresholds the agreement.
+
+Executor commands
+-----------------
+
+``AC_consensus_point`` (``candidates`` / ``cluster_radius`` → ``{found, result}``) and
+``AC_consensus_element`` (``candidates`` / ``elements`` → ``{found, element, agreement}``). They
+are exposed as the MCP tools ``ac_consensus_point`` / ``ac_consensus_element`` (read-only) and
+as the Script Builder commands **Grounding Consensus Point** / **Grounding Consensus Element**
+under **Native UI**.

docs/source/Eng/eng_index.rst

@@ -194,6 +194,7 @@ Comprehensive guides for all AutoControl features.
    doc/new_features/v169_features_doc
    doc/new_features/v170_features_doc
    doc/new_features/v171_features_doc
+   doc/new_features/v172_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

docs/source/Zh/doc/new_features/v172_features_doc.rst

@@ -0,0 +1,42 @@
+Grounding 自我一致性(提案共識)
+================================
+
+一個目標可同時以多種方式 grounding——set-of-marks、OCR、樣板比對、a11y 樹,或模型的 N 次
+抽樣——而它們未必一致。``ab_locator`` / ``element_scoring`` 依歷史可靠度排序定位器*策略*,
+卻從不把*同時*的提案融合成單一共識點並附離散度指標;``action_grounding.snap_to_element``
+把*單一*座標貼到最近元素,沒有「不一致」的概念。``grounding_consensus`` 將候選點聚類(或對
+候選元素投票),回傳一致同意的目標加上 *agreement* 比例與 *spread*,並標記低一致度目標,
+讓呼叫端可改為放大 / 詢問人工,而非盲目點擊。
+
+純標準函式庫幾何;確定性、可在無裝置下單元測試。不匯入 ``PySide6``。
+
+無頭 API
+--------
+
+.. code-block:: python
+
+    from je_auto_control import consensus_point, consensus_element, is_confident
+
+    proposals = [[480, 260], [484, 258], [477, 263], [900, 100]]  # 一個離群
+    result = consensus_point(proposals, cluster_radius=24)
+    if is_confident(result, min_agreement=0.6):
+        click(*result.point)
+    else:
+        print("提案不一致:", result.agreement, result.spread)
+
+    # 或把提案投票到已知元素清單
+    element, agreement = consensus_element(proposals, elements)
+
+``consensus_point`` 將 ``[x, y]`` 或 ``[x, y, weight]`` 候選聚類,回傳 ``ConsensusResult``
+(``point`` / ``agreement``——勝出群權重佔總權重 / ``spread`` / ``n_clusters``),空輸入回傳
+``None``。``consensus_element`` 把每個提案投到最近元素,回傳 ``(element, agreement)``。
+``is_confident`` 對 agreement 設門檻。
+
+執行器指令
+----------
+
+``AC_consensus_point``(``candidates`` / ``cluster_radius`` → ``{found, result}``)與
+``AC_consensus_element``(``candidates`` / ``elements`` → ``{found, element, agreement}``)。
+兩者以 MCP 工具 ``ac_consensus_point`` / ``ac_consensus_element``(唯讀)及 Script Builder 指令
+**Grounding Consensus Point** / **Grounding Consensus Element**(位於 **Native UI** 分類下)
+形式提供。

docs/source/Zh/zh_index.rst

@@ -194,6 +194,7 @@ AutoControl 所有功能的完整使用指南。
    doc/new_features/v169_features_doc
    doc/new_features/v170_features_doc
    doc/new_features/v171_features_doc
+   doc/new_features/v172_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

je_auto_control/__init__.py

@@ -331,6 +331,10 @@
 from je_auto_control.utils.step_repair import (
     RepairOutcome, RepairPolicy, next_tactic, plan_repair, run_with_repair,
 )
+# Self-consistency over multiple grounding proposals for one target
+from je_auto_control.utils.grounding_consensus import (
+    ConsensusResult, consensus_element, consensus_point, is_confident,
+)
 # Locate on-screen regions by colour (mask + connected components)
 from je_auto_control.utils.color_region import (
     find_color_region, find_color_regions,
@@ -1283,6 +1287,10 @@ def start_autocontrol_gui(*args, **kwargs):
     "plan_repair",
     "next_tactic",
     "run_with_repair",
+    "ConsensusResult",
+    "consensus_point",
+    "consensus_element",
+    "is_confident",
     "find_color_region",
     "find_color_regions",
     "ssim_compare",

je_auto_control/gui/script_builder/command_schema.py

@@ -3195,7 +3195,7 @@
         "AC_effect_near_point", "Native UI", "Effect Near Point?",
         fields=(
             FieldSpec("before", FieldType.STRING,
                       placeholder='[{"role":"button","x":0,"y":0}]'),
             FieldSpec("after", FieldType.STRING,
                       placeholder='[{"role":"button","x":0,"y":0}]'),
             FieldSpec("point", FieldType.STRING, placeholder="[50, 50]"),
@@ -3225,6 +3225,25 @@
         ),
         description="Ordered repair tactics for a failed/no-effect action verdict.",
     ))
+    specs.append(CommandSpec(
+        "AC_consensus_point", "Native UI", "Grounding Consensus Point",
+        fields=(
+            FieldSpec("candidates", FieldType.STRING,
+                      placeholder="[[100, 100], [104, 98], [97, 103]]"),
+            FieldSpec("cluster_radius", FieldType.INT, optional=True, default=24),
+        ),
+        description="Agreed target point from clustered grounding proposals.",
+    ))
+    specs.append(CommandSpec(
+        "AC_consensus_element", "Native UI", "Grounding Consensus Element",
+        fields=(
+            FieldSpec("candidates", FieldType.STRING,
+                      placeholder="[[8, 8], [12, 10]]"),
+            FieldSpec("elements", FieldType.STRING,
+                      placeholder='[{"role":"button","x":0,"y":0}]'),
+        ),
+        description="Vote grounding proposals to the nearest element.",
+    ))
     specs.append(CommandSpec(
         "AC_validate_action", "Native UI", "Validate / Snap Action",
         fields=(

je_auto_control/utils/executor/action_executor.py

@@ -4168,6 +4168,31 @@ def _plan_repair(verdict: Any, max_attempts: Any = 3) -> Dict[str, Any]:
     return {"count": len(tactics), "tactics": tactics}
 
 
+def _consensus_point(candidates: Any, cluster_radius: Any = 24) -> Dict[str, Any]:
+    """Adapter: agreed target point from clustered grounding proposals."""
+    import json
+    from je_auto_control.utils.grounding_consensus import consensus_point
+    if isinstance(candidates, str):
+        candidates = json.loads(candidates)
+    result = consensus_point(candidates, cluster_radius=float(cluster_radius))
+    return {"found": result is not None,
+            "result": result.to_dict() if result else None}
+
+
+def _consensus_element(candidates: Any, elements: Any) -> Dict[str, Any]:
+    """Adapter: vote grounding proposals to the nearest element."""
+    import json
+    from je_auto_control.utils.grounding_consensus import consensus_element
+    if isinstance(candidates, str):
+        candidates = json.loads(candidates)
+    if isinstance(elements, str):
+        elements = json.loads(elements)
+    winner = consensus_element(candidates, elements)
+    return {"found": winner is not None,
+            "element": winner[0] if winner else None,
+            "agreement": winner[1] if winner else 0.0}
+
+
 def _validate_action(action: Any, screen: Any = None,
                      targets: Any = None) -> Dict[str, Any]:
     """Adapter: validate a coordinate action (bounds + optional snap-to-target)."""
@@ -6051,6 +6076,8 @@ def __init__(self):
             "AC_effect_near_point": _effect_near_point,
             "AC_check_postcondition": _check_postcondition,
             "AC_plan_repair": _plan_repair,
+            "AC_consensus_point": _consensus_point,
+            "AC_consensus_element": _consensus_element,
             "AC_validate_action": _validate_action,
             "AC_replay_trace": _replay_trace,
             "AC_match_elements": _match_elements,

je_auto_control/utils/grounding_consensus/__init__.py

@@ -0,0 +1,8 @@
+"""Self-consistency over multiple grounding proposals for one target."""
+from je_auto_control.utils.grounding_consensus.grounding_consensus import (
+    ConsensusResult, consensus_element, consensus_point, is_confident,
+)
+
+__all__ = [
+    "ConsensusResult", "consensus_point", "consensus_element", "is_confident",
+]

je_auto_control/utils/grounding_consensus/grounding_consensus.py

@@ -0,0 +1,119 @@
+"""Self-consistency over multiple grounding proposals for one target.
+
+A target can be grounded several ways at once — set-of-marks, OCR, template match, the a11y
+tree, or N samples from a model — and they don't always agree. ``ab_locator`` /
+``element_scoring`` rank locator *strategies* by historical reliability but never fuse
+*simultaneous* proposals into one consensus point with a dispersion metric, and
+``action_grounding.snap_to_element`` snaps a *single* coordinate to the nearest element with no
+notion of disagreement. ``grounding_consensus`` clusters the candidate points (or votes
+candidate elements), returns the agreed target plus an *agreement* fraction and *spread*, and
+flags low-agreement targets so the caller can zoom / ask a human instead of clicking blind.
+
+Pure-stdlib geometry; deterministic and unit-testable with no device. Imports no ``PySide6``.
+"""
+from dataclasses import asdict, dataclass
+from typing import Any, Dict, List, Optional, Sequence, Tuple
+
+Candidate = Any
+Element = Dict[str, Any]
+
+
+@dataclass(frozen=True)
+class ConsensusResult:
+    """The agreed target point plus its agreement fraction, spread and cluster count."""
+
+    point: List[int]
+    agreement: float
+    spread: float
+    n_clusters: int
+
+    def to_dict(self) -> Dict[str, Any]:
+        """Return the result as a plain dict."""
+        return asdict(self)
+
+
+def _xyw(candidate: Candidate) -> Tuple[float, float, float]:
+    """Normalise a candidate to ``(x, y, weight)`` from a dict or ``[x, y[, w]]``."""
+    if isinstance(candidate, dict):
+        return (float(candidate.get("x", 0)), float(candidate.get("y", 0)),
+                float(candidate.get("weight", 1.0)))
+    seq = list(candidate)
+    return float(seq[0]), float(seq[1]), float(seq[2]) if len(seq) > 2 else 1.0
+
+
+def _assign(point: Tuple[float, float, float], clusters: List[Dict[str, Any]],
+            radius: float) -> bool:
+    """Add ``point`` to the first cluster whose centroid is within ``radius``."""
+    x, y, weight = point
+    for cluster in clusters:
+        cx, cy = cluster["sx"] / cluster["w"], cluster["sy"] / cluster["w"]
+        if abs(x - cx) <= radius and abs(y - cy) <= radius:
+            cluster["sx"] += x * weight
+            cluster["sy"] += y * weight
+            cluster["w"] += weight
+            cluster["members"].append((x, y))
+            return True
+    return False
+
+
+def consensus_point(candidates: Sequence[Candidate], *,
+                    cluster_radius: float = 24) -> Optional[ConsensusResult]:
+    """Cluster candidate points and return the agreed target, or ``None`` if empty.
+
+    ``agreement`` is the winning cluster's weight over the total; ``spread`` is the
+    largest member distance from its centroid; ``n_clusters`` is how many groups formed.
+    """
+    points = [_xyw(c) for c in candidates]
+    if not points:
+        return None
+    total = sum(weight for _, _, weight in points)
+    clusters: List[Dict[str, Any]] = []
+    for point in points:
+        if not _assign(point, clusters, float(cluster_radius)):
+            x, y, weight = point
+            clusters.append({"sx": x * weight, "sy": y * weight, "w": weight,
+                             "members": [(x, y)]})
+    best = max(clusters, key=lambda c: c["w"])
+    cx, cy = best["sx"] / best["w"], best["sy"] / best["w"]
+    spread = max(abs(mx - cx) + abs(my - cy) for mx, my in best["members"])
+    return ConsensusResult([int(round(cx)), int(round(cy))],
+                           round(best["w"] / total, 4), round(spread, 2),
+                           len(clusters))
+
+
+def _center(element: Element) -> Tuple[float, float]:
+    return (float(element.get("x", 0)) + float(element.get("width", 0)) / 2.0,
+            float(element.get("y", 0)) + float(element.get("height", 0)) / 2.0)
+
+
+def _nearest_index(x: float, y: float, elements: Sequence[Element]) -> int:
+    """Index of the element whose centre is closest (Manhattan) to ``(x, y)``."""
+    best_index, best_distance = 0, None
+    for index, element in enumerate(elements):
+        cx, cy = _center(element)
+        distance = abs(cx - x) + abs(cy - y)
+        if best_distance is None or distance < best_distance:
+            best_index, best_distance = index, distance
+    return best_index
+
+
+def consensus_element(candidates: Sequence[Candidate],
+                      elements: Sequence[Element]
+                      ) -> Optional[Tuple[Element, float]]:
+    """Vote each candidate point to its nearest element; return ``(winner, agreement)``."""
+    if not elements or not candidates:
+        return None
+    votes = [0.0] * len(elements)
+    total = 0.0
+    for candidate in candidates:
+        x, y, weight = _xyw(candidate)
+        votes[_nearest_index(x, y, elements)] += weight
+        total += weight
+    best = max(range(len(votes)), key=votes.__getitem__)
+    return elements[best], round(votes[best] / total, 4)
+
+
+def is_confident(result: Optional[ConsensusResult], *,
+                 min_agreement: float = 0.6) -> bool:
+    """Return whether a consensus result clears the ``min_agreement`` threshold."""
+    return result is not None and result.agreement >= float(min_agreement)

je_auto_control/utils/mcp_server/tools/_factories.py

@@ -3409,6 +3409,36 @@ def observation_tools() -> List[MCPTool]:
             handler=h.plan_repair,
             annotations=READ_ONLY,
         ),
+        MCPTool(
+            name="ac_consensus_point",
+            description=("Fuse several grounding proposals into one agreed point: "
+                         "cluster 'candidates' ([[x,y] or [x,y,weight]]) and return "
+                         "{found, result:{point,agreement,spread,n_clusters}}. Low "
+                         "'agreement' = the proposals disagree — don't click blind. "
+                         "'cluster_radius'."),
+            input_schema=schema({
+                "candidates": {"type": "array",
+                               "items": {"type": "array",
+                                         "items": {"type": "number"}}},
+                "cluster_radius": {"type": "number"}},
+                required=["candidates"]),
+            handler=h.consensus_point,
+            annotations=READ_ONLY,
+        ),
+        MCPTool(
+            name="ac_consensus_element",
+            description=("Vote grounding 'candidates' ([[x,y]…]) to the nearest of "
+                         "'elements' (role/name/x/y/width/height dicts) and return "
+                         "{found, element, agreement}."),
+            input_schema=schema({
+                "candidates": {"type": "array",
+                               "items": {"type": "array",
+                                         "items": {"type": "number"}}},
+                "elements": {"type": "array", "items": {"type": "object"}}},
+                required=["candidates", "elements"]),
+            handler=h.consensus_element,
+            annotations=READ_ONLY,
+        ),
     ]