Merge pull request #395 from Integration-Automation/feat/match-stability-batch

JE-Chen · web-flow · commit fbe54d891f21 · 2026-06-24T07:40:09.000+08:00
Add match_stability: pre-match settle gating + match persistence
diff --git a/README/WHATS_NEW_zh-CN.md b/README/WHATS_NEW_zh-CN.md
@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-24) — 匹配前安定门 + 命中稳定性
+
+避免在动画进行中匹配,并确认命中跨帧维持稳定。完整参考:[`docs/source/Zh/doc/new_features/v180_features_doc.rst`](../docs/source/Zh/doc/new_features/v180_features_doc.rst)。
+
+- **`region_stability` / `match_persistence`**(`AC_region_stability`、`AC_match_persistence`):`smart_waits.wait_until_screen_stable` 以布尔门控实时循环——无法对可注入帧序列评分稳定度,也无法检查某*命中*是否维持。`region_stability` 以相邻帧 SSIM 评分(`{stable, mean_ssim, min_ssim}`);`match_persistence` 确认 template 在*每一*帧都找到且中心于 `agree_px` 内一致(`{persisted, n_hits, jitter}`)。重用 `ssim` + `visual_match` + `grounding_consensus`;帧可注入;不导入 `PySide6`。
+
 ## 本次更新 (2026-06-24) — 色彩感知模板匹配(HSV)
 
 区分形状相同的红色与绿色状态点。完整参考:[`docs/source/Zh/doc/new_features/v179_features_doc.rst`](../docs/source/Zh/doc/new_features/v179_features_doc.rst)。
diff --git a/README/WHATS_NEW_zh-TW.md b/README/WHATS_NEW_zh-TW.md
@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-24) — 比對前安定閘 + 命中穩定性
+
+避免在動畫進行中比對,並確認命中跨幀維持穩定。完整參考:[`docs/source/Zh/doc/new_features/v180_features_doc.rst`](../docs/source/Zh/doc/new_features/v180_features_doc.rst)。
+
+- **`region_stability` / `match_persistence`**(`AC_region_stability`、`AC_match_persistence`):`smart_waits.wait_until_screen_stable` 以布林閘控即時迴圈——無法對可注入幀序列評分穩定度,也無法檢查某*命中*是否維持。`region_stability` 以相鄰幀 SSIM 評分(`{stable, mean_ssim, min_ssim}`);`match_persistence` 確認 template 在*每一*幀都找到且中心於 `agree_px` 內一致(`{persisted, n_hits, jitter}`)。重用 `ssim` + `visual_match` + `grounding_consensus`;幀可注入;不匯入 `PySide6`。
+
 ## 本次更新 (2026-06-24) — 色彩感知樣板比對(HSV)
 
 區分形狀相同的紅色與綠色狀態點。完整參考:[`docs/source/Zh/doc/new_features/v179_features_doc.rst`](../docs/source/Zh/doc/new_features/v179_features_doc.rst)。
diff --git a/WHATS_NEW.md b/WHATS_NEW.md
@@ -1,5 +1,11 @@
 # What's New — AutoControl
 
+## What's new (2026-06-24) — Pre-Match Settle Gating + Match Persistence
+
+Avoid matching mid-animation, and confirm a hit holds steady across frames. Full reference: [`docs/source/Eng/doc/new_features/v180_features_doc.rst`](docs/source/Eng/doc/new_features/v180_features_doc.rst).
+
+- **`region_stability` / `match_persistence`** (`AC_region_stability`, `AC_match_persistence`): `smart_waits.wait_until_screen_stable` gates a live loop with a boolean — it can't score stability on an injectable frame sequence or check whether a *match* held steady. `region_stability` scores consecutive-frame SSIM (`{stable, mean_ssim, min_ssim}`); `match_persistence` confirms a template is found in *every* frame with the centres agreeing within `agree_px` (`{persisted, n_hits, jitter}`). Reuses `ssim` + `visual_match` + `grounding_consensus`; injectable frames; no `PySide6`.
+
 ## What's new (2026-06-24) — Colour-Aware Template Matching (HSV)
 
 Tell a red status dot from a green one of identical shape. Full reference: [`docs/source/Eng/doc/new_features/v179_features_doc.rst`](docs/source/Eng/doc/new_features/v179_features_doc.rst).
diff --git a/docs/source/Eng/doc/new_features/v180_features_doc.rst b/docs/source/Eng/doc/new_features/v180_features_doc.rst
@@ -0,0 +1,43 @@
+Pre-Match Settle Gating + Match Persistence
+===========================================
+
+Matching mid-animation is a top flakiness source: the spinner is still spinning, a transition
+is half-done, and the matcher fires on a transient. ``smart_waits.wait_until_screen_stable``
+gates a *live polling loop* and returns a boolean — it cannot score stability on an *injectable*
+sequence of frames, nor tell you whether a *match* held steady across them. ``match_stability``
+adds both, on injected frames so it is unit-testable: ``region_stability`` scores how settled a
+frame sequence is (consecutive-frame SSIM), and ``match_persistence`` checks that a template's
+hit holds at the same place across the frames (low jitter), not just in one lucky frame.
+
+It reuses ``ssim.ssim_compare``, ``visual_match.match_template`` and
+``grounding_consensus.consensus_point`` — no new matching code. The frames are injectable
+(ndarray / path / PIL). Imports no ``PySide6``.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import region_stability, match_persistence
+
+    frames = [grab(), grab(), grab()]   # a few snapshots of the same region
+    if region_stability(frames, settle_threshold=0.99)["stable"]:
+        do_the_match()
+
+    result = match_persistence("button.png", frames, min_score=0.8, agree_px=8)
+    if result["persisted"]:
+        print("steady hit, jitter", result["jitter"])
+
+``region_stability`` returns ``{stable, mean_ssim, min_ssim}`` — ``stable`` when the lowest
+consecutive-frame SSIM clears ``settle_threshold``. ``match_persistence`` returns
+``{persisted, n_hits, jitter}`` — ``persisted`` when the template is found in *every* frame and
+the hit centres agree within ``agree_px`` (``jitter`` is their spread; 0 = rock steady).
+
+Executor commands
+-----------------
+
+``AC_region_stability`` (``frames`` / ``settle_threshold`` → ``{stable, mean_ssim, min_ssim}``)
+and ``AC_match_persistence`` (``template`` / ``frames`` / ``min_score`` / ``agree_px`` →
+``{persisted, n_hits, jitter}``). They are exposed as the MCP tools ``ac_region_stability`` /
+``ac_match_persistence`` (read-only) and as the Script Builder commands **Region Stability
+(frames)** / **Match Persistence (frames)** under **Image**.
diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst
@@ -202,6 +202,7 @@ Comprehensive guides for all AutoControl features.
    doc/new_features/v177_features_doc
    doc/new_features/v178_features_doc
    doc/new_features/v179_features_doc
+   doc/new_features/v180_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc
diff --git a/docs/source/Zh/doc/new_features/v180_features_doc.rst b/docs/source/Zh/doc/new_features/v180_features_doc.rst
@@ -0,0 +1,41 @@
+比對前安定閘 + 命中穩定性
+==========================
+
+在動畫進行中比對是主要的不穩定來源:轉圈圖示還在轉、轉場做到一半,比對器命中暫態。
+``smart_waits.wait_until_screen_stable`` 閘控*即時輪詢迴圈*並回傳布林——它無法對*可注入*的
+幀序列評分穩定度,也無法告訴你某個*命中*是否跨幀維持。``match_stability`` 補上兩者,且作用於
+注入幀因此可單元測試:``region_stability`` 以相鄰幀 SSIM 評分幀序列的安定度,``match_persistence``
+檢查某 template 的命中是否跨幀維持在同一處(jitter 小),而非只在某一幸運幀。
+
+本功能重用 ``ssim.ssim_compare``、``visual_match.match_template`` 與
+``grounding_consensus.consensus_point``——不新增比對程式。幀可注入(ndarray / 路徑 / PIL)。
+不匯入 ``PySide6``。
+
+無頭 API
+--------
+
+.. code-block:: python
+
+    from je_auto_control import region_stability, match_persistence
+
+    frames = [grab(), grab(), grab()]   # 同一區域的幾張快照
+    if region_stability(frames, settle_threshold=0.99)["stable"]:
+        do_the_match()
+
+    result = match_persistence("button.png", frames, min_score=0.8, agree_px=8)
+    if result["persisted"]:
+        print("穩定命中, jitter", result["jitter"])
+
+``region_stability`` 回傳 ``{stable, mean_ssim, min_ssim}``——當最低相鄰幀 SSIM 超過
+``settle_threshold`` 時 ``stable``。``match_persistence`` 回傳 ``{persisted, n_hits, jitter}``
+——當 template 在*每一*幀都找到且命中中心於 ``agree_px`` 內一致時 ``persisted``(``jitter``
+為其離散度;0 = 穩如磐石)。
+
+執行器指令
+----------
+
+``AC_region_stability``(``frames`` / ``settle_threshold`` → ``{stable, mean_ssim, min_ssim}``)
+與 ``AC_match_persistence``(``template`` / ``frames`` / ``min_score`` / ``agree_px`` →
+``{persisted, n_hits, jitter}``)。兩者以 MCP 工具 ``ac_region_stability`` /
+``ac_match_persistence``(唯讀)及 Script Builder 指令 **Region Stability (frames)** /
+**Match Persistence (frames)**(位於 **Image** 分類下)形式提供。
diff --git a/docs/source/Zh/zh_index.rst b/docs/source/Zh/zh_index.rst
@@ -202,6 +202,7 @@ AutoControl 所有功能的完整使用指南。
    doc/new_features/v177_features_doc
    doc/new_features/v178_features_doc
    doc/new_features/v179_features_doc
+   doc/new_features/v180_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc
diff --git a/je_auto_control/__init__.py b/je_auto_control/__init__.py
@@ -299,6 +299,10 @@
 from je_auto_control.utils.color_match import (
     match_color, match_color_all,
 )
+# Pre-match settle gating + match persistence over a frame sequence
+from je_auto_control.utils.match_stability import (
+    match_persistence, region_stability,
+)
 # Otsu auto-thresholding for template matching (no hand-tuned min_score)
 from je_auto_control.utils.match_autothresh import (
     auto_threshold, match_auto,
@@ -1286,6 +1290,8 @@ def start_autocontrol_gui(*args, **kwargs):
     "vote_centers",
     "match_color",
     "match_color_all",
+    "region_stability",
+    "match_persistence",
     "SubPixelMatch",
     "match_subpixel",
     "refine_peak",
diff --git a/je_auto_control/gui/script_builder/command_schema.py b/je_auto_control/gui/script_builder/command_schema.py
@@ -458,6 +458,28 @@ def _add_image_specs(specs: List[CommandSpec]) -> None:
         ),
         description="Find every colour (HSV) match of a template (NMS-deduped).",
     ))
+    specs.append(CommandSpec(
+        "AC_region_stability", "Image", "Region Stability (frames)",
+        fields=(
+            FieldSpec("frames", FieldType.STRING,
+                      placeholder='["f1.png", "f2.png", "f3.png"]'),
+            FieldSpec("settle_threshold", FieldType.FLOAT, optional=True,
+                      default=0.99, min_value=0.0, max_value=1.0),
+        ),
+        description="Is a frame sequence settled? (consecutive-frame SSIM).",
+    ))
+    specs.append(CommandSpec(
+        "AC_match_persistence", "Image", "Match Persistence (frames)",
+        fields=(
+            FieldSpec("template", FieldType.FILE_PATH),
+            FieldSpec("frames", FieldType.STRING,
+                      placeholder='["f1.png", "f2.png", "f3.png"]'),
+            FieldSpec("min_score", FieldType.FLOAT, optional=True, default=0.8,
+                      min_value=0.0, max_value=1.0),
+            FieldSpec("agree_px", FieldType.INT, optional=True, default=8),
+        ),
+        description="Does a template match hold steady across frames? (jitter).",
+    ))
     specs.append(CommandSpec(
         "AC_grid_cells", "Image", "Grid Cells (coarse grounding)",
         fields=(
diff --git a/je_auto_control/utils/executor/action_executor.py b/je_auto_control/utils/executor/action_executor.py
@@ -3459,6 +3459,26 @@ def _match_color_all(template: str, channels: Any = None, min_score: Any = 0.7,
     return {"count": len(matches), "matches": [m.to_dict() for m in matches]}
 
 
+def _region_stability(frames: Any, settle_threshold: Any = 0.99) -> Dict[str, Any]:
+    """Adapter: how settled an injected frame sequence is (consecutive SSIM)."""
+    import json
+    from je_auto_control.utils.match_stability import region_stability
+    if isinstance(frames, str):
+        frames = json.loads(frames)
+    return region_stability(frames, settle_threshold=float(settle_threshold))
+
+
+def _match_persistence(template: str, frames: Any, min_score: Any = 0.8,
+                       agree_px: Any = 8) -> Dict[str, Any]:
+    """Adapter: whether a template match holds steady across frames."""
+    import json
+    from je_auto_control.utils.match_stability import match_persistence
+    if isinstance(frames, str):
+        frames = json.loads(frames)
+    return match_persistence(template, frames, min_score=float(min_score),
+                             agree_px=float(agree_px))
+
+
 def _region_arg(value: Any) -> Optional[List[int]]:
     """Coerce a JSON-string / list region arg into a list of ints, or None."""
     import json
@@ -6186,6 +6206,8 @@ def __init__(self):
             "AC_vote_centers": _vote_centers,
             "AC_match_color": _match_color,
             "AC_match_color_all": _match_color_all,
+            "AC_region_stability": _region_stability,
+            "AC_match_persistence": _match_persistence,
             "AC_grid_cells": _grid_cells,
             "AC_cell_for_point": _cell_for_point,
             "AC_point_for_cell": _point_for_cell,
diff --git a/je_auto_control/utils/match_stability/__init__.py b/je_auto_control/utils/match_stability/__init__.py
@@ -0,0 +1,6 @@
+"""Pre-match settle gating and match persistence over a sequence of frames."""
+from je_auto_control.utils.match_stability.match_stability import (
+    match_persistence, region_stability,
+)
+
+__all__ = ["region_stability", "match_persistence"]
diff --git a/je_auto_control/utils/match_stability/match_stability.py b/je_auto_control/utils/match_stability/match_stability.py
@@ -0,0 +1,62 @@
+"""Pre-match settle gating and match persistence over a sequence of frames.
+
+Matching mid-animation is a top flakiness source: the spinner is still spinning, a transition
+is half-done, and the matcher fires on a transient. ``smart_waits.wait_until_screen_stable``
+gates a *live polling loop* and returns a boolean — it cannot score stability on an *injectable*
+sequence of frames, nor tell you whether a *match* held steady across them. ``match_stability``
+adds both, on injected frames so it is unit-testable: ``region_stability`` scores how settled a
+frame sequence is (consecutive-frame SSIM), and ``match_persistence`` checks that a template's
+hit holds at the same place across the frames (low jitter), not just in one lucky frame.
+
+Reuses ``ssim.ssim_compare``, ``visual_match.match_template`` and
+``grounding_consensus.consensus_point``; no new matching code. The frames are injectable
+(ndarray / path / PIL); OpenCV + NumPy arrive lazily via those modules. Imports no ``PySide6``.
+"""
+from typing import Any, Dict, List, Optional, Sequence
+
+ImageSource = Any
+
+
+def region_stability(frames: Sequence[ImageSource], *,
+                     settle_threshold: float = 0.99) -> Dict[str, Any]:
+    """Score how settled a frame sequence is via consecutive-frame SSIM.
+
+    Returns ``{stable, mean_ssim, min_ssim}`` — ``stable`` is true when the lowest
+    consecutive-frame SSIM is at least ``settle_threshold`` (nothing moved between frames).
+    A sequence shorter than two frames is trivially stable.
+    """
+    from je_auto_control.utils.ssim import ssim_compare
+    frame_list = list(frames)
+    if len(frame_list) < 2:
+        return {"stable": True, "mean_ssim": 1.0, "min_ssim": 1.0}
+    scores = [ssim_compare(frame_list[i], frame_list[i + 1])
+              for i in range(len(frame_list) - 1)]
+    lowest = min(scores)
+    return {"stable": lowest >= float(settle_threshold),
+            "mean_ssim": round(sum(scores) / len(scores), 4),
+            "min_ssim": round(lowest, 4)}
+
+
+def match_persistence(template: ImageSource, frames: Sequence[ImageSource], *,
+                      min_score: float = 0.8, agree_px: float = 8) -> Dict[str, Any]:
+    """Check that ``template`` matches the same place across ``frames``.
+
+    Returns ``{persisted, n_hits, jitter}`` — ``persisted`` is true when the template is
+    found in *every* frame and the hit centres agree (cluster within ``agree_px``);
+    ``jitter`` is the spread of the agreeing centres (0 = rock steady).
+    """
+    from je_auto_control.utils.grounding_consensus import consensus_point
+    from je_auto_control.utils.visual_match import match_template
+    frame_list = list(frames)
+    centers: List[List[int]] = []
+    for frame in frame_list:
+        match = match_template(template, haystack=frame, min_score=float(min_score))
+        if match is not None:
+            centers.append(match.center)
+    if not centers:
+        return {"persisted": False, "n_hits": 0, "jitter": None}
+    result = consensus_point(centers, cluster_radius=float(agree_px))
+    jitter: Optional[float] = result.spread if result else None
+    persisted = (len(centers) == len(frame_list) and result is not None
+                 and result.agreement >= 0.9)
+    return {"persisted": persisted, "n_hits": len(centers), "jitter": jitter}
diff --git a/je_auto_control/utils/mcp_server/tools/_factories.py b/je_auto_control/utils/mcp_server/tools/_factories.py
@@ -3913,6 +3913,36 @@ def rotated_match_tools() -> List[MCPTool]:
             handler=h.match_color_all,
             annotations=READ_ONLY,
         ),
+        MCPTool(
+            name="ac_region_stability",
+            description=("Score how settled a sequence of 'frames' (image paths) is by "
+                         "consecutive-frame SSIM: {stable, mean_ssim, min_ssim}. "
+                         "stable=true when nothing moved between frames "
+                         "(min_ssim >= 'settle_threshold') - match only when stable to "
+                         "avoid mid-animation hits."),
+            input_schema=schema({
+                "frames": {"type": "array", "items": {"type": "string"}},
+                "settle_threshold": {"type": "number"}},
+                required=["frames"]),
+            handler=h.region_stability,
+            annotations=READ_ONLY,
+        ),
+        MCPTool(
+            name="ac_match_persistence",
+            description=("Check a 'template' matches the same place across 'frames' "
+                         "(image paths): {persisted, n_hits, jitter}. persisted=true "
+                         "when found in every frame and the centres agree within "
+                         "'agree_px' - a steady match, not one lucky frame. "
+                         "'min_score'."),
+            input_schema=schema({
+                "template": {"type": "string"},
+                "frames": {"type": "array", "items": {"type": "string"}},
+                "min_score": {"type": "number"},
+                "agree_px": {"type": "number"}},
+                required=["template", "frames"]),
+            handler=h.match_persistence,
+            annotations=READ_ONLY,
+        ),
     ]
 
 
diff --git a/je_auto_control/utils/mcp_server/tools/_handlers.py b/je_auto_control/utils/mcp_server/tools/_handlers.py
@@ -2164,6 +2164,16 @@ def match_color_all(template, channels=None, min_score=0.7, max_results=20,
                             region)
 
 
+def region_stability(frames, settle_threshold=0.99):
+    from je_auto_control.utils.executor.action_executor import _region_stability
+    return _region_stability(frames, settle_threshold)
+
+
+def match_persistence(template, frames, min_score=0.8, agree_px=8):
+    from je_auto_control.utils.executor.action_executor import _match_persistence
+    return _match_persistence(template, frames, min_score, agree_px)
+
+
 def grid_cells(rows, cols, region=None):
     from je_auto_control.utils.executor.action_executor import _grid_cells
     return _grid_cells(rows, cols, region)
diff --git a/test/unit_test/headless/test_match_stability_batch.py b/test/unit_test/headless/test_match_stability_batch.py
