Add localized motion / activity detection (absdiff)

JE-Chen · JE-Chen · commit b4d8a90b65d9 · 2026-06-23T10:53:20.000+08:00
diff --git a/README/WHATS_NEW_zh-CN.md b/README/WHATS_NEW_zh-CN.md
@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-23) — 局部动态 / 活动检测
+
+找出两帧之间哪些子区域在动。完整参考:[`docs/source/Zh/doc/new_features/v146_features_doc.rst`](../docs/source/Zh/doc/new_features/v146_features_doc.rst)。
+
+- **`changed_regions` / `has_motion` / `activity_score`**(`AC_changed_regions`、`AC_has_motion`):`wait_until_screen_stable` 是布尔轮询、`ssim_changed_regions` 是结构性(忽略快速动态)、`diff_screenshots` 非活动区块。本功能是便宜的 absdiff 路径——对逐像素差做门槛、膨胀,返回移动区域方框(由大到小)、布尔值,以及移动像素比例。挑选安静区域或定位转圈动画。两个可注入帧 → 无头可测;沿用共用连通元件辅助;执行器中 `after` 默认为即时屏幕截取。
+
 ## 本次更新 (2026-06-23) — 色彩直方图指纹与变化检测
 
 判断画面在光照 / 缩放下是否仍是「同一个」。完整参考:[`docs/source/Zh/doc/new_features/v145_features_doc.rst`](../docs/source/Zh/doc/new_features/v145_features_doc.rst)。
diff --git a/README/WHATS_NEW_zh-TW.md b/README/WHATS_NEW_zh-TW.md
@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-23) — 局部動態 / 活動偵測
+
+找出兩幀之間哪些子區域在動。完整參考:[`docs/source/Zh/doc/new_features/v146_features_doc.rst`](../docs/source/Zh/doc/new_features/v146_features_doc.rst)。
+
+- **`changed_regions` / `has_motion` / `activity_score`**(`AC_changed_regions`、`AC_has_motion`):`wait_until_screen_stable` 是布林輪詢、`ssim_changed_regions` 是結構性(忽略快速動態)、`diff_screenshots` 非活動區塊。本功能是便宜的 absdiff 路徑——對逐像素差做門檻、膨脹,回傳移動區域方框(由大到小)、布林值,以及移動像素比例。挑選安靜區域或定位轉圈動畫。兩個可注入幀 → 無頭可測;沿用共用連通元件輔助;執行器中 `after` 預設為即時螢幕擷取。
+
 ## 本次更新 (2026-06-23) — 色彩直方圖指紋與變化偵測
 
 判斷畫面在光照 / 縮放下是否仍是「同一個」。完整參考:[`docs/source/Zh/doc/new_features/v145_features_doc.rst`](../docs/source/Zh/doc/new_features/v145_features_doc.rst)。
diff --git a/WHATS_NEW.md b/WHATS_NEW.md
@@ -1,5 +1,11 @@
 # What's New — AutoControl
 
+## What's new (2026-06-23) — Localized Motion / Activity Detection
+
+Find which sub-regions are animating between two frames. Full reference: [`docs/source/Eng/doc/new_features/v146_features_doc.rst`](docs/source/Eng/doc/new_features/v146_features_doc.rst).
+
+- **`changed_regions` / `has_motion` / `activity_score`** (`AC_changed_regions`, `AC_has_motion`): `wait_until_screen_stable` is a boolean poll, `ssim_changed_regions` is structural (ignores fast motion), `diff_screenshots` isn't activity blobs. This is the cheap absdiff path — threshold the per-pixel difference, dilate, and return the moved-region boxes (largest first), a boolean, and the fraction of pixels that moved. Pick a quiet area or locate a spinner. Two injectable frames → headless-testable; reuses the shared connected-components helper; `after` defaults to a live screen grab in the executor.
+
 ## What's new (2026-06-23) — Colour-Histogram Fingerprint & Change Detection
 
 Tell whether the view is "the same" despite lighting / scale. Full reference: [`docs/source/Eng/doc/new_features/v145_features_doc.rst`](docs/source/Eng/doc/new_features/v145_features_doc.rst).
diff --git a/docs/source/Eng/doc/new_features/v146_features_doc.rst b/docs/source/Eng/doc/new_features/v146_features_doc.rst
@@ -0,0 +1,45 @@
+Localized Motion / Activity Detection
+=====================================
+
+Three near-neighbours, all distinct: ``wait_until_screen_stable`` returns a *boolean*
+over a live poll loop (not localized boxes on an injectable pair); ``ssim_changed_regions``
+is *structural* (Gaussian-windowed SSIM, illumination-tolerant — it deliberately ignores
+the fast pixel motion you want for "where is the spinner / video / animation");
+``diff_screenshots`` highlights pixel diffs but is not framed as activity blobs with a
+score. ``changed_regions`` / ``has_motion`` / ``activity_score`` are the cheap absdiff
+path: which sub-regions are *moving* between two frames, so a script can pick a quiet
+area or locate a busy spinner.
+
+They operate on two injectable frames (ndarray / path / PIL), so they are headless-
+testable on synthetic arrays, and reuse the shared connected-component helper. OpenCV +
+NumPy come in via ``je_open_cv``. Imports no ``PySide6``.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import changed_regions, has_motion, activity_score
+
+    before = screenshot_to_array()
+    # ... time passes ...
+    for box in changed_regions(before, after):     # boxes that moved, largest first
+        print(box["x"], box["y"], box["width"], box["height"])
+
+    if has_motion(before, after):
+        print("still animating, activity =", activity_score(before, after))
+
+``changed_regions`` thresholds the absolute difference (``threshold``), denoises
+(``blur``), dilates and returns ``{x, y, width, height, area, center}`` blobs of at
+least ``min_area``, largest first. ``has_motion`` is the boolean form; ``activity_score``
+is the fraction (0..1) of pixels that moved. Frames of different sizes raise
+``ValueError``.
+
+Executor commands
+-----------------
+
+``AC_changed_regions`` (``before`` / ``after`` / ``threshold`` / ``min_area`` /
+``blur`` → ``{count, regions}``) and ``AC_has_motion`` (``before`` / ``after`` /
+``threshold`` / ``min_area`` → ``{moved, activity}``); ``after`` defaults to a live
+screen grab. They are exposed as the MCP tools ``ac_changed_regions`` /
+``ac_has_motion`` and as Script Builder commands under **Image**.
diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst
@@ -168,6 +168,7 @@ Comprehensive guides for all AutoControl features.
    doc/new_features/v143_features_doc
    doc/new_features/v144_features_doc
    doc/new_features/v145_features_doc
+   doc/new_features/v146_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc
diff --git a/docs/source/Zh/doc/new_features/v146_features_doc.rst b/docs/source/Zh/doc/new_features/v146_features_doc.rst
@@ -0,0 +1,37 @@
+局部動態 / 活動偵測
+====================
+
+三個相近鄰居,各有區別:``wait_until_screen_stable`` 在即時輪詢迴圈上回傳*布林值*(不是對可注入配對的局部方框);
+``ssim_changed_regions`` 是*結構性*的(高斯視窗 SSIM、耐光照——刻意忽略你想抓的快速像素動態);``diff_screenshots``
+標示像素差但不以活動區塊 + 分數呈現。``changed_regions`` / ``has_motion`` / ``activity_score`` 是便宜的 absdiff
+路徑:兩幀之間哪些子區域在*移動*,讓腳本能挑選安靜區域或定位忙碌的轉圈動畫。
+
+它們在兩個可注入的幀(ndarray / 路徑 / PIL)上運作,因此可對合成陣列做無頭測試,並沿用共用的連通元件輔助函式。
+OpenCV + NumPy 透過 ``je_open_cv`` 引入。不匯入 ``PySide6``。
+
+無頭 API
+--------
+
+.. code-block:: python
+
+    from je_auto_control import changed_regions, has_motion, activity_score
+
+    before = screenshot_to_array()
+    # ... 經過一段時間 ...
+    for box in changed_regions(before, after):     # 移動的方框,由大到小
+        print(box["x"], box["y"], box["width"], box["height"])
+
+    if has_motion(before, after):
+        print("still animating, activity =", activity_score(before, after))
+
+``changed_regions`` 對絕對差做門檻(``threshold``)、去噪(``blur``)、膨脹,回傳至少 ``min_area`` 的
+``{x, y, width, height, area, center}`` 區塊,由大到小。``has_motion`` 是布林形式;``activity_score`` 是移動像素的
+比例(0..1)。不同尺寸的幀會丟出 ``ValueError``。
+
+執行器命令
+----------
+
+``AC_changed_regions``(``before`` / ``after`` / ``threshold`` / ``min_area`` / ``blur`` → ``{count, regions}``)與
+``AC_has_motion``(``before`` / ``after`` / ``threshold`` / ``min_area`` → ``{moved, activity}``);``after`` 預設為
+即時螢幕擷取。它們以 MCP 工具 ``ac_changed_regions`` / ``ac_has_motion`` 以及 Script Builder 中 **Image** 分類下的
+命令提供。
diff --git a/docs/source/Zh/zh_index.rst b/docs/source/Zh/zh_index.rst
@@ -168,6 +168,7 @@ AutoControl 所有功能的完整使用指南。
    doc/new_features/v143_features_doc
    doc/new_features/v144_features_doc
    doc/new_features/v145_features_doc
+   doc/new_features/v146_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc
diff --git a/je_auto_control/__init__.py b/je_auto_control/__init__.py
@@ -343,6 +343,10 @@
 from je_auto_control.utils.img_histogram import (
     compare_histograms, histogram_changed, image_histogram,
 )
+# Localized change / activity detection between two frames (absdiff)
+from je_auto_control.utils.motion_regions import (
+    activity_score, changed_regions, has_motion,
+)
 # CI workflow annotations (GitHub Actions)
 from je_auto_control.utils.ci_annotations import (
     emit_annotations, format_annotation,
@@ -1202,6 +1206,9 @@ def start_autocontrol_gui(*args, **kwargs):
     "image_histogram",
     "compare_histograms",
     "histogram_changed",
+    "changed_regions",
+    "has_motion",
+    "activity_score",
     "emit_annotations", "format_annotation",
     "ClipboardHistory", "default_clipboard_history",
     "analyze_heal_log", "heal_stats", "scan_secrets",
diff --git a/je_auto_control/gui/script_builder/command_schema.py b/je_auto_control/gui/script_builder/command_schema.py
@@ -532,6 +532,27 @@ def _add_image_specs(specs: List[CommandSpec]) -> None:
         ),
         description="Detect a palette/view change vs a reference (illumination-robust).",
     ))
+    specs.append(CommandSpec(
+        "AC_changed_regions", "Image", "Changed Regions (motion)",
+        fields=(
+            FieldSpec("before", FieldType.FILE_PATH),
+            FieldSpec("after", FieldType.FILE_PATH, optional=True),
+            FieldSpec("threshold", FieldType.INT, optional=True, default=25),
+            FieldSpec("min_area", FieldType.INT, optional=True, default=80),
+            FieldSpec("blur", FieldType.INT, optional=True, default=5),
+        ),
+        description="Boxes of regions that moved between two frames (after=screen).",
+    ))
+    specs.append(CommandSpec(
+        "AC_has_motion", "Image", "Has Motion?",
+        fields=(
+            FieldSpec("before", FieldType.FILE_PATH),
+            FieldSpec("after", FieldType.FILE_PATH, optional=True),
+            FieldSpec("threshold", FieldType.INT, optional=True, default=25),
+            FieldSpec("min_area", FieldType.INT, optional=True, default=80),
+        ),
+        description="Whether anything moved between two frames (+ activity score).",
+    ))
 
 
 def _add_ocr_specs(specs: List[CommandSpec]) -> None:
diff --git a/je_auto_control/utils/executor/action_executor.py b/je_auto_control/utils/executor/action_executor.py
@@ -3707,6 +3707,34 @@ def _histogram_changed(reference: str, current: Any = None, method: str =
             "score": compare_histograms(ref_hist, cur_hist, method=str(method))}
 
 
+def _changed_regions(before: str, after: Any = None, threshold: Any = 25,
+                     min_area: Any = 80, blur: Any = 5) -> Dict[str, Any]:
+    """Adapter: boxes of regions that moved between two frames (after=screen)."""
+    from je_auto_control.utils.motion_regions import changed_regions
+    regions = changed_regions(before, _resolve_after(after), threshold=int(threshold),
+                              min_area=int(min_area), blur=int(blur))
+    return {"count": len(regions), "regions": regions}
+
+
+def _has_motion(before: str, after: Any = None, threshold: Any = 25,
+                min_area: Any = 80) -> Dict[str, Any]:
+    """Adapter: whether anything moved between two frames (after=screen)."""
+    from je_auto_control.utils.motion_regions import activity_score, has_motion
+    resolved = _resolve_after(after)
+    return {"moved": has_motion(before, resolved, threshold=int(threshold),
+                                min_area=int(min_area)),
+            "activity": activity_score(before, resolved, threshold=int(threshold))}
+
+
+def _resolve_after(after: Any):
+    """Return the 'after' frame, grabbing the screen when it is not given."""
+    if after is not None:
+        return after
+    import numpy as np
+    from je_auto_control.utils.cv2_utils.screenshot import pil_screenshot
+    return np.asarray(pil_screenshot().convert("RGB"))
+
+
 def _with_modifiers(modifiers: Any, actions: Any) -> Dict[str, Any]:
     """Adapter: run nested actions while modifier keys are held down."""
     import json
@@ -5452,6 +5480,8 @@ def __init__(self):
             "AC_get_clipboard_html": _get_clipboard_html,
             "AC_image_histogram": _image_histogram,
             "AC_histogram_changed": _histogram_changed,
+            "AC_changed_regions": _changed_regions,
+            "AC_has_motion": _has_motion,
             "AC_tile_rect": _tile_rect,
             "AC_grid_rects": _grid_rects,
             "AC_cascade_rects": _cascade_rects,
diff --git a/je_auto_control/utils/mcp_server/tools/_factories.py b/je_auto_control/utils/mcp_server/tools/_factories.py
@@ -3123,6 +3123,41 @@ def img_histogram_tools() -> List[MCPTool]:
     ]
 
 
+def motion_regions_tools() -> List[MCPTool]:
+    return [
+        MCPTool(
+            name="ac_changed_regions",
+            description=("Boxes of regions that MOVED between 'before' (image path) "
+                         "and 'after' (path; default: the live screen) via absdiff. "
+                         "Returns {count, regions}. For spinners / animations / "
+                         "picking a quiet area. 'threshold'/'min_area'/'blur'."),
+            input_schema=schema({
+                "before": {"type": "string"},
+                "after": {"type": "string"},
+                "threshold": {"type": "integer"},
+                "min_area": {"type": "integer"},
+                "blur": {"type": "integer"}},
+                required=["before"]),
+            handler=h.changed_regions,
+            annotations=READ_ONLY,
+        ),
+        MCPTool(
+            name="ac_has_motion",
+            description=("Whether anything moved between 'before' and 'after' "
+                         "(default: screen). Returns {moved, activity} where "
+                         "activity is the fraction of pixels that changed."),
+            input_schema=schema({
+                "before": {"type": "string"},
+                "after": {"type": "string"},
+                "threshold": {"type": "integer"},
+                "min_area": {"type": "integer"}},
+                required=["before"]),
+            handler=h.has_motion,
+            annotations=READ_ONLY,
+        ),
+    ]
+
+
 def ssim_tools() -> List[MCPTool]:
     return [
         MCPTool(
@@ -6629,7 +6664,7 @@ def media_assert_tools() -> List[MCPTool]:
     monitor_layout_tools, actionability_tools, element_parse_tools,
     hsv_segment_tools, text_regions_tools, edge_lines_tools, expect_poll_tools,
     locator_chain_tools, rich_clipboard_tools, img_histogram_tools,
-    plugin_sdk_tools, governance_tools,
+    motion_regions_tools, plugin_sdk_tools, governance_tools,
     credential_lease_tools, egress_tools, approval_testing_tools,
     trajectory_eval_tools, compliance_tools, agent_trace_tools,
     video_report_tools, fuzzy_tools, artifact_store_tools, image_dedup_tools,
diff --git a/je_auto_control/utils/mcp_server/tools/_handlers.py b/je_auto_control/utils/mcp_server/tools/_handlers.py
@@ -2258,6 +2258,16 @@ def histogram_changed(reference, current=None, method="correlation",
     return _histogram_changed(reference, current, method, threshold, space, region)
 
 
+def changed_regions(before, after=None, threshold=25, min_area=80, blur=5):
+    from je_auto_control.utils.executor.action_executor import _changed_regions
+    return _changed_regions(before, after, threshold, min_area, blur)
+
+
+def has_motion(before, after=None, threshold=25, min_area=80):
+    from je_auto_control.utils.executor.action_executor import _has_motion
+    return _has_motion(before, after, threshold, min_area)
+
+
 def detect_drift(reference, current, threshold=0.25, bins=10):
     from je_auto_control.utils.executor.action_executor import _detect_drift
     return _detect_drift(reference, current, threshold, bins)
diff --git a/je_auto_control/utils/motion_regions/__init__.py b/je_auto_control/utils/motion_regions/__init__.py
@@ -0,0 +1,6 @@
+"""Localized change / activity detection between two frames (absdiff)."""
+from je_auto_control.utils.motion_regions.motion_regions import (
+    activity_score, changed_regions, has_motion,
+)
+
+__all__ = ["activity_score", "changed_regions", "has_motion"]
diff --git a/je_auto_control/utils/motion_regions/motion_regions.py b/je_auto_control/utils/motion_regions/motion_regions.py
@@ -0,0 +1,67 @@
+"""Localized change / activity detection between two frames (cheap absdiff).
+
+Three near-neighbours, all distinct: ``wait_until_screen_stable`` returns a *boolean*
+over a live poll loop (not localized boxes on an injectable pair); ``ssim_changed_regions``
+is *structural* (Gaussian-windowed SSIM, illumination-tolerant — deliberately ignores
+the fast pixel motion you'd want for "where is the spinner / video / animation");
+``diff_screenshots`` highlights pixel diffs but is not framed as activity blobs with a
+score. This is the cheap absdiff path: which sub-regions are *moving* right now, so a
+script can pick a quiet region or locate a busy spinner.
+
+Operates on two injectable frames (ndarray / path / PIL), so it is headless-testable on
+synthetic arrays. OpenCV + NumPy come in via ``je_open_cv``; reuses the shared
+connected-component helper. Imports no ``PySide6``.
+"""
+from typing import Any, Dict, List
+
+from je_auto_control.utils.visual_match.visual_match import _to_gray
+
+ImageSource = Any
+
+
+def _diff_mask(before: ImageSource, after: ImageSource, threshold: int, blur: int):
+    """Return the binary motion mask between two frames (same size required)."""
+    import cv2
+    first = _to_gray(before)
+    second = _to_gray(after)
+    if first.shape != second.shape:
+        raise ValueError(f"frames must be the same size: {first.shape} vs "
+                         f"{second.shape}")
+    if int(blur) > 0:
+        kernel = int(blur) | 1
+        first = cv2.GaussianBlur(first, (kernel, kernel), 0)
+        second = cv2.GaussianBlur(second, (kernel, kernel), 0)
+    _retval, mask = cv2.threshold(cv2.absdiff(first, second), int(threshold), 255,
+                                  cv2.THRESH_BINARY)
+    return mask
+
+
+def changed_regions(before: ImageSource, after: ImageSource, *,
+                    threshold: int = 25, min_area: int = 80,
+                    blur: int = 5) -> List[Dict[str, Any]]:
+    """Return boxes of the regions that moved between ``before`` and ``after``.
+
+    A pixel counts as moved where the absolute difference exceeds ``threshold``;
+    connected moved pixels covering at least ``min_area`` are returned as
+    ``{x, y, width, height, area, center}`` largest first. ``blur`` denoises first.
+    """
+    import cv2
+    from je_auto_control.utils.cv2_utils.blobs import connected_boxes
+    mask = _diff_mask(before, after, threshold, blur)
+    kernel = cv2.getStructuringElement(cv2.MORPH_RECT, (3, 3))
+    mask = cv2.dilate(mask, kernel, iterations=2)
+    return connected_boxes(mask, int(min_area))
+
+
+def has_motion(before: ImageSource, after: ImageSource, *, threshold: int = 25,
+               min_area: int = 80) -> bool:
+    """Return whether any region of at least ``min_area`` moved between the frames."""
+    return bool(changed_regions(before, after, threshold=threshold,
+                                min_area=min_area))
+
+
+def activity_score(before: ImageSource, after: ImageSource, *,
+                   threshold: int = 25) -> float:
+    """Return the fraction (0..1) of pixels that moved between the two frames."""
+    mask = _diff_mask(before, after, threshold, 0)
+    return round(float((mask > 0).sum()) / mask.size, 4)
diff --git a/test/unit_test/headless/test_motion_regions_batch.py b/test/unit_test/headless/test_motion_regions_batch.py
