diff --git a/README/WHATS_NEW_zh-CN.md b/README/WHATS_NEW_zh-CN.md index 51f42407..a22ef05c 100644 --- a/README/WHATS_NEW_zh-CN.md +++ b/README/WHATS_NEW_zh-CN.md @@ -1,5 +1,11 @@ # 本次更新 — AutoControl +## 本次更新 (2026-06-24) — 变化量序列的稳定检测 + +判断 UI 何时安定下来——以纯粹、可测试的函数作用于变化序列。完整参考:[`docs/source/Zh/doc/new_features/v175_features_doc.rst`](../docs/source/Zh/doc/new_features/v175_features_doc.rst)。 + +- **`settle_point` / `is_settled` / `SettleTracker`**(`AC_settle_point`):`smart_waits.wait_until_screen_stable` 把稳定逻辑包在 `time.sleep` 循环内、作用于实时帧——你无法喂记录好的序列,也无法单元测试该决策。本功能把它抽离:给定一串*变化量*(像素差 / 元素数差 / 0-1 digest 是否变),在变化量连续 `quiet_samples` 次维持 ≤ `max_churn` 时报告稳定(尖峰重置 run)。`settle_point` 返回稳定索引,`SettleTracker` 为供实时循环的增量形式。纯标准库,不需时钟、不需捕获;不导入 `PySide6`。 + ## 本次更新 (2026-06-24) — OCR 行的段落与列表分组 把 OCR 行分组成段落,并检测项目符号 / 编号列表。完整参考:[`docs/source/Zh/doc/new_features/v174_features_doc.rst`](../docs/source/Zh/doc/new_features/v174_features_doc.rst)。 diff --git a/README/WHATS_NEW_zh-TW.md b/README/WHATS_NEW_zh-TW.md index 65ca68c9..624a9d32 100644 --- a/README/WHATS_NEW_zh-TW.md +++ b/README/WHATS_NEW_zh-TW.md @@ -1,5 +1,11 @@ # 本次更新 — AutoControl +## 本次更新 (2026-06-24) — 變化量序列的穩定偵測 + +判斷 UI 何時安定下來——以純粹、可測試的函式作用於變化序列。完整參考:[`docs/source/Zh/doc/new_features/v175_features_doc.rst`](../docs/source/Zh/doc/new_features/v175_features_doc.rst)。 + +- **`settle_point` / `is_settled` / `SettleTracker`**(`AC_settle_point`):`smart_waits.wait_until_screen_stable` 把穩定邏輯包在 `time.sleep` 迴圈內、作用於即時幀——你無法餵記錄好的序列,也無法單元測試該決策。本功能把它抽離:給定一串*變化量*(像素差 / 元素數差 / 0-1 digest 是否變),在變化量連續 `quiet_samples` 次維持 ≤ `max_churn` 時回報穩定(尖峰重置 run)。`settle_point` 回傳穩定索引,`SettleTracker` 為供即時迴圈的增量形式。純標準函式庫,不需時鐘、不需擷取;不匯入 `PySide6`。 + ## 本次更新 (2026-06-24) — OCR 行的段落與清單分組 把 OCR 行分組成段落,並偵測項目符號 / 編號清單。完整參考:[`docs/source/Zh/doc/new_features/v174_features_doc.rst`](../docs/source/Zh/doc/new_features/v174_features_doc.rst)。 diff --git a/WHATS_NEW.md b/WHATS_NEW.md index 144438db..7e1cb2d9 100644 --- a/WHATS_NEW.md +++ b/WHATS_NEW.md @@ -1,5 +1,11 @@ # What's New — AutoControl +## What's new (2026-06-24) — Settle Detection Over a Churn Series + +Decide when the UI has gone quiet — as a pure, testable function over a change series. Full reference: [`docs/source/Eng/doc/new_features/v175_features_doc.rst`](docs/source/Eng/doc/new_features/v175_features_doc.rst). + +- **`settle_point` / `is_settled` / `SettleTracker`** (`AC_settle_point`): `smart_waits.wait_until_screen_stable` bakes the settle logic inside a `time.sleep` loop over live frames — you can't feed it a recorded series or unit-test the decision. This extracts it: given a stream of *churn* values (pixel delta / element-count delta / 0-1 digest-changed), it reports when churn stayed ≤ `max_churn` for `quiet_samples` in a row (a spike resets the run). `settle_point` returns the settle index, `SettleTracker` is the incremental form for a live loop. Pure-stdlib, no clock, no capture; no `PySide6`. + ## What's new (2026-06-24) — Paragraph & List Grouping of OCR Lines Group OCR lines into paragraphs and detect bulleted / numbered lists. Full reference: [`docs/source/Eng/doc/new_features/v174_features_doc.rst`](docs/source/Eng/doc/new_features/v174_features_doc.rst). diff --git a/docs/source/Eng/doc/new_features/v175_features_doc.rst b/docs/source/Eng/doc/new_features/v175_features_doc.rst new file mode 100644 index 00000000..ba3faf3a --- /dev/null +++ b/docs/source/Eng/doc/new_features/v175_features_doc.rst @@ -0,0 +1,43 @@ +Settle Detection Over a Churn Series +==================================== + +``smart_waits.wait_until_screen_stable`` and ``actionability``'s stability check bake the +settle logic *inside* a ``time.sleep`` polling loop over live pixel frames — you cannot feed +them a recorded series of a11y-element counts or screen-diff metrics, and you cannot unit-test +the *decision* independently of capture. ``settle_detector`` extracts that decision: it takes a +stream of *churn* values (how much changed each sample — a pixel delta, an element-count delta, +a digest-changed 0/1, anything) and reports when the churn has stayed at or below ``max_churn`` +for ``quiet_samples`` in a row. A spike resets the quiet run, so "settled then changed again" +is handled. + +Pure-stdlib; deterministic and unit-testable on an injected series with no capture and no +clock. Imports no ``PySide6``. + +Headless API +------------ + +.. code-block:: python + + from je_auto_control import settle_point, is_settled, SettleTracker + + churns = [5, 4, 0.5, 0.3, 0.2] # per-frame change metric + settle_point(churns, quiet_samples=3, max_churn=1.0) # -> 4 + is_settled(churns, quiet_samples=3, max_churn=1.0) # -> True + + # incremental, for a live loop (you supply the churn each tick) + tracker = SettleTracker(quiet_samples=3, max_churn=1.0) + state = tracker.update(current_churn) + if state.settled: + observe_now() + +``settle_point`` returns the index at which the series first settles (or ``None``); +``is_settled`` is the boolean. ``SettleTracker`` is the incremental form: ``update(churn)`` +returns a ``SettleState`` (``settled`` / ``quiet_run`` / ``churn``); ``reset`` clears the run +(e.g. right after acting again). + +Executor command +---------------- + +``AC_settle_point`` (``churns`` / ``quiet_samples`` / ``max_churn`` → ``{settled, index}``) is +exposed as the MCP tool ``ac_settle_point`` (read-only) and as the Script Builder command +**Settle Point (churn series)** under **Flow**. diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst index 7d1ed13f..e9906164 100644 --- a/docs/source/Eng/eng_index.rst +++ b/docs/source/Eng/eng_index.rst @@ -197,6 +197,7 @@ Comprehensive guides for all AutoControl features. doc/new_features/v172_features_doc doc/new_features/v173_features_doc doc/new_features/v174_features_doc + doc/new_features/v175_features_doc doc/ocr_backends/ocr_backends_doc doc/observability/observability_doc doc/operations_layer/operations_layer_doc diff --git a/docs/source/Zh/doc/new_features/v175_features_doc.rst b/docs/source/Zh/doc/new_features/v175_features_doc.rst new file mode 100644 index 00000000..346ef90b --- /dev/null +++ b/docs/source/Zh/doc/new_features/v175_features_doc.rst @@ -0,0 +1,39 @@ +變化量序列的穩定偵測 +==================== + +``smart_waits.wait_until_screen_stable`` 與 ``actionability`` 的穩定檢查把穩定邏輯包在 +``time.sleep`` 輪詢迴圈內、作用於即時像素幀——你無法餵給它一段記錄好的 a11y 元素數或畫面 +差異指標序列,也無法獨立於擷取去單元測試那個*決策*。``settle_detector`` 把該決策抽離:它接收 +一串*變化量*(churn,每個樣本變了多少——像素差、元素數差、digest 是否變的 0/1,皆可),並在 +變化量連續 ``quiet_samples`` 次維持在 ``max_churn`` 以下時回報穩定。尖峰會重置 quiet run,因此 +「穩定後又變動」也能處理。 + +純標準函式庫;確定性、可在注入序列上單元測試,不需擷取、不需時鐘。不匯入 ``PySide6``。 + +無頭 API +-------- + +.. code-block:: python + + from je_auto_control import settle_point, is_settled, SettleTracker + + churns = [5, 4, 0.5, 0.3, 0.2] # 每幀變化量指標 + settle_point(churns, quiet_samples=3, max_churn=1.0) # -> 4 + is_settled(churns, quiet_samples=3, max_churn=1.0) # -> True + + # 增量版,供即時迴圈(你每 tick 提供 churn) + tracker = SettleTracker(quiet_samples=3, max_churn=1.0) + state = tracker.update(current_churn) + if state.settled: + observe_now() + +``settle_point`` 回傳序列首次穩定的索引(或 ``None``);``is_settled`` 為布林。``SettleTracker`` +為增量形式:``update(churn)`` 回傳 ``SettleState``(``settled`` / ``quiet_run`` / ``churn``); +``reset`` 清除 run(例如在再次動作後)。 + +執行器指令 +---------- + +``AC_settle_point``(``churns`` / ``quiet_samples`` / ``max_churn`` → ``{settled, index}``) +以 MCP 工具 ``ac_settle_point``(唯讀)及 Script Builder 指令 **Settle Point (churn series)** +(位於 **Flow** 分類下)形式提供。 diff --git a/docs/source/Zh/zh_index.rst b/docs/source/Zh/zh_index.rst index 198b3e6a..7acbd579 100644 --- a/docs/source/Zh/zh_index.rst +++ b/docs/source/Zh/zh_index.rst @@ -197,6 +197,7 @@ AutoControl 所有功能的完整使用指南。 doc/new_features/v172_features_doc doc/new_features/v173_features_doc doc/new_features/v174_features_doc + doc/new_features/v175_features_doc doc/ocr_backends/ocr_backends_doc doc/observability/observability_doc doc/operations_layer/operations_layer_doc diff --git a/je_auto_control/__init__.py b/je_auto_control/__init__.py index 0be599b3..497ea8ac 100644 --- a/je_auto_control/__init__.py +++ b/je_auto_control/__init__.py @@ -343,6 +343,10 @@ from je_auto_control.utils.grounding_consensus import ( ConsensusResult, consensus_element, consensus_point, is_confident, ) +# Decide when a UI has settled, as a pure seam over a churn series +from je_auto_control.utils.settle_detector import ( + SettleState, SettleTracker, is_settled, settle_point, +) # Locate on-screen regions by colour (mask + connected components) from je_auto_control.utils.color_region import ( find_color_region, find_color_regions, @@ -1304,6 +1308,10 @@ def start_autocontrol_gui(*args, **kwargs): "consensus_point", "consensus_element", "is_confident", + "SettleState", + "SettleTracker", + "settle_point", + "is_settled", "find_color_region", "find_color_regions", "ssim_compare", diff --git a/je_auto_control/gui/script_builder/command_schema.py b/je_auto_control/gui/script_builder/command_schema.py index a022c6d5..58d4e45f 100644 --- a/je_auto_control/gui/script_builder/command_schema.py +++ b/je_auto_control/gui/script_builder/command_schema.py @@ -3273,6 +3273,16 @@ def _add_set_of_marks_specs(specs: List[CommandSpec]) -> None: ), description="Agreed target point from clustered grounding proposals.", )) + specs.append(CommandSpec( + "AC_settle_point", "Flow", "Settle Point (churn series)", + fields=( + FieldSpec("churns", FieldType.STRING, + placeholder="[5, 4, 0.5, 0.3, 0.2]"), + FieldSpec("quiet_samples", FieldType.INT, optional=True, default=3), + FieldSpec("max_churn", FieldType.FLOAT, optional=True, default=1.0), + ), + description="Index where a churn series first settles (offline settle check).", + )) specs.append(CommandSpec( "AC_consensus_element", "Native UI", "Grounding Consensus Element", fields=( diff --git a/je_auto_control/utils/executor/action_executor.py b/je_auto_control/utils/executor/action_executor.py index cc4bd1b7..df1b0ccb 100644 --- a/je_auto_control/utils/executor/action_executor.py +++ b/je_auto_control/utils/executor/action_executor.py @@ -4232,6 +4232,19 @@ def _consensus_element(candidates: Any, elements: Any) -> Dict[str, Any]: "agreement": winner[1] if winner else 0.0} +def _settle_point(churns: Any, quiet_samples: Any = 3, + max_churn: Any = 1.0) -> Dict[str, Any]: + """Adapter: index at which a churn series first settles (or settled=False).""" + import json + from je_auto_control.utils.settle_detector import settle_point + if isinstance(churns, str): + churns = json.loads(churns) + index = settle_point([float(c) for c in churns], + quiet_samples=int(quiet_samples), + max_churn=float(max_churn)) + return {"settled": index is not None, "index": index} + + def _validate_action(action: Any, screen: Any = None, targets: Any = None) -> Dict[str, Any]: """Adapter: validate a coordinate action (bounds + optional snap-to-target).""" @@ -6121,6 +6134,7 @@ def __init__(self): "AC_plan_repair": _plan_repair, "AC_consensus_point": _consensus_point, "AC_consensus_element": _consensus_element, + "AC_settle_point": _settle_point, "AC_validate_action": _validate_action, "AC_replay_trace": _replay_trace, "AC_match_elements": _match_elements, diff --git a/je_auto_control/utils/mcp_server/tools/_factories.py b/je_auto_control/utils/mcp_server/tools/_factories.py index 2793876e..784fc086 100644 --- a/je_auto_control/utils/mcp_server/tools/_factories.py +++ b/je_auto_control/utils/mcp_server/tools/_factories.py @@ -3439,6 +3439,21 @@ def observation_tools() -> List[MCPTool]: handler=h.consensus_element, annotations=READ_ONLY, ), + MCPTool( + name="ac_settle_point", + description=("Decide when a UI settled from a 'churns' series (how much " + "changed each sample). Returns {settled, index} — the index " + "where churn first stayed <= 'max_churn' for 'quiet_samples' " + "in a row (a spike resets the run). Feed pixel deltas / " + "element-count deltas / 0-1 digest-changed flags."), + input_schema=schema({ + "churns": {"type": "array", "items": {"type": "number"}}, + "quiet_samples": {"type": "integer"}, + "max_churn": {"type": "number"}}, + required=["churns"]), + handler=h.settle_point, + annotations=READ_ONLY, + ), ] diff --git a/je_auto_control/utils/mcp_server/tools/_handlers.py b/je_auto_control/utils/mcp_server/tools/_handlers.py index 8a6fc2ef..8876b59d 100644 --- a/je_auto_control/utils/mcp_server/tools/_handlers.py +++ b/je_auto_control/utils/mcp_server/tools/_handlers.py @@ -2478,6 +2478,11 @@ def consensus_element(candidates, elements): return _consensus_element(candidates, elements) +def settle_point(churns, quiet_samples=3, max_churn=1.0): + from je_auto_control.utils.executor.action_executor import _settle_point + return _settle_point(churns, quiet_samples, max_churn) + + def validate_action(action, screen=None, targets=None): from je_auto_control.utils.executor.action_executor import _validate_action return _validate_action(action, screen, targets) diff --git a/je_auto_control/utils/settle_detector/__init__.py b/je_auto_control/utils/settle_detector/__init__.py new file mode 100644 index 00000000..d41630a0 --- /dev/null +++ b/je_auto_control/utils/settle_detector/__init__.py @@ -0,0 +1,6 @@ +"""Decide when a UI has settled, as a pure seam over a churn series.""" +from je_auto_control.utils.settle_detector.settle_detector import ( + SettleState, SettleTracker, is_settled, settle_point, +) + +__all__ = ["SettleState", "SettleTracker", "settle_point", "is_settled"] diff --git a/je_auto_control/utils/settle_detector/settle_detector.py b/je_auto_control/utils/settle_detector/settle_detector.py new file mode 100644 index 00000000..d729480b --- /dev/null +++ b/je_auto_control/utils/settle_detector/settle_detector.py @@ -0,0 +1,70 @@ +"""Decide when a UI has settled, as a pure seam over a churn series. + +``smart_waits.wait_until_screen_stable`` and ``actionability``'s stability check bake the +settle logic *inside* a ``time.sleep`` polling loop over live pixel frames — you cannot feed +them a recorded series of a11y-element counts or screen-diff metrics, and you cannot unit-test +the *decision* independently of capture. ``settle_detector`` extracts that decision: it takes a +stream of *churn* values (how much changed each sample — pixel delta, element-count delta, a +digest-changed 0/1, anything) and reports when the churn has stayed at or below ``max_churn`` +for ``quiet_samples`` in a row. A spike resets the quiet run, so "settled then changed again" +is handled. + +Pure-stdlib; deterministic and unit-testable on an injected series with no capture, no clock. +Imports no ``PySide6``. +""" +from dataclasses import asdict, dataclass +from typing import Any, Dict, Optional, Sequence + + +@dataclass(frozen=True) +class SettleState: + """One settle observation: whether settled, the quiet run length, latest churn.""" + + settled: bool + quiet_run: int + churn: float + + def to_dict(self) -> Dict[str, Any]: + """Return the state as a plain dict.""" + return asdict(self) + + +class SettleTracker: + """Incremental settle detector: feed churn values, ask if it has gone quiet.""" + + def __init__(self, quiet_samples: int = 3, max_churn: float = 1.0) -> None: + """Settle after ``quiet_samples`` consecutive churns <= ``max_churn``.""" + self.quiet_samples = int(quiet_samples) + self.max_churn = float(max_churn) + self.quiet_run = 0 + + def update(self, churn: float) -> SettleState: + """Feed the next churn value and return the current settle state.""" + churn = float(churn) + if churn <= self.max_churn: + self.quiet_run += 1 + else: + self.quiet_run = 0 + return SettleState(self.quiet_run >= self.quiet_samples, self.quiet_run, + churn) + + def reset(self) -> None: + """Clear the quiet run (e.g. after acting again).""" + self.quiet_run = 0 + + +def settle_point(churns: Sequence[float], *, quiet_samples: int = 3, + max_churn: float = 1.0) -> Optional[int]: + """Return the index at which the churn series first becomes settled, or ``None``.""" + tracker = SettleTracker(quiet_samples, max_churn) + for index, churn in enumerate(churns): + if tracker.update(churn).settled: + return index + return None + + +def is_settled(churns: Sequence[float], *, quiet_samples: int = 3, + max_churn: float = 1.0) -> bool: + """Return whether the churn series settles at any point.""" + return settle_point(churns, quiet_samples=quiet_samples, + max_churn=max_churn) is not None diff --git a/test/unit_test/headless/test_settle_detector_batch.py b/test/unit_test/headless/test_settle_detector_batch.py new file mode 100644 index 00000000..dbb89ff4 --- /dev/null +++ b/test/unit_test/headless/test_settle_detector_batch.py @@ -0,0 +1,52 @@ +"""Headless tests for the settle decision over a churn series (pure stdlib).""" +import je_auto_control as ac +from je_auto_control.utils.settle_detector import ( + SettleTracker, is_settled, settle_point, +) + + +def test_settle_point_after_quiet_run(): + # 5, 4 are noisy; then three values <= 1.0 → settled at index 4 + assert settle_point([5, 4, 0.5, 0.3, 0.2], quiet_samples=3, + max_churn=1.0) == 4 + + +def test_spike_resets_quiet_run(): + # quiet, quiet, SPIKE, quiet x3 → settles only at the final index + assert settle_point([0.2, 0.2, 5, 0.1, 0.1, 0.1], quiet_samples=3, + max_churn=1.0) == 5 + + +def test_never_settles_is_none(): + assert settle_point([5, 4, 3], quiet_samples=2, max_churn=1.0) is None + + +def test_is_settled_bool(): + assert is_settled([0.1, 0.1], quiet_samples=2, max_churn=1.0) is True + assert is_settled([9, 8], quiet_samples=2, max_churn=1.0) is False + + +def test_tracker_incremental_and_reset(): + tracker = SettleTracker(quiet_samples=2, max_churn=1.0) + assert tracker.update(0.5).settled is False + state = tracker.update(0.4) + assert state.settled is True and state.quiet_run == 2 + tracker.reset() + assert tracker.update(0.3).settled is False # run cleared + + +# --- wiring --------------------------------------------------------------- + +def test_wiring(): + assert "AC_settle_point" in set(ac.executor.known_commands()) + from je_auto_control.utils.mcp_server.tools import build_default_tool_registry + names = {t.name for t in build_default_tool_registry()} + assert "ac_settle_point" in names + from je_auto_control.gui.script_builder.command_schema import _build_specs + specs = {s.command for s in _build_specs()} + assert "AC_settle_point" in specs + + +def test_facade_exports(): + for name in ("settle_point", "is_settled", "SettleTracker", "SettleState"): + assert hasattr(ac, name) and name in ac.__all__