Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions README/WHATS_NEW_zh-CN.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
# 本次更新 — AutoControl

## 本次更新 (2026-06-24) — 变化量序列的稳定检测

判断 UI 何时安定下来——以纯粹、可测试的函数作用于变化序列。完整参考:[`docs/source/Zh/doc/new_features/v175_features_doc.rst`](../docs/source/Zh/doc/new_features/v175_features_doc.rst)。

- **`settle_point` / `is_settled` / `SettleTracker`**(`AC_settle_point`):`smart_waits.wait_until_screen_stable` 把稳定逻辑包在 `time.sleep` 循环内、作用于实时帧——你无法喂记录好的序列,也无法单元测试该决策。本功能把它抽离:给定一串*变化量*(像素差 / 元素数差 / 0-1 digest 是否变),在变化量连续 `quiet_samples` 次维持 ≤ `max_churn` 时报告稳定(尖峰重置 run)。`settle_point` 返回稳定索引,`SettleTracker` 为供实时循环的增量形式。纯标准库,不需时钟、不需捕获;不导入 `PySide6`。

## 本次更新 (2026-06-24) — OCR 行的段落与列表分组

把 OCR 行分组成段落,并检测项目符号 / 编号列表。完整参考:[`docs/source/Zh/doc/new_features/v174_features_doc.rst`](../docs/source/Zh/doc/new_features/v174_features_doc.rst)。
Expand Down
6 changes: 6 additions & 0 deletions README/WHATS_NEW_zh-TW.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
# 本次更新 — AutoControl

## 本次更新 (2026-06-24) — 變化量序列的穩定偵測

判斷 UI 何時安定下來——以純粹、可測試的函式作用於變化序列。完整參考:[`docs/source/Zh/doc/new_features/v175_features_doc.rst`](../docs/source/Zh/doc/new_features/v175_features_doc.rst)。

- **`settle_point` / `is_settled` / `SettleTracker`**(`AC_settle_point`):`smart_waits.wait_until_screen_stable` 把穩定邏輯包在 `time.sleep` 迴圈內、作用於即時幀——你無法餵記錄好的序列,也無法單元測試該決策。本功能把它抽離:給定一串*變化量*(像素差 / 元素數差 / 0-1 digest 是否變),在變化量連續 `quiet_samples` 次維持 ≤ `max_churn` 時回報穩定(尖峰重置 run)。`settle_point` 回傳穩定索引,`SettleTracker` 為供即時迴圈的增量形式。純標準函式庫,不需時鐘、不需擷取;不匯入 `PySide6`。

## 本次更新 (2026-06-24) — OCR 行的段落與清單分組

把 OCR 行分組成段落,並偵測項目符號 / 編號清單。完整參考:[`docs/source/Zh/doc/new_features/v174_features_doc.rst`](../docs/source/Zh/doc/new_features/v174_features_doc.rst)。
Expand Down
6 changes: 6 additions & 0 deletions WHATS_NEW.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
# What's New — AutoControl

## What's new (2026-06-24) — Settle Detection Over a Churn Series

Decide when the UI has gone quiet — as a pure, testable function over a change series. Full reference: [`docs/source/Eng/doc/new_features/v175_features_doc.rst`](docs/source/Eng/doc/new_features/v175_features_doc.rst).

- **`settle_point` / `is_settled` / `SettleTracker`** (`AC_settle_point`): `smart_waits.wait_until_screen_stable` bakes the settle logic inside a `time.sleep` loop over live frames — you can't feed it a recorded series or unit-test the decision. This extracts it: given a stream of *churn* values (pixel delta / element-count delta / 0-1 digest-changed), it reports when churn stayed ≤ `max_churn` for `quiet_samples` in a row (a spike resets the run). `settle_point` returns the settle index, `SettleTracker` is the incremental form for a live loop. Pure-stdlib, no clock, no capture; no `PySide6`.

## What's new (2026-06-24) — Paragraph & List Grouping of OCR Lines

Group OCR lines into paragraphs and detect bulleted / numbered lists. Full reference: [`docs/source/Eng/doc/new_features/v174_features_doc.rst`](docs/source/Eng/doc/new_features/v174_features_doc.rst).
Expand Down
43 changes: 43 additions & 0 deletions docs/source/Eng/doc/new_features/v175_features_doc.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
Settle Detection Over a Churn Series
====================================

``smart_waits.wait_until_screen_stable`` and ``actionability``'s stability check bake the
settle logic *inside* a ``time.sleep`` polling loop over live pixel frames — you cannot feed
them a recorded series of a11y-element counts or screen-diff metrics, and you cannot unit-test
the *decision* independently of capture. ``settle_detector`` extracts that decision: it takes a
stream of *churn* values (how much changed each sample — a pixel delta, an element-count delta,
a digest-changed 0/1, anything) and reports when the churn has stayed at or below ``max_churn``
for ``quiet_samples`` in a row. A spike resets the quiet run, so "settled then changed again"
is handled.

Pure-stdlib; deterministic and unit-testable on an injected series with no capture and no
clock. Imports no ``PySide6``.

Headless API
------------

.. code-block:: python

from je_auto_control import settle_point, is_settled, SettleTracker

churns = [5, 4, 0.5, 0.3, 0.2] # per-frame change metric
settle_point(churns, quiet_samples=3, max_churn=1.0) # -> 4
is_settled(churns, quiet_samples=3, max_churn=1.0) # -> True

# incremental, for a live loop (you supply the churn each tick)
tracker = SettleTracker(quiet_samples=3, max_churn=1.0)
state = tracker.update(current_churn)
if state.settled:
observe_now()

``settle_point`` returns the index at which the series first settles (or ``None``);
``is_settled`` is the boolean. ``SettleTracker`` is the incremental form: ``update(churn)``
returns a ``SettleState`` (``settled`` / ``quiet_run`` / ``churn``); ``reset`` clears the run
(e.g. right after acting again).

Executor command
----------------

``AC_settle_point`` (``churns`` / ``quiet_samples`` / ``max_churn`` → ``{settled, index}``) is
exposed as the MCP tool ``ac_settle_point`` (read-only) and as the Script Builder command
**Settle Point (churn series)** under **Flow**.
1 change: 1 addition & 0 deletions docs/source/Eng/eng_index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -197,6 +197,7 @@ Comprehensive guides for all AutoControl features.
doc/new_features/v172_features_doc
doc/new_features/v173_features_doc
doc/new_features/v174_features_doc
doc/new_features/v175_features_doc
doc/ocr_backends/ocr_backends_doc
doc/observability/observability_doc
doc/operations_layer/operations_layer_doc
Expand Down
39 changes: 39 additions & 0 deletions docs/source/Zh/doc/new_features/v175_features_doc.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
變化量序列的穩定偵測
====================

``smart_waits.wait_until_screen_stable`` 與 ``actionability`` 的穩定檢查把穩定邏輯包在
``time.sleep`` 輪詢迴圈內、作用於即時像素幀——你無法餵給它一段記錄好的 a11y 元素數或畫面
差異指標序列,也無法獨立於擷取去單元測試那個*決策*。``settle_detector`` 把該決策抽離:它接收
一串*變化量*(churn,每個樣本變了多少——像素差、元素數差、digest 是否變的 0/1,皆可),並在
變化量連續 ``quiet_samples`` 次維持在 ``max_churn`` 以下時回報穩定。尖峰會重置 quiet run,因此
「穩定後又變動」也能處理。

純標準函式庫;確定性、可在注入序列上單元測試,不需擷取、不需時鐘。不匯入 ``PySide6``。

無頭 API
--------

.. code-block:: python

from je_auto_control import settle_point, is_settled, SettleTracker

churns = [5, 4, 0.5, 0.3, 0.2] # 每幀變化量指標
settle_point(churns, quiet_samples=3, max_churn=1.0) # -> 4
is_settled(churns, quiet_samples=3, max_churn=1.0) # -> True

# 增量版,供即時迴圈(你每 tick 提供 churn)
tracker = SettleTracker(quiet_samples=3, max_churn=1.0)
state = tracker.update(current_churn)
if state.settled:
observe_now()

``settle_point`` 回傳序列首次穩定的索引(或 ``None``);``is_settled`` 為布林。``SettleTracker``
為增量形式:``update(churn)`` 回傳 ``SettleState``(``settled`` / ``quiet_run`` / ``churn``);
``reset`` 清除 run(例如在再次動作後)。

執行器指令
----------

``AC_settle_point``(``churns`` / ``quiet_samples`` / ``max_churn`` → ``{settled, index}``)
以 MCP 工具 ``ac_settle_point``(唯讀)及 Script Builder 指令 **Settle Point (churn series)**
(位於 **Flow** 分類下)形式提供。
1 change: 1 addition & 0 deletions docs/source/Zh/zh_index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -197,6 +197,7 @@ AutoControl 所有功能的完整使用指南。
doc/new_features/v172_features_doc
doc/new_features/v173_features_doc
doc/new_features/v174_features_doc
doc/new_features/v175_features_doc
doc/ocr_backends/ocr_backends_doc
doc/observability/observability_doc
doc/operations_layer/operations_layer_doc
Expand Down
8 changes: 8 additions & 0 deletions je_auto_control/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -343,6 +343,10 @@
from je_auto_control.utils.grounding_consensus import (
ConsensusResult, consensus_element, consensus_point, is_confident,
)
# Decide when a UI has settled, as a pure seam over a churn series
from je_auto_control.utils.settle_detector import (
SettleState, SettleTracker, is_settled, settle_point,
)
# Locate on-screen regions by colour (mask + connected components)
from je_auto_control.utils.color_region import (
find_color_region, find_color_regions,
Expand Down Expand Up @@ -1304,6 +1308,10 @@ def start_autocontrol_gui(*args, **kwargs):
"consensus_point",
"consensus_element",
"is_confident",
"SettleState",
"SettleTracker",
"settle_point",
"is_settled",
"find_color_region",
"find_color_regions",
"ssim_compare",
Expand Down
10 changes: 10 additions & 0 deletions je_auto_control/gui/script_builder/command_schema.py
Original file line number Diff line number Diff line change
Expand Up @@ -3273,6 +3273,16 @@ def _add_set_of_marks_specs(specs: List[CommandSpec]) -> None:
),
description="Agreed target point from clustered grounding proposals.",
))
specs.append(CommandSpec(
"AC_settle_point", "Flow", "Settle Point (churn series)",
fields=(
FieldSpec("churns", FieldType.STRING,
placeholder="[5, 4, 0.5, 0.3, 0.2]"),
FieldSpec("quiet_samples", FieldType.INT, optional=True, default=3),
FieldSpec("max_churn", FieldType.FLOAT, optional=True, default=1.0),
),
description="Index where a churn series first settles (offline settle check).",
))
specs.append(CommandSpec(
"AC_consensus_element", "Native UI", "Grounding Consensus Element",
fields=(
Expand Down
14 changes: 14 additions & 0 deletions je_auto_control/utils/executor/action_executor.py
Original file line number Diff line number Diff line change
Expand Up @@ -4232,6 +4232,19 @@ def _consensus_element(candidates: Any, elements: Any) -> Dict[str, Any]:
"agreement": winner[1] if winner else 0.0}


def _settle_point(churns: Any, quiet_samples: Any = 3,
max_churn: Any = 1.0) -> Dict[str, Any]:
"""Adapter: index at which a churn series first settles (or settled=False)."""
import json
from je_auto_control.utils.settle_detector import settle_point
if isinstance(churns, str):
churns = json.loads(churns)
index = settle_point([float(c) for c in churns],
quiet_samples=int(quiet_samples),
max_churn=float(max_churn))
return {"settled": index is not None, "index": index}


def _validate_action(action: Any, screen: Any = None,
targets: Any = None) -> Dict[str, Any]:
"""Adapter: validate a coordinate action (bounds + optional snap-to-target)."""
Expand Down Expand Up @@ -6121,6 +6134,7 @@ def __init__(self):
"AC_plan_repair": _plan_repair,
"AC_consensus_point": _consensus_point,
"AC_consensus_element": _consensus_element,
"AC_settle_point": _settle_point,
"AC_validate_action": _validate_action,
"AC_replay_trace": _replay_trace,
"AC_match_elements": _match_elements,
Expand Down
15 changes: 15 additions & 0 deletions je_auto_control/utils/mcp_server/tools/_factories.py
Original file line number Diff line number Diff line change
Expand Up @@ -3439,6 +3439,21 @@ def observation_tools() -> List[MCPTool]:
handler=h.consensus_element,
annotations=READ_ONLY,
),
MCPTool(
name="ac_settle_point",
description=("Decide when a UI settled from a 'churns' series (how much "
"changed each sample). Returns {settled, index} — the index "
"where churn first stayed <= 'max_churn' for 'quiet_samples' "
"in a row (a spike resets the run). Feed pixel deltas / "
"element-count deltas / 0-1 digest-changed flags."),
input_schema=schema({
"churns": {"type": "array", "items": {"type": "number"}},
"quiet_samples": {"type": "integer"},
"max_churn": {"type": "number"}},
required=["churns"]),
handler=h.settle_point,
annotations=READ_ONLY,
),
]


Expand Down
5 changes: 5 additions & 0 deletions je_auto_control/utils/mcp_server/tools/_handlers.py
Original file line number Diff line number Diff line change
Expand Up @@ -2478,6 +2478,11 @@ def consensus_element(candidates, elements):
return _consensus_element(candidates, elements)


def settle_point(churns, quiet_samples=3, max_churn=1.0):
from je_auto_control.utils.executor.action_executor import _settle_point
return _settle_point(churns, quiet_samples, max_churn)


def validate_action(action, screen=None, targets=None):
from je_auto_control.utils.executor.action_executor import _validate_action
return _validate_action(action, screen, targets)
Expand Down
6 changes: 6 additions & 0 deletions je_auto_control/utils/settle_detector/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
"""Decide when a UI has settled, as a pure seam over a churn series."""
from je_auto_control.utils.settle_detector.settle_detector import (
SettleState, SettleTracker, is_settled, settle_point,
)

__all__ = ["SettleState", "SettleTracker", "settle_point", "is_settled"]
70 changes: 70 additions & 0 deletions je_auto_control/utils/settle_detector/settle_detector.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
"""Decide when a UI has settled, as a pure seam over a churn series.

``smart_waits.wait_until_screen_stable`` and ``actionability``'s stability check bake the
settle logic *inside* a ``time.sleep`` polling loop over live pixel frames — you cannot feed
them a recorded series of a11y-element counts or screen-diff metrics, and you cannot unit-test
the *decision* independently of capture. ``settle_detector`` extracts that decision: it takes a
stream of *churn* values (how much changed each sample — pixel delta, element-count delta, a
digest-changed 0/1, anything) and reports when the churn has stayed at or below ``max_churn``
for ``quiet_samples`` in a row. A spike resets the quiet run, so "settled then changed again"
is handled.

Pure-stdlib; deterministic and unit-testable on an injected series with no capture, no clock.
Imports no ``PySide6``.
"""
from dataclasses import asdict, dataclass
from typing import Any, Dict, Optional, Sequence


@dataclass(frozen=True)
class SettleState:
"""One settle observation: whether settled, the quiet run length, latest churn."""

settled: bool
quiet_run: int
churn: float

def to_dict(self) -> Dict[str, Any]:
"""Return the state as a plain dict."""
return asdict(self)


class SettleTracker:
"""Incremental settle detector: feed churn values, ask if it has gone quiet."""

def __init__(self, quiet_samples: int = 3, max_churn: float = 1.0) -> None:
"""Settle after ``quiet_samples`` consecutive churns <= ``max_churn``."""
self.quiet_samples = int(quiet_samples)
self.max_churn = float(max_churn)
self.quiet_run = 0

def update(self, churn: float) -> SettleState:
"""Feed the next churn value and return the current settle state."""
churn = float(churn)
if churn <= self.max_churn:
self.quiet_run += 1
else:
self.quiet_run = 0
return SettleState(self.quiet_run >= self.quiet_samples, self.quiet_run,
churn)

def reset(self) -> None:
"""Clear the quiet run (e.g. after acting again)."""
self.quiet_run = 0


def settle_point(churns: Sequence[float], *, quiet_samples: int = 3,
max_churn: float = 1.0) -> Optional[int]:
"""Return the index at which the churn series first becomes settled, or ``None``."""
tracker = SettleTracker(quiet_samples, max_churn)
for index, churn in enumerate(churns):
if tracker.update(churn).settled:
return index
return None


def is_settled(churns: Sequence[float], *, quiet_samples: int = 3,
max_churn: float = 1.0) -> bool:
"""Return whether the churn series settles at any point."""
return settle_point(churns, quiet_samples=quiet_samples,
max_churn=max_churn) is not None
52 changes: 52 additions & 0 deletions test/unit_test/headless/test_settle_detector_batch.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
"""Headless tests for the settle decision over a churn series (pure stdlib)."""
import je_auto_control as ac
from je_auto_control.utils.settle_detector import (
SettleTracker, is_settled, settle_point,
)


def test_settle_point_after_quiet_run():
# 5, 4 are noisy; then three values <= 1.0 → settled at index 4
assert settle_point([5, 4, 0.5, 0.3, 0.2], quiet_samples=3,
max_churn=1.0) == 4


def test_spike_resets_quiet_run():
# quiet, quiet, SPIKE, quiet x3 → settles only at the final index
assert settle_point([0.2, 0.2, 5, 0.1, 0.1, 0.1], quiet_samples=3,
max_churn=1.0) == 5


def test_never_settles_is_none():
assert settle_point([5, 4, 3], quiet_samples=2, max_churn=1.0) is None


def test_is_settled_bool():
assert is_settled([0.1, 0.1], quiet_samples=2, max_churn=1.0) is True
assert is_settled([9, 8], quiet_samples=2, max_churn=1.0) is False


def test_tracker_incremental_and_reset():
tracker = SettleTracker(quiet_samples=2, max_churn=1.0)
assert tracker.update(0.5).settled is False
state = tracker.update(0.4)
assert state.settled is True and state.quiet_run == 2
tracker.reset()
assert tracker.update(0.3).settled is False # run cleared


# --- wiring ---------------------------------------------------------------

def test_wiring():
assert "AC_settle_point" in set(ac.executor.known_commands())
from je_auto_control.utils.mcp_server.tools import build_default_tool_registry
names = {t.name for t in build_default_tool_registry()}
assert "ac_settle_point" in names
from je_auto_control.gui.script_builder.command_schema import _build_specs
specs = {s.command for s in _build_specs()}
assert "AC_settle_point" in specs


def test_facade_exports():
for name in ("settle_point", "is_settled", "SettleTracker", "SettleState"):
assert hasattr(ac, name) and name in ac.__all__
Loading