Add token-budgeted observation delta (what changed)

README/WHATS_NEW_zh-CN.md

@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-24) — 令牌预算化的观测增量(变更了什么)
+
+告诉代理*变更了什么*,而非再次整个屏幕。完整参考:[`docs/source/Zh/doc/new_features/v163_features_doc.rst`](../docs/source/Zh/doc/new_features/v163_features_doc.rst)。
+
+- **`delta_observation` / `delta_index` / `summarize_delta`**(`AC_delta_observation`):`serialize_observation` 渲染单一整帧(每回合都撑爆令牌预算);`element_diff` 提供稳定 ID 对应但止于 matched/added/removed 的元素配对。本功能正是缺少的序列化器——比对两帧,将配对元素分类为 changed(role/name/enabled/value/移动)或 stable,只渲染变动部分:`+ [i] role "name"` / `~ [i] … (fields)` / `- …`(added 与 changed 优先、stable 略去、上限 `max_lines`)。重用 `element_diff.match_elements` + `observation.observation_index`。纯标准库;不导入 `PySide6`。
+
 ## 本次更新 (2026-06-24) — 以 OCR 文字填入框线网格(可定址表格)
 
 把有框线表格的线条 + OCR 文字转成可定址的 `R x C` 表格。完整参考:[`docs/source/Zh/doc/new_features/v162_features_doc.rst`](../docs/source/Zh/doc/new_features/v162_features_doc.rst)。

README/WHATS_NEW_zh-TW.md

@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-24) — 權杖預算化的觀測差異(變更了什麼)
+
+告訴代理*變更了什麼*,而非再次整個畫面。完整參考:[`docs/source/Zh/doc/new_features/v163_features_doc.rst`](../docs/source/Zh/doc/new_features/v163_features_doc.rst)。
+
+- **`delta_observation` / `delta_index` / `summarize_delta`**(`AC_delta_observation`):`serialize_observation` 渲染單一整幀(每回合都撐爆權杖預算);`element_diff` 提供穩定 ID 對應但止於 matched/added/removed 的元素配對。本功能正是缺少的序列化器——比對兩幀,將配對元素分類為 changed(role/name/enabled/value/移動)或 stable,只渲染變動部分:`+ [i] role "name"` / `~ [i] … (fields)` / `- …`(added 與 changed 優先、stable 略去、上限 `max_lines`)。重用 `element_diff.match_elements` + `observation.observation_index`。純標準函式庫;不匯入 `PySide6`。
+
 ## 本次更新 (2026-06-24) — 以 OCR 文字填入框線網格(可定址表格)
 
 把有框線表格的線條 + OCR 文字轉成可定址的 `R x C` 表格。完整參考:[`docs/source/Zh/doc/new_features/v162_features_doc.rst`](../docs/source/Zh/doc/new_features/v162_features_doc.rst)。

WHATS_NEW.md

@@ -1,5 +1,11 @@
 # What's New — AutoControl
 
+## What's new (2026-06-24) — Token-Budgeted Observation Delta (What Changed)
+
+Tell an agent *what changed* since the last step, not the whole screen again. Full reference: [`docs/source/Eng/doc/new_features/v163_features_doc.rst`](docs/source/Eng/doc/new_features/v163_features_doc.rst).
+
+- **`delta_observation` / `delta_index` / `summarize_delta`** (`AC_delta_observation`): `serialize_observation` renders one full frame (blows the token budget every turn); `element_diff` gives the stable-ID correspondence but stops at matched/added/removed element pairs. This is the missing serializer — it diffs two frames, classifies matched elements as changed (role/name/enabled/value/moved) or stable, and renders only the churn as `+ [i] role "name"` / `~ [i] … (fields)` / `- …` lines (added & changed first, stable dropped, capped at `max_lines`). Reuses `element_diff.match_elements` + `observation.observation_index`. Pure-stdlib; no `PySide6`.
+
 ## What's new (2026-06-24) — Fill a Ruling-Line Grid With OCR Text (Addressable Tables)
 
 Turn a bordered table's lines + OCR words into an addressable `R x C` table. Full reference: [`docs/source/Eng/doc/new_features/v162_features_doc.rst`](docs/source/Eng/doc/new_features/v162_features_doc.rst).

docs/source/Eng/doc/new_features/v163_features_doc.rst

@@ -0,0 +1,45 @@
+Token-Budgeted Observation Delta (What Changed)
+===============================================
+
+``observation.serialize_observation`` renders *one full frame* of the UI — feeding it to a
+model every turn blows the very token budget that module was built to respect, and forces the
+model to re-read the whole screen just to spot the one new dialog. ``element_diff`` gives the
+stable-ID correspondence between two frames but stops at matched / added / removed *element
+pairs* — it does not render a compact, indexed, budget-capped delta the model can act on.
+
+``observation_delta`` is the missing serializer: it diffs the previous and current observation,
+classifies each matched element as *changed* (role / name / enabled / value / moved) or
+*stable*, and renders only the churn — ``+ [i] role "name"`` (appeared) / ``- role "name"``
+(vanished) / ``~ [i] role "name" (fields)`` (changed) — added and changed first, stable dropped,
+capped at ``max_lines``. The model sees *what changed* instead of the whole screen again.
+
+Pure-stdlib over element dicts; reuses ``element_diff.match_elements`` for the overlap join and
+``observation.observation_index`` for reading-order indexing. Imports no ``PySide6``.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import delta_observation, delta_index, summarize_delta
+
+    summary = delta_observation(prev_elements, curr_elements, max_lines=40)
+    # + [12] dialog "Saved"
+    # ~ [4] button "Submit" (enabled)
+    # - button "Spinner"
+
+    delta = delta_index(prev_elements, curr_elements)   # {added, removed, changed, stable}
+    text = summarize_delta(delta, max_lines=20)
+
+``delta_index`` returns ``{added, removed, changed, stable}`` (``changed`` items are
+``{"after", "fields"}``); ``summarize_delta`` renders a ``delta_index`` result as budget-capped
+``+`` / ``~`` / ``-`` lines; ``delta_observation`` indexes both frames (reading order, viewport
+clip, interactive-only) then diffs and renders in one call.
+
+Executor command
+----------------
+
+``AC_delta_observation`` (``prev`` / ``curr`` / ``viewport`` / ``max_elements`` / ``max_lines``
+/ ``interactive_only`` → ``{summary, added, removed, changed}``) is exposed as the MCP tool
+``ac_delta_observation`` (read-only) and as the Script Builder command **Observation: Delta
+(what changed)** under **Native UI**.

docs/source/Eng/eng_index.rst

@@ -185,6 +185,7 @@ Comprehensive guides for all AutoControl features.
    doc/new_features/v160_features_doc
    doc/new_features/v161_features_doc
    doc/new_features/v162_features_doc
+   doc/new_features/v163_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

docs/source/Zh/doc/new_features/v163_features_doc.rst

@@ -0,0 +1,44 @@
+權杖預算化的觀測差異(變更了什麼)
+====================================
+
+``observation.serialize_observation`` 渲染*單一整幀*的 UI——每回合都餵給模型會撐爆該模組
+原本就要節省的權杖預算,並迫使模型為了發現那一個新對話框而重讀整個畫面。``element_diff``
+提供兩幀之間的穩定 ID 對應,但止於 matched / added / removed 的*元素配對*——它不會渲染出
+模型可據以行動的精簡、帶索引、受預算限制的差異。
+
+``observation_delta`` 正是缺少的序列化器:它比對前一幀與當前觀測,將每個配對元素分類為
+*changed*(role / name / enabled / value / 移動)或 *stable*,並只渲染變動部分——
+``+ [i] role "name"``(出現)/ ``- role "name"``(消失)/ ``~ [i] role "name" (fields)``
+(變更)——added 與 changed 優先、stable 略去、上限為 ``max_lines``。模型看到的是*變更了什麼*,
+而非再次整個畫面。
+
+純標準函式庫,作用於元素字典;重用 ``element_diff.match_elements`` 做重疊配對、
+``observation.observation_index`` 做閱讀順序索引。不匯入 ``PySide6``。
+
+無頭 API
+--------
+
+.. code-block:: python
+
+    from je_auto_control import delta_observation, delta_index, summarize_delta
+
+    summary = delta_observation(prev_elements, curr_elements, max_lines=40)
+    # + [12] dialog "Saved"
+    # ~ [4] button "Submit" (enabled)
+    # - button "Spinner"
+
+    delta = delta_index(prev_elements, curr_elements)   # {added, removed, changed, stable}
+    text = summarize_delta(delta, max_lines=20)
+
+``delta_index`` 回傳 ``{added, removed, changed, stable}``(``changed`` 項目為
+``{"after", "fields"}``);``summarize_delta`` 把 ``delta_index`` 結果渲染為受預算限制的
+``+`` / ``~`` / ``-`` 行;``delta_observation`` 將兩幀索引化(閱讀順序、視口裁切、僅互動元素)
+後再比對並渲染,一次完成。
+
+執行器指令
+----------
+
+``AC_delta_observation``(``prev`` / ``curr`` / ``viewport`` / ``max_elements`` / ``max_lines``
+/ ``interactive_only`` → ``{summary, added, removed, changed}``)以 MCP 工具
+``ac_delta_observation``(唯讀)及 Script Builder 指令 **Observation: Delta (what changed)**
+(位於 **Native UI** 分類下)形式提供。

docs/source/Zh/zh_index.rst

@@ -185,6 +185,7 @@ AutoControl 所有功能的完整使用指南。
    doc/new_features/v160_features_doc
    doc/new_features/v161_features_doc
    doc/new_features/v162_features_doc
+   doc/new_features/v163_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

je_auto_control/__init__.py

@@ -295,6 +295,10 @@
 from je_auto_control.utils.table_grid_fill import (
     assign_text_to_grid, populate_table, table_to_csv, table_to_records,
 )
+# Token-budgeted observation delta (what changed between two UI frames)
+from je_auto_control.utils.observation_delta import (
+    delta_index, delta_observation, summarize_delta,
+)
 # Locate on-screen regions by colour (mask + connected components)
 from je_auto_control.utils.color_region import (
     find_color_region, find_color_regions,
@@ -1217,6 +1221,9 @@ def start_autocontrol_gui(*args, **kwargs):
     "populate_table",
     "table_to_records",
     "table_to_csv",
+    "delta_index",
+    "delta_observation",
+    "summarize_delta",
     "find_color_region",
     "find_color_regions",
     "ssim_compare",

je_auto_control/gui/script_builder/command_schema.py

@@ -3047,7 +3047,7 @@
             FieldSpec("elements", FieldType.STRING,
                       placeholder='[{"role":"button","name":"OK","x":..,"y":..}]'),
             FieldSpec("viewport", FieldType.STRING, optional=True,
                       placeholder="[x, y, w, h]"),
             FieldSpec("max_elements", FieldType.INT, optional=True, default=80),
         ),
         description="Indexed text observation of UI elements for a VLM (act by index).",
@@ -3063,6 +3063,20 @@
         ),
         description="Reading-ordered, viewport-clipped, indexed element list.",
     ))
+    specs.append(CommandSpec(
+        "AC_delta_observation", "Native UI", "Observation: Delta (what changed)",
+        fields=(
+            FieldSpec("prev", FieldType.STRING,
+                      placeholder='[{"role":"button","name":"OK","x":..,"y":..}]'),
+            FieldSpec("curr", FieldType.STRING,
+                      placeholder='[{"role":"button","name":"OK","x":..,"y":..}]'),
+            FieldSpec("viewport", FieldType.STRING, optional=True,
+                      placeholder="[x, y, w, h]"),
+            FieldSpec("max_elements", FieldType.INT, optional=True, default=80),
+            FieldSpec("max_lines", FieldType.INT, optional=True, default=40),
+        ),
+        description="Token-budgeted '+/~/-' summary of what changed between frames.",
+    ))
     specs.append(CommandSpec(
         "AC_validate_action", "Native UI", "Validate / Snap Action",
         fields=(

je_auto_control/utils/executor/action_executor.py

@@ -3980,6 +3980,28 @@ def _observation_index(elements: Any, viewport: Any = None,
     return {"count": len(indexed), "elements": indexed}
 
 
+def _delta_observation(prev: Any, curr: Any, viewport: Any = None,
+                       max_elements: Any = 80, max_lines: Any = 40,
+                       interactive_only: Any = True) -> Dict[str, Any]:
+    """Adapter: token-budgeted "what changed" delta between two element frames."""
+    import json
+    from je_auto_control.utils.observation_delta import (delta_index,
+                                                         delta_observation)
+    if isinstance(prev, str):
+        prev = json.loads(prev)
+    if isinstance(curr, str):
+        curr = json.loads(curr)
+    if isinstance(viewport, str):
+        viewport = json.loads(viewport) if viewport.strip() else None
+    text = delta_observation(list(prev), list(curr), viewport=viewport,
+                             max_elements=int(max_elements),
+                             interactive_only=bool(interactive_only),
+                             max_lines=int(max_lines))
+    delta = delta_index(list(prev), list(curr))
+    return {"summary": text, "added": len(delta["added"]),
+            "removed": len(delta["removed"]), "changed": len(delta["changed"])}
+
+
 def _validate_action(action: Any, screen: Any = None,
                      targets: Any = None) -> Dict[str, Any]:
     """Adapter: validate a coordinate action (bounds + optional snap-to-target)."""
@@ -5849,6 +5871,7 @@ def __init__(self):
             "AC_cua_command": _cua_command,
             "AC_serialize_observation": _serialize_observation,
             "AC_observation_index": _observation_index,
+            "AC_delta_observation": _delta_observation,
             "AC_validate_action": _validate_action,
             "AC_replay_trace": _replay_trace,
             "AC_match_elements": _match_elements,

je_auto_control/utils/mcp_server/tools/_factories.py

@@ -3331,6 +3331,25 @@ def observation_tools() -> List[MCPTool]:
             handler=h.observation_index,
             annotations=READ_ONLY,
         ),
+        MCPTool(
+            name="ac_delta_observation",
+            description=("Token-budgeted 'what changed since last step': diff 'prev' "
+                         "vs 'curr' element lists and render ONLY the churn — "
+                         "'+ [i] role \"name\"' (appeared) / '~ [i] …' (changed) / "
+                         "'- …' (vanished), stable omitted, capped at 'max_lines'. "
+                         "Returns {summary, added, removed, changed}. Feed this each "
+                         "turn instead of the whole screen."),
+            input_schema=schema({
+                "prev": {"type": "array", "items": {"type": "object"}},
+                "curr": {"type": "array", "items": {"type": "object"}},
+                "viewport": {"type": "array", "items": {"type": "integer"}},
+                "max_elements": {"type": "integer"},
+                "max_lines": {"type": "integer"},
+                "interactive_only": {"type": "boolean"}},
+                required=["prev", "curr"]),
+            handler=h.delta_observation,
+            annotations=READ_ONLY,
+        ),
     ]
 
 

je_auto_control/utils/mcp_server/tools/_handlers.py

@@ -2371,6 +2371,13 @@ def observation_index(elements, viewport=None, max_elements=80):
     return _observation_index(elements, viewport, max_elements)
 
 
+def delta_observation(prev, curr, viewport=None, max_elements=80, max_lines=40,
+                      interactive_only=True):
+    from je_auto_control.utils.executor.action_executor import _delta_observation
+    return _delta_observation(prev, curr, viewport, max_elements, max_lines,
+                              interactive_only)
+
+
 def validate_action(action, screen=None, targets=None):
     from je_auto_control.utils.executor.action_executor import _validate_action
     return _validate_action(action, screen, targets)

je_auto_control/utils/observation_delta/__init__.py

@@ -0,0 +1,6 @@
+"""Token-budgeted observation delta: what changed between two UI frames."""
+from je_auto_control.utils.observation_delta.observation_delta import (
+    delta_index, delta_observation, summarize_delta,
+)
+
+__all__ = ["delta_index", "delta_observation", "summarize_delta"]

je_auto_control/utils/observation_delta/observation_delta.py

@@ -0,0 +1,97 @@
+"""Token-budgeted "what changed since last step" observation delta.
+
+``observation.serialize_observation`` renders *one full frame* of the UI — feeding it to a
+model every turn blows the very token budget that module was built to respect, and forces the
+model to re-read the whole screen to spot the one new dialog. ``element_diff`` gives the
+stable-ID correspondence between two frames but stops at matched/added/removed *element pairs*
+— it does not render a compact, indexed, budget-capped delta the model can act on.
+
+This is the missing serializer: it diffs the previous and current observation, classifies each
+matched element as *changed* (role / name / enabled / moved) or *stable*, and renders only the
+churn as ``+ [i] role "name"`` (appeared) / ``- role "name"`` (vanished) / ``~ [i] role "name"``
+(changed) lines — added and changed first, stable dropped — capped at ``max_lines``. The model
+sees "what changed" instead of the whole screen again.
+
+Pure-stdlib over element dicts; reuses ``element_diff.match_elements`` for the overlap join and
+``observation.observation_index`` for reading-order indexing. Imports no ``PySide6``.
+"""
+from typing import Any, Dict, List, Optional, Sequence
+
+Element = Dict[str, Any]
+_CHANGE_KEYS = ("role", "name", "enabled", "value")
+
+
+def _center(element: Element) -> List[int]:
+    return [int(element.get("x", 0)) + int(element.get("width", 0)) // 2,
+            int(element.get("y", 0)) + int(element.get("height", 0)) // 2]
+
+
+def _changed_fields(before: Element, after: Element, move_threshold: int) -> List[str]:
+    """Return the attribute names that differ between two matched elements."""
+    fields = [key for key in _CHANGE_KEYS if before.get(key) != after.get(key)]
+    bx, by = _center(before)
+    ax, ay = _center(after)
+    if abs(ax - bx) > move_threshold or abs(ay - by) > move_threshold:
+        fields.append("moved")
+    return fields
+
+
+def delta_index(prev: Sequence[Element], curr: Sequence[Element], *,
+                iou_threshold: float = 0.5,
+                move_threshold: int = 5) -> Dict[str, List[Any]]:
+    """Diff two element lists into ``{added, removed, changed, stable}``.
+
+    ``changed`` items are ``{"after", "fields"}`` (the differing attribute names);
+    ``added`` / ``removed`` / ``stable`` are element lists. Matching is by IoU overlap.
+    """
+    from je_auto_control.utils.element_diff import match_elements
+    result = match_elements(list(prev), list(curr), iou_threshold=float(iou_threshold))
+    changed, stable = [], []
+    for pair in result["matched"]:
+        fields = _changed_fields(pair["before"], pair["after"], int(move_threshold))
+        if fields:
+            changed.append({"after": pair["after"], "fields": fields})
+        else:
+            stable.append(pair["after"])
+    return {"added": result["added"], "removed": result["removed"],
+            "changed": changed, "stable": stable}
+
+
+def _line(prefix: str, element: Element, suffix: str = "") -> str:
+    cx, cy = _center(element)
+    index = element.get("index")
+    marker = f"[{index}] " if index is not None else ""
+    return (f'{prefix} {marker}{element.get("role", "element")} '
+            f'"{element.get("name", "")}" @({cx},{cy}){suffix}')
+
+
+def summarize_delta(delta: Dict[str, List[Any]], *, max_lines: int = 40) -> str:
+    """Render a ``delta_index`` result as budget-capped ``+`` / ``~`` / ``-`` lines.
+
+    Added and changed elements come first (most actionable), then removed; stable
+    elements are omitted. Overflow past ``max_lines`` is summarised as ``… (+N more)``.
+    """
+    lines = [_line("+", element) for element in delta["added"]]
+    lines += [_line("~", item["after"], f' ({"/".join(item["fields"])})')
+              for item in delta["changed"]]
+    lines += [_line("-", element) for element in delta["removed"]]
+    if len(lines) > max_lines:
+        hidden = len(lines) - max_lines
+        lines = lines[:max_lines] + [f"… (+{hidden} more)"]
+    return "\n".join(lines)
+
+
+def delta_observation(prev: Sequence[Element], curr: Sequence[Element], *,
+                      viewport: Optional[Sequence[int]] = None,
+                      max_elements: int = 80, interactive_only: bool = True,
+                      iou_threshold: float = 0.5, move_threshold: int = 5,
+                      max_lines: int = 40) -> str:
+    """Index both frames, diff them and render the budget-capped change summary."""
+    from je_auto_control.utils.observation import observation_index
+    prev_idx = observation_index(prev, viewport=viewport, max_elements=max_elements,
+                                 interactive_only=interactive_only)
+    curr_idx = observation_index(curr, viewport=viewport, max_elements=max_elements,
+                                 interactive_only=interactive_only)
+    delta = delta_index(prev_idx, curr_idx, iou_threshold=iou_threshold,
+                        move_threshold=move_threshold)
+    return summarize_delta(delta, max_lines=max_lines)