Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@

## Table of Contents

- [What's new (2026-06-20) — Self-Healing Locator Write-Back](#whats-new-2026-06-20--self-healing-locator-write-back)
- [What's new (2026-06-20) — DMN-Style Decision Tables](#whats-new-2026-06-20--dmn-style-decision-tables)
- [What's new (2026-06-20) — Saga / Compensating Rollback](#whats-new-2026-06-20--saga--compensating-rollback)
- [What's new (2026-06-20) — JSONPath Querying](#whats-new-2026-06-20--jsonpath-querying)
Expand Down Expand Up @@ -106,6 +107,12 @@

---

## What's new (2026-06-20) — Self-Healing Locator Write-Back

Persist corrected locators so heals aren't forgotten. Full reference: [`docs/source/Eng/doc/new_features/v54_features_doc.rst`](docs/source/Eng/doc/new_features/v54_features_doc.rst).

- **`RepairStore` / `repair_from_heal`** (`AC_repair_record` / `AC_repair_resolved` / `AC_repair_pending` / `AC_repair_approve`, `ac_*`): runtime self-healing previously **threw away** the corrected location, so every run re-healed. This records the corrected locator (coords/VLM description/method) from a heal, **auto-applies** it when `confidence >= auto_threshold` (default 0.9) or queues a reviewable suggestion, and `resolved(key)` returns the learned fix for reuse. Closes the heal→durable-fix loop; pure-stdlib, fully testable.

## What's new (2026-06-20) — DMN-Style Decision Tables

Externalize branching into reviewable rule tables. Full reference: [`docs/source/Eng/doc/new_features/v53_features_doc.rst`](docs/source/Eng/doc/new_features/v53_features_doc.rst).
Expand Down
7 changes: 7 additions & 0 deletions README/README_zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@

## 目录

- [本次更新 (2026-06-20) — 自我修复定位器回写](#本次更新-2026-06-20--自我修复定位器回写)
- [本次更新 (2026-06-20) — DMN 式决策表](#本次更新-2026-06-20--dmn-式决策表)
- [本次更新 (2026-06-20) — Saga / 补偿回滚](#本次更新-2026-06-20--saga--补偿回滚)
- [本次更新 (2026-06-20) — JSONPath 查询](#本次更新-2026-06-20--jsonpath-查询)
Expand Down Expand Up @@ -105,6 +106,12 @@

---

## 本次更新 (2026-06-20) — 自我修复定位器回写

保存修正定位器,使修复不被遗忘。完整参考:[`docs/source/Zh/doc/new_features/v54_features_doc.rst`](../docs/source/Zh/doc/new_features/v54_features_doc.rst)。

- **`RepairStore` / `repair_from_heal`**(`AC_repair_record` / `AC_repair_resolved` / `AC_repair_pending` / `AC_repair_approve`、`ac_*`):运行期自我修复过去会**丢弃**修正后的位置,因此每次都重新修复。本功能记录该次修复的修正定位器(坐标/VLM 描述/方法),在 `confidence >= auto_threshold`(默认 0.9)时**自动套用**或排入可审查建议,`resolved(key)` 返回已学到的修正供重用。封闭「修复→持久修正」循环;纯标准库、可完整测试。

## 本次更新 (2026-06-20) — DMN 式决策表

将分支外部化为可审查的规则表。完整参考:[`docs/source/Zh/doc/new_features/v53_features_doc.rst`](../docs/source/Zh/doc/new_features/v53_features_doc.rst)。
Expand Down
7 changes: 7 additions & 0 deletions README/README_zh-TW.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@

## 目錄

- [本次更新 (2026-06-20) — 自我修復定位器回寫](#本次更新-2026-06-20--自我修復定位器回寫)
- [本次更新 (2026-06-20) — DMN 式決策表](#本次更新-2026-06-20--dmn-式決策表)
- [本次更新 (2026-06-20) — Saga / 補償回溯](#本次更新-2026-06-20--saga--補償回溯)
- [本次更新 (2026-06-20) — JSONPath 查詢](#本次更新-2026-06-20--jsonpath-查詢)
Expand Down Expand Up @@ -105,6 +106,12 @@

---

## 本次更新 (2026-06-20) — 自我修復定位器回寫

保存修正定位器,使修復不被遺忘。完整參考:[`docs/source/Zh/doc/new_features/v54_features_doc.rst`](../docs/source/Zh/doc/new_features/v54_features_doc.rst)。

- **`RepairStore` / `repair_from_heal`**(`AC_repair_record` / `AC_repair_resolved` / `AC_repair_pending` / `AC_repair_approve`、`ac_*`):執行期自我修復過去會**丟棄**修正後的位置,因此每次都重新修復。本功能記錄該次修復的修正定位器(座標/VLM 描述/方法),在 `confidence >= auto_threshold`(預設 0.9)時**自動套用**或排入可審查建議,`resolved(key)` 回傳已學到的修正供重用。封閉「修復→持久修正」迴圈;純標準函式庫、可完整測試。

## 本次更新 (2026-06-20) — DMN 式決策表

將分支外部化為可審查的規則表。完整參考:[`docs/source/Zh/doc/new_features/v53_features_doc.rst`](../docs/source/Zh/doc/new_features/v53_features_doc.rst)。
Expand Down
55 changes: 55 additions & 0 deletions docs/source/Eng/doc/new_features/v54_features_doc.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
Self-Healing Locator Write-Back
===============================

The self-healing locator finds an element at a new place at runtime and logs the
heal — but the *corrected* location was then thrown away, so the next run healed
from scratch. ``RepairStore`` closes that loop: it records the corrected locator
(coordinates / VLM description / method) from a heal, **auto-applies** it when
confidence is high enough, otherwise queues it as a *pending suggestion* for
review. A later run reads the learned fix via :meth:`RepairStore.resolved`.

JSON-backed (via the shared ``json_store`` helper); pure standard library;
confidence and threshold are explicit, so behavior is deterministic and fully
unit-testable. Imports no ``PySide6``.

Headless API
------------

.. code-block:: python

from je_auto_control import RepairStore, repair_from_heal

store = RepairStore("repairs.json")
# high-confidence heal -> applied immediately:
store.record("login_btn", method="vlm", coordinates=[120, 64],
description="Login", confidence=0.95)
store.resolved("login_btn") # -> {method, coordinates, description}

# low-confidence heal -> queued for review:
s = store.record("save_btn", method="image", coordinates=[5, 5],
confidence=0.5)
store.pending() # -> [the save_btn suggestion]
store.approve(s.id) # now store.resolved("save_btn") works

# straight from a HealEvent (object or dict):
repair_from_heal(heal_event, "login_btn", store=store, confidence=0.9)

``record`` auto-applies when ``confidence >= auto_threshold`` (default 0.9), else
files a ``pending`` suggestion; ``approve`` / ``reject`` decide queued ones;
``resolved(key)`` returns the latest applied/approved corrected locator (or
``None``) — the durable fix a future run reuses instead of re-healing.

Executor commands
-----------------

================================ ===================================================
Command Effect
================================ ===================================================
``AC_repair_record`` Persist a corrected locator (auto-apply or queue).
``AC_repair_resolved`` Get the learned corrected locator for a key.
``AC_repair_pending`` List suggestions awaiting review.
``AC_repair_approve`` Approve a pending suggestion.
================================ ===================================================

The same operations are exposed as MCP tools (``ac_repair_*``) and as Script
Builder commands under **Tools**.
1 change: 1 addition & 0 deletions docs/source/Eng/eng_index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,7 @@ Comprehensive guides for all AutoControl features.
doc/new_features/v51_features_doc
doc/new_features/v52_features_doc
doc/new_features/v53_features_doc
doc/new_features/v54_features_doc
doc/ocr_backends/ocr_backends_doc
doc/observability/observability_doc
doc/operations_layer/operations_layer_doc
Expand Down
51 changes: 51 additions & 0 deletions docs/source/Zh/doc/new_features/v54_features_doc.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@
自我修復定位器回寫
==================

自我修復定位器會在執行期於新位置找到元素並記錄該次修復 —— 但*修正後*的位置隨即被丟棄,
因此下次執行又得從頭修復。``RepairStore`` 補上這個迴圈:它記錄該次修復的修正定位器(座
標 / VLM 描述 / 方法),在信心足夠高時**自動套用**,否則排入*待審建議*。之後的執行可透過
:meth:`RepairStore.resolved` 讀取已學到的修正。

JSON 後端(透過共用 ``json_store`` 助手);純標準函式庫;信心與門檻皆為明確值,因此行為
具確定性且可完整單元測試。不匯入 ``PySide6``。

無頭 API
--------

.. code-block:: python

from je_auto_control import RepairStore, repair_from_heal

store = RepairStore("repairs.json")
# 高信心修復 -> 立即套用:
store.record("login_btn", method="vlm", coordinates=[120, 64],
description="Login", confidence=0.95)
store.resolved("login_btn") # -> {method, coordinates, description}

# 低信心修復 -> 排入審查:
s = store.record("save_btn", method="image", coordinates=[5, 5],
confidence=0.5)
store.pending() # -> [save_btn 建議]
store.approve(s.id) # 此後 store.resolved("save_btn") 可用

# 直接從 HealEvent(物件或 dict):
repair_from_heal(heal_event, "login_btn", store=store, confidence=0.9)

``record`` 在 ``confidence >= auto_threshold``(預設 0.9)時自動套用,否則建立 ``pending``
建議;``approve`` / ``reject`` 決定佇列中的項目;``resolved(key)`` 回傳最新已套用/已核准的
修正定位器(或 ``None``)—— 供未來執行重用而不必重新修復的持久修正。

執行器指令
----------

================================ ===================================================
指令 效果
================================ ===================================================
``AC_repair_record`` 保存修正定位器(自動套用或排入佇列)。
``AC_repair_resolved`` 取得某鍵已學到的修正定位器。
``AC_repair_pending`` 列出待審的建議。
``AC_repair_approve`` 核准一個待審建議。
================================ ===================================================

相同操作亦提供為 MCP 工具(``ac_repair_*``),以及 Script Builder 中 **Tools** 分類下的指
令。
1 change: 1 addition & 0 deletions docs/source/Zh/zh_index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,7 @@ AutoControl 所有功能的完整使用指南。
doc/new_features/v51_features_doc
doc/new_features/v52_features_doc
doc/new_features/v53_features_doc
doc/new_features/v54_features_doc
doc/ocr_backends/ocr_backends_doc
doc/observability/observability_doc
doc/operations_layer/operations_layer_doc
Expand Down
5 changes: 5 additions & 0 deletions je_auto_control/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -286,6 +286,10 @@
from je_auto_control.utils.decision_table import (
DecisionTable, Rule, evaluate_table,
)
# Self-healing write-back: persist corrected locators from heal events
from je_auto_control.utils.locator_repair import (
RepairStore, RepairSuggestion, repair_from_heal,
)
# Background popup/interrupt watchdog (unattended automation)
from je_auto_control.utils.watchdog import (
PopupWatchdog, WatchdogRule, default_popup_watchdog,
Expand Down Expand Up @@ -751,6 +755,7 @@ def start_autocontrol_gui(*args, **kwargs):
"json_extract", "json_query", "json_query_one",
"Saga", "SagaResult", "run_saga",
"DecisionTable", "Rule", "evaluate_table",
"RepairStore", "RepairSuggestion", "repair_from_heal",
# MCP server
"AuditLogger", "HttpMCPServer", "MCPContent", "MCPPrompt",
"MCPPromptArgument", "MCPResource", "MCPServer", "MCPTool",
Expand Down
37 changes: 37 additions & 0 deletions je_auto_control/gui/script_builder/command_schema.py
Original file line number Diff line number Diff line change
Expand Up @@ -1165,6 +1165,43 @@ def _add_misc_specs(specs: List[CommandSpec]) -> None:
),
description="Evaluate inputs against a rule table (hit policy).",
))
specs.append(CommandSpec(
"AC_repair_record", "Tools", "Locator Repair: Record",
fields=(
FieldSpec("key", FieldType.STRING),
FieldSpec("method", FieldType.STRING, placeholder="vlm/image"),
FieldSpec("coordinates", FieldType.STRING, optional=True,
placeholder="[10, 20]"),
FieldSpec("description", FieldType.STRING, optional=True),
FieldSpec("confidence", FieldType.FLOAT, optional=True,
default=1.0),
FieldSpec("auto_threshold", FieldType.FLOAT, optional=True,
default=0.9),
FieldSpec("db", FieldType.STRING, optional=True),
),
description="Persist a corrected locator from a heal (auto/queue).",
))
specs.append(CommandSpec(
"AC_repair_resolved", "Tools", "Locator Repair: Resolved",
fields=(
FieldSpec("key", FieldType.STRING),
FieldSpec("db", FieldType.STRING, optional=True),
),
description="Get the learned corrected locator for a key.",
))
specs.append(CommandSpec(
"AC_repair_pending", "Tools", "Locator Repair: Pending",
fields=(FieldSpec("db", FieldType.STRING, optional=True),),
description="List locator-repair suggestions awaiting review.",
))
specs.append(CommandSpec(
"AC_repair_approve", "Tools", "Locator Repair: Approve",
fields=(
FieldSpec("suggestion_id", FieldType.STRING),
FieldSpec("db", FieldType.STRING, optional=True),
),
description="Approve a pending locator-repair suggestion.",
))
specs.append(CommandSpec(
"AC_generate_sop", "Report", "Generate SOP Document",
fields=(
Expand Down
38 changes: 38 additions & 0 deletions je_auto_control/utils/executor/action_executor.py
Original file line number Diff line number Diff line change
Expand Up @@ -3339,6 +3339,40 @@ def _decision_table(spec: Any, context: Any) -> Dict[str, Any]:
return {"result": evaluate_table(spec, context)}


def _repair_record(key: str, method: str, coordinates: Any = None,
description: Optional[str] = None, confidence: float = 1.0,
auto_threshold: float = 0.9,
db: Optional[str] = None) -> Dict[str, Any]:
"""Adapter: record a corrected locator from a heal (auto-apply or queue)."""
import json
from je_auto_control.utils.locator_repair import RepairStore
if isinstance(coordinates, str):
coordinates = json.loads(coordinates)
sug = RepairStore(db).record(
key, method=method, coordinates=coordinates, description=description,
confidence=confidence, auto_threshold=auto_threshold)
return {"id": sug.id, "status": sug.status}


def _repair_resolved(key: str, db: Optional[str] = None) -> Dict[str, Any]:
"""Adapter: return the learned corrected locator for a key (or null)."""
from je_auto_control.utils.locator_repair import RepairStore
return {"locator": RepairStore(db).resolved(key)}


def _repair_pending(db: Optional[str] = None) -> Dict[str, Any]:
"""Adapter: list locator-repair suggestions awaiting review."""
from je_auto_control.utils.locator_repair import RepairStore
return {"pending": RepairStore(db).pending()}


def _repair_approve(suggestion_id: str,
db: Optional[str] = None) -> Dict[str, Any]:
"""Adapter: approve a pending locator-repair suggestion."""
from je_auto_control.utils.locator_repair import RepairStore
return {"approved": RepairStore(db).approve(suggestion_id)}


class Executor:
"""
Executor
Expand Down Expand Up @@ -3623,6 +3657,10 @@ def __init__(self):
"AC_json_extract": _json_extract,
"AC_run_saga": _run_saga,
"AC_decision_table": _decision_table,
"AC_repair_record": _repair_record,
"AC_repair_resolved": _repair_resolved,
"AC_repair_pending": _repair_pending,
"AC_repair_approve": _repair_approve,
"AC_a11y_record_start": _a11y_record_start,
"AC_a11y_record_stop": _a11y_record_stop,
"AC_a11y_record_events": _a11y_record_events,
Expand Down
6 changes: 6 additions & 0 deletions je_auto_control/utils/locator_repair/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
"""Self-healing write-back: persist corrected locators from heal events."""
from je_auto_control.utils.locator_repair.locator_repair import (
RepairStore, RepairSuggestion, repair_from_heal,
)

__all__ = ["RepairStore", "RepairSuggestion", "repair_from_heal"]
Loading
Loading