Skip to content

Commit 010f0d2

Browse files
JE-ChenJE-Chen
authored
Merge pull request #262 from Integration-Automation/feat/locator-repair-batch
Add self-healing locator write-back (learned-locator store)
2 parents e69e8a1 + 6f3fa1e commit 010f0d2

15 files changed

Lines changed: 501 additions & 1 deletion

File tree

‎README.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313

1414
## Table of Contents
1515

16+
- [What's new (2026-06-20) — Self-Healing Locator Write-Back](#whats-new-2026-06-20--self-healing-locator-write-back)
1617
- [What's new (2026-06-20) — DMN-Style Decision Tables](#whats-new-2026-06-20--dmn-style-decision-tables)
1718
- [What's new (2026-06-20) — Saga / Compensating Rollback](#whats-new-2026-06-20--saga--compensating-rollback)
1819
- [What's new (2026-06-20) — JSONPath Querying](#whats-new-2026-06-20--jsonpath-querying)
@@ -106,6 +107,12 @@
106107

107108
---
108109

110+
## What's new (2026-06-20) — Self-Healing Locator Write-Back
111+
112+
Persist corrected locators so heals aren't forgotten. Full reference: [`docs/source/Eng/doc/new_features/v54_features_doc.rst`](docs/source/Eng/doc/new_features/v54_features_doc.rst).
113+
114+
- **`RepairStore` / `repair_from_heal`** (`AC_repair_record` / `AC_repair_resolved` / `AC_repair_pending` / `AC_repair_approve`, `ac_*`): runtime self-healing previously **threw away** the corrected location, so every run re-healed. This records the corrected locator (coords/VLM description/method) from a heal, **auto-applies** it when `confidence >= auto_threshold` (default 0.9) or queues a reviewable suggestion, and `resolved(key)` returns the learned fix for reuse. Closes the heal→durable-fix loop; pure-stdlib, fully testable.
115+
109116
## What's new (2026-06-20) — DMN-Style Decision Tables
110117

111118
Externalize branching into reviewable rule tables. Full reference: [`docs/source/Eng/doc/new_features/v53_features_doc.rst`](docs/source/Eng/doc/new_features/v53_features_doc.rst).

‎README/README_zh-CN.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212

1313
## 目录
1414

15+
- [本次更新 (2026-06-20) — 自我修复定位器回写](#本次更新-2026-06-20--自我修复定位器回写)
1516
- [本次更新 (2026-06-20) — DMN 式决策表](#本次更新-2026-06-20--dmn-式决策表)
1617
- [本次更新 (2026-06-20) — Saga / 补偿回滚](#本次更新-2026-06-20--saga--补偿回滚)
1718
- [本次更新 (2026-06-20) — JSONPath 查询](#本次更新-2026-06-20--jsonpath-查询)
@@ -105,6 +106,12 @@
105106

106107
---
107108

109+
## 本次更新 (2026-06-20) — 自我修复定位器回写
110+
111+
保存修正定位器,使修复不被遗忘。完整参考:[`docs/source/Zh/doc/new_features/v54_features_doc.rst`](../docs/source/Zh/doc/new_features/v54_features_doc.rst)。
112+
113+
- **`RepairStore` / `repair_from_heal`**(`AC_repair_record` / `AC_repair_resolved` / `AC_repair_pending` / `AC_repair_approve`、`ac_*`):运行期自我修复过去会**丢弃**修正后的位置,因此每次都重新修复。本功能记录该次修复的修正定位器(坐标/VLM 描述/方法),在 `confidence >= auto_threshold`(默认 0.9)时**自动套用**或排入可审查建议,`resolved(key)` 返回已学到的修正供重用。封闭「修复→持久修正」循环;纯标准库、可完整测试。
114+
108115
## 本次更新 (2026-06-20) — DMN 式决策表
109116

110117
将分支外部化为可审查的规则表。完整参考:[`docs/source/Zh/doc/new_features/v53_features_doc.rst`](../docs/source/Zh/doc/new_features/v53_features_doc.rst)。

‎README/README_zh-TW.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212

1313
## 目錄
1414

15+
- [本次更新 (2026-06-20) — 自我修復定位器回寫](#本次更新-2026-06-20--自我修復定位器回寫)
1516
- [本次更新 (2026-06-20) — DMN 式決策表](#本次更新-2026-06-20--dmn-式決策表)
1617
- [本次更新 (2026-06-20) — Saga / 補償回溯](#本次更新-2026-06-20--saga--補償回溯)
1718
- [本次更新 (2026-06-20) — JSONPath 查詢](#本次更新-2026-06-20--jsonpath-查詢)
@@ -105,6 +106,12 @@
105106

106107
---
107108

109+
## 本次更新 (2026-06-20) — 自我修復定位器回寫
110+
111+
保存修正定位器,使修復不被遺忘。完整參考:[`docs/source/Zh/doc/new_features/v54_features_doc.rst`](../docs/source/Zh/doc/new_features/v54_features_doc.rst)。
112+
113+
- **`RepairStore` / `repair_from_heal`**(`AC_repair_record` / `AC_repair_resolved` / `AC_repair_pending` / `AC_repair_approve`、`ac_*`):執行期自我修復過去會**丟棄**修正後的位置,因此每次都重新修復。本功能記錄該次修復的修正定位器(座標/VLM 描述/方法),在 `confidence >= auto_threshold`(預設 0.9)時**自動套用**或排入可審查建議,`resolved(key)` 回傳已學到的修正供重用。封閉「修復→持久修正」迴圈;純標準函式庫、可完整測試。
114+
108115
## 本次更新 (2026-06-20) — DMN 式決策表
109116

110117
將分支外部化為可審查的規則表。完整參考:[`docs/source/Zh/doc/new_features/v53_features_doc.rst`](../docs/source/Zh/doc/new_features/v53_features_doc.rst)。

‎docs/source/Eng/doc/new_features/v54_features_doc.rst‎

Lines changed: 55 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,55 @@
1+
Self-Healing Locator Write-Back
2+
===============================
3+
4+
The self-healing locator finds an element at a new place at runtime and logs the
5+
heal — but the *corrected* location was then thrown away, so the next run healed
6+
from scratch. ``RepairStore`` closes that loop: it records the corrected locator
7+
(coordinates / VLM description / method) from a heal, **auto-applies** it when
8+
confidence is high enough, otherwise queues it as a *pending suggestion* for
9+
review. A later run reads the learned fix via :meth:`RepairStore.resolved`.
10+
11+
JSON-backed (via the shared ``json_store`` helper); pure standard library;
12+
confidence and threshold are explicit, so behavior is deterministic and fully
13+
unit-testable. Imports no ``PySide6``.
14+
15+
Headless API
16+
------------
17+
18+
.. code-block:: python
19+
20+
from je_auto_control import RepairStore, repair_from_heal
21+
22+
store = RepairStore("repairs.json")
23+
# high-confidence heal -> applied immediately:
24+
store.record("login_btn", method="vlm", coordinates=[120, 64],
25+
description="Login", confidence=0.95)
26+
store.resolved("login_btn") # -> {method, coordinates, description}
27+
28+
# low-confidence heal -> queued for review:
29+
s = store.record("save_btn", method="image", coordinates=[5, 5],
30+
confidence=0.5)
31+
store.pending() # -> [the save_btn suggestion]
32+
store.approve(s.id) # now store.resolved("save_btn") works
33+
34+
# straight from a HealEvent (object or dict):
35+
repair_from_heal(heal_event, "login_btn", store=store, confidence=0.9)
36+
37+
``record`` auto-applies when ``confidence >= auto_threshold`` (default 0.9), else
38+
files a ``pending`` suggestion; ``approve`` / ``reject`` decide queued ones;
39+
``resolved(key)`` returns the latest applied/approved corrected locator (or
40+
``None``) — the durable fix a future run reuses instead of re-healing.
41+
42+
Executor commands
43+
-----------------
44+
45+
================================ ===================================================
46+
Command Effect
47+
================================ ===================================================
48+
``AC_repair_record`` Persist a corrected locator (auto-apply or queue).
49+
``AC_repair_resolved`` Get the learned corrected locator for a key.
50+
``AC_repair_pending`` List suggestions awaiting review.
51+
``AC_repair_approve`` Approve a pending suggestion.
52+
================================ ===================================================
53+
54+
The same operations are exposed as MCP tools (``ac_repair_*``) and as Script
55+
Builder commands under **Tools**.

‎docs/source/Eng/eng_index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,7 @@ Comprehensive guides for all AutoControl features.
7676
doc/new_features/v51_features_doc
7777
doc/new_features/v52_features_doc
7878
doc/new_features/v53_features_doc
79+
doc/new_features/v54_features_doc
7980
doc/ocr_backends/ocr_backends_doc
8081
doc/observability/observability_doc
8182
doc/operations_layer/operations_layer_doc

‎docs/source/Zh/doc/new_features/v54_features_doc.rst‎

Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
自我修復定位器回寫
2+
==================
3+
4+
自我修復定位器會在執行期於新位置找到元素並記錄該次修復 —— 但*修正後*的位置隨即被丟棄,
5+
因此下次執行又得從頭修復。``RepairStore`` 補上這個迴圈:它記錄該次修復的修正定位器(座
6+
標 / VLM 描述 / 方法),在信心足夠高時**自動套用**,否則排入*待審建議*。之後的執行可透過
7+
:meth:`RepairStore.resolved` 讀取已學到的修正。
8+
9+
JSON 後端(透過共用 ``json_store`` 助手);純標準函式庫;信心與門檻皆為明確值,因此行為
10+
具確定性且可完整單元測試。不匯入 ``PySide6``。
11+
12+
無頭 API
13+
--------
14+
15+
.. code-block:: python
16+
17+
from je_auto_control import RepairStore, repair_from_heal
18+
19+
store = RepairStore("repairs.json")
20+
# 高信心修復 -> 立即套用:
21+
store.record("login_btn", method="vlm", coordinates=[120, 64],
22+
description="Login", confidence=0.95)
23+
store.resolved("login_btn") # -> {method, coordinates, description}
24+
25+
# 低信心修復 -> 排入審查:
26+
s = store.record("save_btn", method="image", coordinates=[5, 5],
27+
confidence=0.5)
28+
store.pending() # -> [save_btn 建議]
29+
store.approve(s.id) # 此後 store.resolved("save_btn") 可用
30+
31+
# 直接從 HealEvent(物件或 dict):
32+
repair_from_heal(heal_event, "login_btn", store=store, confidence=0.9)
33+
34+
``record`` 在 ``confidence >= auto_threshold``(預設 0.9)時自動套用,否則建立 ``pending``
35+
建議;``approve`` / ``reject`` 決定佇列中的項目;``resolved(key)`` 回傳最新已套用/已核准的
36+
修正定位器(或 ``None``)—— 供未來執行重用而不必重新修復的持久修正。
37+
38+
執行器指令
39+
----------
40+
41+
================================ ===================================================
42+
指令 效果
43+
================================ ===================================================
44+
``AC_repair_record`` 保存修正定位器(自動套用或排入佇列)。
45+
``AC_repair_resolved`` 取得某鍵已學到的修正定位器。
46+
``AC_repair_pending`` 列出待審的建議。
47+
``AC_repair_approve`` 核准一個待審建議。
48+
================================ ===================================================
49+
50+
相同操作亦提供為 MCP 工具(``ac_repair_*``),以及 Script Builder 中 **Tools** 分類下的指
51+
令。

‎docs/source/Zh/zh_index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -76,6 +76,7 @@ AutoControl 所有功能的完整使用指南。
7676
doc/new_features/v51_features_doc
7777
doc/new_features/v52_features_doc
7878
doc/new_features/v53_features_doc
79+
doc/new_features/v54_features_doc
7980
doc/ocr_backends/ocr_backends_doc
8081
doc/observability/observability_doc
8182
doc/operations_layer/operations_layer_doc

‎je_auto_control/__init__.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -286,6 +286,10 @@
286286
from je_auto_control.utils.decision_table import (
287287
DecisionTable, Rule, evaluate_table,
288288
)
289+
# Self-healing write-back: persist corrected locators from heal events
290+
from je_auto_control.utils.locator_repair import (
291+
RepairStore, RepairSuggestion, repair_from_heal,
292+
)
289293
# Background popup/interrupt watchdog (unattended automation)
290294
from je_auto_control.utils.watchdog import (
291295
PopupWatchdog, WatchdogRule, default_popup_watchdog,
@@ -751,6 +755,7 @@ def start_autocontrol_gui(*args, **kwargs):
751755
"json_extract", "json_query", "json_query_one",
752756
"Saga", "SagaResult", "run_saga",
753757
"DecisionTable", "Rule", "evaluate_table",
758+
"RepairStore", "RepairSuggestion", "repair_from_heal",
754759
# MCP server
755760
"AuditLogger", "HttpMCPServer", "MCPContent", "MCPPrompt",
756761
"MCPPromptArgument", "MCPResource", "MCPServer", "MCPTool",

‎je_auto_control/gui/script_builder/command_schema.py‎

Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1165,6 +1165,43 @@ def _add_misc_specs(specs: List[CommandSpec]) -> None:
11651165
),
11661166
description="Evaluate inputs against a rule table (hit policy).",
11671167
))
1168+
specs.append(CommandSpec(
1169+
"AC_repair_record", "Tools", "Locator Repair: Record",
1170+
fields=(
1171+
FieldSpec("key", FieldType.STRING),
1172+
FieldSpec("method", FieldType.STRING, placeholder="vlm/image"),
1173+
FieldSpec("coordinates", FieldType.STRING, optional=True,
1174+
placeholder="[10, 20]"),
1175+
FieldSpec("description", FieldType.STRING, optional=True),
1176+
FieldSpec("confidence", FieldType.FLOAT, optional=True,
1177+
default=1.0),
1178+
FieldSpec("auto_threshold", FieldType.FLOAT, optional=True,
1179+
default=0.9),
1180+
FieldSpec("db", FieldType.STRING, optional=True),
1181+
),
1182+
description="Persist a corrected locator from a heal (auto/queue).",
1183+
))
1184+
specs.append(CommandSpec(
1185+
"AC_repair_resolved", "Tools", "Locator Repair: Resolved",
1186+
fields=(
1187+
FieldSpec("key", FieldType.STRING),
1188+
FieldSpec("db", FieldType.STRING, optional=True),
1189+
),
1190+
description="Get the learned corrected locator for a key.",
1191+
))
1192+
specs.append(CommandSpec(
1193+
"AC_repair_pending", "Tools", "Locator Repair: Pending",
1194+
fields=(FieldSpec("db", FieldType.STRING, optional=True),),
1195+
description="List locator-repair suggestions awaiting review.",
1196+
))
1197+
specs.append(CommandSpec(
1198+
"AC_repair_approve", "Tools", "Locator Repair: Approve",
1199+
fields=(
1200+
FieldSpec("suggestion_id", FieldType.STRING),
1201+
FieldSpec("db", FieldType.STRING, optional=True),
1202+
),
1203+
description="Approve a pending locator-repair suggestion.",
1204+
))
11681205
specs.append(CommandSpec(
11691206
"AC_generate_sop", "Report", "Generate SOP Document",
11701207
fields=(

‎je_auto_control/utils/executor/action_executor.py‎

Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3339,6 +3339,40 @@ def _decision_table(spec: Any, context: Any) -> Dict[str, Any]:
33393339
return {"result": evaluate_table(spec, context)}
33403340

33413341

3342+
def _repair_record(key: str, method: str, coordinates: Any = None,
3343+
description: Optional[str] = None, confidence: float = 1.0,
3344+
auto_threshold: float = 0.9,
3345+
db: Optional[str] = None) -> Dict[str, Any]:
3346+
"""Adapter: record a corrected locator from a heal (auto-apply or queue)."""
3347+
import json
3348+
from je_auto_control.utils.locator_repair import RepairStore
3349+
if isinstance(coordinates, str):
3350+
coordinates = json.loads(coordinates)
3351+
sug = RepairStore(db).record(
3352+
key, method=method, coordinates=coordinates, description=description,
3353+
confidence=confidence, auto_threshold=auto_threshold)
3354+
return {"id": sug.id, "status": sug.status}
3355+
3356+
3357+
def _repair_resolved(key: str, db: Optional[str] = None) -> Dict[str, Any]:
3358+
"""Adapter: return the learned corrected locator for a key (or null)."""
3359+
from je_auto_control.utils.locator_repair import RepairStore
3360+
return {"locator": RepairStore(db).resolved(key)}
3361+
3362+
3363+
def _repair_pending(db: Optional[str] = None) -> Dict[str, Any]:
3364+
"""Adapter: list locator-repair suggestions awaiting review."""
3365+
from je_auto_control.utils.locator_repair import RepairStore
3366+
return {"pending": RepairStore(db).pending()}
3367+
3368+
3369+
def _repair_approve(suggestion_id: str,
3370+
db: Optional[str] = None) -> Dict[str, Any]:
3371+
"""Adapter: approve a pending locator-repair suggestion."""
3372+
from je_auto_control.utils.locator_repair import RepairStore
3373+
return {"approved": RepairStore(db).approve(suggestion_id)}
3374+
3375+
33423376
class Executor:
33433377
"""
33443378
Executor
@@ -3623,6 +3657,10 @@ def __init__(self):
36233657
"AC_json_extract": _json_extract,
36243658
"AC_run_saga": _run_saga,
36253659
"AC_decision_table": _decision_table,
3660+
"AC_repair_record": _repair_record,
3661+
"AC_repair_resolved": _repair_resolved,
3662+
"AC_repair_pending": _repair_pending,
3663+
"AC_repair_approve": _repair_approve,
36263664
"AC_a11y_record_start": _a11y_record_start,
36273665
"AC_a11y_record_stop": _a11y_record_stop,
36283666
"AC_a11y_record_events": _a11y_record_events,

0 commit comments

Comments
 (0)