diff --git a/README.md b/README.md index 654c10da..43965af5 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,7 @@ ## Table of Contents +- [What's new (2026-06-20) — Self-Healing Locator Write-Back](#whats-new-2026-06-20--self-healing-locator-write-back) - [What's new (2026-06-20) — DMN-Style Decision Tables](#whats-new-2026-06-20--dmn-style-decision-tables) - [What's new (2026-06-20) — Saga / Compensating Rollback](#whats-new-2026-06-20--saga--compensating-rollback) - [What's new (2026-06-20) — JSONPath Querying](#whats-new-2026-06-20--jsonpath-querying) @@ -106,6 +107,12 @@ --- +## What's new (2026-06-20) — Self-Healing Locator Write-Back + +Persist corrected locators so heals aren't forgotten. Full reference: [`docs/source/Eng/doc/new_features/v54_features_doc.rst`](docs/source/Eng/doc/new_features/v54_features_doc.rst). + +- **`RepairStore` / `repair_from_heal`** (`AC_repair_record` / `AC_repair_resolved` / `AC_repair_pending` / `AC_repair_approve`, `ac_*`): runtime self-healing previously **threw away** the corrected location, so every run re-healed. This records the corrected locator (coords/VLM description/method) from a heal, **auto-applies** it when `confidence >= auto_threshold` (default 0.9) or queues a reviewable suggestion, and `resolved(key)` returns the learned fix for reuse. Closes the heal→durable-fix loop; pure-stdlib, fully testable. + ## What's new (2026-06-20) — DMN-Style Decision Tables Externalize branching into reviewable rule tables. Full reference: [`docs/source/Eng/doc/new_features/v53_features_doc.rst`](docs/source/Eng/doc/new_features/v53_features_doc.rst). diff --git a/README/README_zh-CN.md b/README/README_zh-CN.md index 036b9db9..dea4b2a1 100644 --- a/README/README_zh-CN.md +++ b/README/README_zh-CN.md @@ -12,6 +12,7 @@ ## 目录 +- [本次更新 (2026-06-20) — 自我修复定位器回写](#本次更新-2026-06-20--自我修复定位器回写) - [本次更新 (2026-06-20) — DMN 式决策表](#本次更新-2026-06-20--dmn-式决策表) - [本次更新 (2026-06-20) — Saga / 补偿回滚](#本次更新-2026-06-20--saga--补偿回滚) - [本次更新 (2026-06-20) — JSONPath 查询](#本次更新-2026-06-20--jsonpath-查询) @@ -105,6 +106,12 @@ --- +## 本次更新 (2026-06-20) — 自我修复定位器回写 + +保存修正定位器,使修复不被遗忘。完整参考:[`docs/source/Zh/doc/new_features/v54_features_doc.rst`](../docs/source/Zh/doc/new_features/v54_features_doc.rst)。 + +- **`RepairStore` / `repair_from_heal`**(`AC_repair_record` / `AC_repair_resolved` / `AC_repair_pending` / `AC_repair_approve`、`ac_*`):运行期自我修复过去会**丢弃**修正后的位置,因此每次都重新修复。本功能记录该次修复的修正定位器(坐标/VLM 描述/方法),在 `confidence >= auto_threshold`(默认 0.9)时**自动套用**或排入可审查建议,`resolved(key)` 返回已学到的修正供重用。封闭「修复→持久修正」循环;纯标准库、可完整测试。 + ## 本次更新 (2026-06-20) — DMN 式决策表 将分支外部化为可审查的规则表。完整参考:[`docs/source/Zh/doc/new_features/v53_features_doc.rst`](../docs/source/Zh/doc/new_features/v53_features_doc.rst)。 diff --git a/README/README_zh-TW.md b/README/README_zh-TW.md index 873784f2..f4819912 100644 --- a/README/README_zh-TW.md +++ b/README/README_zh-TW.md @@ -12,6 +12,7 @@ ## 目錄 +- [本次更新 (2026-06-20) — 自我修復定位器回寫](#本次更新-2026-06-20--自我修復定位器回寫) - [本次更新 (2026-06-20) — DMN 式決策表](#本次更新-2026-06-20--dmn-式決策表) - [本次更新 (2026-06-20) — Saga / 補償回溯](#本次更新-2026-06-20--saga--補償回溯) - [本次更新 (2026-06-20) — JSONPath 查詢](#本次更新-2026-06-20--jsonpath-查詢) @@ -105,6 +106,12 @@ --- +## 本次更新 (2026-06-20) — 自我修復定位器回寫 + +保存修正定位器,使修復不被遺忘。完整參考:[`docs/source/Zh/doc/new_features/v54_features_doc.rst`](../docs/source/Zh/doc/new_features/v54_features_doc.rst)。 + +- **`RepairStore` / `repair_from_heal`**(`AC_repair_record` / `AC_repair_resolved` / `AC_repair_pending` / `AC_repair_approve`、`ac_*`):執行期自我修復過去會**丟棄**修正後的位置,因此每次都重新修復。本功能記錄該次修復的修正定位器(座標/VLM 描述/方法),在 `confidence >= auto_threshold`(預設 0.9)時**自動套用**或排入可審查建議,`resolved(key)` 回傳已學到的修正供重用。封閉「修復→持久修正」迴圈;純標準函式庫、可完整測試。 + ## 本次更新 (2026-06-20) — DMN 式決策表 將分支外部化為可審查的規則表。完整參考:[`docs/source/Zh/doc/new_features/v53_features_doc.rst`](../docs/source/Zh/doc/new_features/v53_features_doc.rst)。 diff --git a/docs/source/Eng/doc/new_features/v54_features_doc.rst b/docs/source/Eng/doc/new_features/v54_features_doc.rst new file mode 100644 index 00000000..74d56fdd --- /dev/null +++ b/docs/source/Eng/doc/new_features/v54_features_doc.rst @@ -0,0 +1,55 @@ +Self-Healing Locator Write-Back +=============================== + +The self-healing locator finds an element at a new place at runtime and logs the +heal — but the *corrected* location was then thrown away, so the next run healed +from scratch. ``RepairStore`` closes that loop: it records the corrected locator +(coordinates / VLM description / method) from a heal, **auto-applies** it when +confidence is high enough, otherwise queues it as a *pending suggestion* for +review. A later run reads the learned fix via :meth:`RepairStore.resolved`. + +JSON-backed (via the shared ``json_store`` helper); pure standard library; +confidence and threshold are explicit, so behavior is deterministic and fully +unit-testable. Imports no ``PySide6``. + +Headless API +------------ + +.. code-block:: python + + from je_auto_control import RepairStore, repair_from_heal + + store = RepairStore("repairs.json") + # high-confidence heal -> applied immediately: + store.record("login_btn", method="vlm", coordinates=[120, 64], + description="Login", confidence=0.95) + store.resolved("login_btn") # -> {method, coordinates, description} + + # low-confidence heal -> queued for review: + s = store.record("save_btn", method="image", coordinates=[5, 5], + confidence=0.5) + store.pending() # -> [the save_btn suggestion] + store.approve(s.id) # now store.resolved("save_btn") works + + # straight from a HealEvent (object or dict): + repair_from_heal(heal_event, "login_btn", store=store, confidence=0.9) + +``record`` auto-applies when ``confidence >= auto_threshold`` (default 0.9), else +files a ``pending`` suggestion; ``approve`` / ``reject`` decide queued ones; +``resolved(key)`` returns the latest applied/approved corrected locator (or +``None``) — the durable fix a future run reuses instead of re-healing. + +Executor commands +----------------- + +================================ =================================================== +Command Effect +================================ =================================================== +``AC_repair_record`` Persist a corrected locator (auto-apply or queue). +``AC_repair_resolved`` Get the learned corrected locator for a key. +``AC_repair_pending`` List suggestions awaiting review. +``AC_repair_approve`` Approve a pending suggestion. +================================ =================================================== + +The same operations are exposed as MCP tools (``ac_repair_*``) and as Script +Builder commands under **Tools**. diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst index 41a89b55..2b119e52 100644 --- a/docs/source/Eng/eng_index.rst +++ b/docs/source/Eng/eng_index.rst @@ -76,6 +76,7 @@ Comprehensive guides for all AutoControl features. doc/new_features/v51_features_doc doc/new_features/v52_features_doc doc/new_features/v53_features_doc + doc/new_features/v54_features_doc doc/ocr_backends/ocr_backends_doc doc/observability/observability_doc doc/operations_layer/operations_layer_doc diff --git a/docs/source/Zh/doc/new_features/v54_features_doc.rst b/docs/source/Zh/doc/new_features/v54_features_doc.rst new file mode 100644 index 00000000..54e79fe0 --- /dev/null +++ b/docs/source/Zh/doc/new_features/v54_features_doc.rst @@ -0,0 +1,51 @@ +自我修復定位器回寫 +================== + +自我修復定位器會在執行期於新位置找到元素並記錄該次修復 —— 但*修正後*的位置隨即被丟棄, +因此下次執行又得從頭修復。``RepairStore`` 補上這個迴圈:它記錄該次修復的修正定位器(座 +標 / VLM 描述 / 方法),在信心足夠高時**自動套用**,否則排入*待審建議*。之後的執行可透過 +:meth:`RepairStore.resolved` 讀取已學到的修正。 + +JSON 後端(透過共用 ``json_store`` 助手);純標準函式庫;信心與門檻皆為明確值,因此行為 +具確定性且可完整單元測試。不匯入 ``PySide6``。 + +無頭 API +-------- + +.. code-block:: python + + from je_auto_control import RepairStore, repair_from_heal + + store = RepairStore("repairs.json") + # 高信心修復 -> 立即套用: + store.record("login_btn", method="vlm", coordinates=[120, 64], + description="Login", confidence=0.95) + store.resolved("login_btn") # -> {method, coordinates, description} + + # 低信心修復 -> 排入審查: + s = store.record("save_btn", method="image", coordinates=[5, 5], + confidence=0.5) + store.pending() # -> [save_btn 建議] + store.approve(s.id) # 此後 store.resolved("save_btn") 可用 + + # 直接從 HealEvent(物件或 dict): + repair_from_heal(heal_event, "login_btn", store=store, confidence=0.9) + +``record`` 在 ``confidence >= auto_threshold``(預設 0.9)時自動套用,否則建立 ``pending`` +建議;``approve`` / ``reject`` 決定佇列中的項目;``resolved(key)`` 回傳最新已套用/已核准的 +修正定位器(或 ``None``)—— 供未來執行重用而不必重新修復的持久修正。 + +執行器指令 +---------- + +================================ =================================================== +指令 效果 +================================ =================================================== +``AC_repair_record`` 保存修正定位器(自動套用或排入佇列)。 +``AC_repair_resolved`` 取得某鍵已學到的修正定位器。 +``AC_repair_pending`` 列出待審的建議。 +``AC_repair_approve`` 核准一個待審建議。 +================================ =================================================== + +相同操作亦提供為 MCP 工具(``ac_repair_*``),以及 Script Builder 中 **Tools** 分類下的指 +令。 diff --git a/docs/source/Zh/zh_index.rst b/docs/source/Zh/zh_index.rst index cc5c06fe..fb3eb03c 100644 --- a/docs/source/Zh/zh_index.rst +++ b/docs/source/Zh/zh_index.rst @@ -76,6 +76,7 @@ AutoControl 所有功能的完整使用指南。 doc/new_features/v51_features_doc doc/new_features/v52_features_doc doc/new_features/v53_features_doc + doc/new_features/v54_features_doc doc/ocr_backends/ocr_backends_doc doc/observability/observability_doc doc/operations_layer/operations_layer_doc diff --git a/je_auto_control/__init__.py b/je_auto_control/__init__.py index c3defdc7..09524450 100644 --- a/je_auto_control/__init__.py +++ b/je_auto_control/__init__.py @@ -286,6 +286,10 @@ from je_auto_control.utils.decision_table import ( DecisionTable, Rule, evaluate_table, ) +# Self-healing write-back: persist corrected locators from heal events +from je_auto_control.utils.locator_repair import ( + RepairStore, RepairSuggestion, repair_from_heal, +) # Background popup/interrupt watchdog (unattended automation) from je_auto_control.utils.watchdog import ( PopupWatchdog, WatchdogRule, default_popup_watchdog, @@ -751,6 +755,7 @@ def start_autocontrol_gui(*args, **kwargs): "json_extract", "json_query", "json_query_one", "Saga", "SagaResult", "run_saga", "DecisionTable", "Rule", "evaluate_table", + "RepairStore", "RepairSuggestion", "repair_from_heal", # MCP server "AuditLogger", "HttpMCPServer", "MCPContent", "MCPPrompt", "MCPPromptArgument", "MCPResource", "MCPServer", "MCPTool", diff --git a/je_auto_control/gui/script_builder/command_schema.py b/je_auto_control/gui/script_builder/command_schema.py index 414f26c0..6f48d13b 100644 --- a/je_auto_control/gui/script_builder/command_schema.py +++ b/je_auto_control/gui/script_builder/command_schema.py @@ -1165,6 +1165,43 @@ def _add_misc_specs(specs: List[CommandSpec]) -> None: ), description="Evaluate inputs against a rule table (hit policy).", )) + specs.append(CommandSpec( + "AC_repair_record", "Tools", "Locator Repair: Record", + fields=( + FieldSpec("key", FieldType.STRING), + FieldSpec("method", FieldType.STRING, placeholder="vlm/image"), + FieldSpec("coordinates", FieldType.STRING, optional=True, + placeholder="[10, 20]"), + FieldSpec("description", FieldType.STRING, optional=True), + FieldSpec("confidence", FieldType.FLOAT, optional=True, + default=1.0), + FieldSpec("auto_threshold", FieldType.FLOAT, optional=True, + default=0.9), + FieldSpec("db", FieldType.STRING, optional=True), + ), + description="Persist a corrected locator from a heal (auto/queue).", + )) + specs.append(CommandSpec( + "AC_repair_resolved", "Tools", "Locator Repair: Resolved", + fields=( + FieldSpec("key", FieldType.STRING), + FieldSpec("db", FieldType.STRING, optional=True), + ), + description="Get the learned corrected locator for a key.", + )) + specs.append(CommandSpec( + "AC_repair_pending", "Tools", "Locator Repair: Pending", + fields=(FieldSpec("db", FieldType.STRING, optional=True),), + description="List locator-repair suggestions awaiting review.", + )) + specs.append(CommandSpec( + "AC_repair_approve", "Tools", "Locator Repair: Approve", + fields=( + FieldSpec("suggestion_id", FieldType.STRING), + FieldSpec("db", FieldType.STRING, optional=True), + ), + description="Approve a pending locator-repair suggestion.", + )) specs.append(CommandSpec( "AC_generate_sop", "Report", "Generate SOP Document", fields=( diff --git a/je_auto_control/utils/executor/action_executor.py b/je_auto_control/utils/executor/action_executor.py index d83118c1..5e93a992 100644 --- a/je_auto_control/utils/executor/action_executor.py +++ b/je_auto_control/utils/executor/action_executor.py @@ -3339,6 +3339,40 @@ def _decision_table(spec: Any, context: Any) -> Dict[str, Any]: return {"result": evaluate_table(spec, context)} +def _repair_record(key: str, method: str, coordinates: Any = None, + description: Optional[str] = None, confidence: float = 1.0, + auto_threshold: float = 0.9, + db: Optional[str] = None) -> Dict[str, Any]: + """Adapter: record a corrected locator from a heal (auto-apply or queue).""" + import json + from je_auto_control.utils.locator_repair import RepairStore + if isinstance(coordinates, str): + coordinates = json.loads(coordinates) + sug = RepairStore(db).record( + key, method=method, coordinates=coordinates, description=description, + confidence=confidence, auto_threshold=auto_threshold) + return {"id": sug.id, "status": sug.status} + + +def _repair_resolved(key: str, db: Optional[str] = None) -> Dict[str, Any]: + """Adapter: return the learned corrected locator for a key (or null).""" + from je_auto_control.utils.locator_repair import RepairStore + return {"locator": RepairStore(db).resolved(key)} + + +def _repair_pending(db: Optional[str] = None) -> Dict[str, Any]: + """Adapter: list locator-repair suggestions awaiting review.""" + from je_auto_control.utils.locator_repair import RepairStore + return {"pending": RepairStore(db).pending()} + + +def _repair_approve(suggestion_id: str, + db: Optional[str] = None) -> Dict[str, Any]: + """Adapter: approve a pending locator-repair suggestion.""" + from je_auto_control.utils.locator_repair import RepairStore + return {"approved": RepairStore(db).approve(suggestion_id)} + + class Executor: """ Executor @@ -3623,6 +3657,10 @@ def __init__(self): "AC_json_extract": _json_extract, "AC_run_saga": _run_saga, "AC_decision_table": _decision_table, + "AC_repair_record": _repair_record, + "AC_repair_resolved": _repair_resolved, + "AC_repair_pending": _repair_pending, + "AC_repair_approve": _repair_approve, "AC_a11y_record_start": _a11y_record_start, "AC_a11y_record_stop": _a11y_record_stop, "AC_a11y_record_events": _a11y_record_events, diff --git a/je_auto_control/utils/locator_repair/__init__.py b/je_auto_control/utils/locator_repair/__init__.py new file mode 100644 index 00000000..b82b4841 --- /dev/null +++ b/je_auto_control/utils/locator_repair/__init__.py @@ -0,0 +1,6 @@ +"""Self-healing write-back: persist corrected locators from heal events.""" +from je_auto_control.utils.locator_repair.locator_repair import ( + RepairStore, RepairSuggestion, repair_from_heal, +) + +__all__ = ["RepairStore", "RepairSuggestion", "repair_from_heal"] diff --git a/je_auto_control/utils/locator_repair/locator_repair.py b/je_auto_control/utils/locator_repair/locator_repair.py new file mode 100644 index 00000000..4d139c24 --- /dev/null +++ b/je_auto_control/utils/locator_repair/locator_repair.py @@ -0,0 +1,111 @@ +"""Turn a successful runtime heal into a durable, review-gated locator fix. + +The self-healing locator finds an element at a new place at runtime and logs the +heal — but the *corrected* location is then thrown away, so the next run heals +again from scratch. ``RepairStore`` closes that loop: it records the corrected +locator (coordinates / VLM description / method) from a heal, **auto-applies** it +when confidence is high enough, otherwise queues it as a *pending suggestion* for +review. A later run reads the learned fix via :meth:`RepairStore.resolved`. + +JSON-backed via the shared ``json_store`` helper; pure standard library; imports +no ``PySide6``. Confidence/threshold are explicit, so behavior is deterministic +and fully unit-testable. +""" +import secrets +from dataclasses import asdict, dataclass +from typing import Any, Dict, List, Optional + +STATUS_PENDING = "pending" +STATUS_APPLIED = "applied" +STATUS_APPROVED = "approved" +STATUS_REJECTED = "rejected" +_USABLE = (STATUS_APPLIED, STATUS_APPROVED) + + +@dataclass +class RepairSuggestion: + """A corrected locator derived from a heal, with a review status.""" + + id: str + key: str + method: str + confidence: float + status: str + coordinates: Optional[List[int]] = None + description: Optional[str] = None + + +class RepairStore: + """Records corrected locators; auto-applies confident ones, queues the rest.""" + + def __init__(self, db_path: Optional[str] = None) -> None: + """``db_path`` persists suggestions across runs (JSON).""" + from je_auto_control.utils.json_store import read_json_dict + self._path = db_path + data = read_json_dict(db_path) + self._items: List[Dict[str, Any]] = list(data.get("suggestions", [])) + + def _flush(self) -> None: + if self._path is not None: + from je_auto_control.utils.json_store import write_json_dict + write_json_dict(self._path, {"suggestions": self._items}) + + def record(self, key: str, *, method: str, + coordinates: Optional[List[int]] = None, + description: Optional[str] = None, confidence: float = 1.0, + auto_threshold: float = 0.9) -> RepairSuggestion: + """Record a corrected locator; auto-apply if ``confidence`` clears bar.""" + status = STATUS_APPLIED if confidence >= auto_threshold \ + else STATUS_PENDING + suggestion = RepairSuggestion( + id=secrets.token_hex(6), key=key, method=method, + confidence=float(confidence), status=status, + coordinates=list(coordinates) if coordinates else None, + description=description) + self._items.append(asdict(suggestion)) + self._flush() + return suggestion + + def _set_status(self, suggestion_id: str, new_status: str) -> bool: + for item in self._items: + if item["id"] == suggestion_id and item["status"] == STATUS_PENDING: + item["status"] = new_status + self._flush() + return True + return False + + def approve(self, suggestion_id: str) -> bool: + """Approve a pending suggestion (makes it usable by ``resolved``).""" + return self._set_status(suggestion_id, STATUS_APPROVED) + + def reject(self, suggestion_id: str) -> bool: + """Reject a pending suggestion.""" + return self._set_status(suggestion_id, STATUS_REJECTED) + + def pending(self) -> List[Dict[str, Any]]: + """Return suggestions awaiting review.""" + return [dict(i) for i in self._items if i["status"] == STATUS_PENDING] + + def resolved(self, key: str) -> Optional[Dict[str, Any]]: + """Return the latest applied/approved corrected locator for ``key``.""" + for item in reversed(self._items): + if item["key"] == key and item["status"] in _USABLE: + return {"method": item["method"], + "coordinates": item["coordinates"], + "description": item["description"]} + return None + + +def repair_from_heal(heal_event: Any, key: str, *, store: RepairStore, + confidence: float = 1.0, + auto_threshold: float = 0.9) -> RepairSuggestion: + """Record a repair from a ``HealEvent`` (object or dict) for ``key``.""" + def _field(name: str) -> Any: + if isinstance(heal_event, dict): + return heal_event.get(name) + return getattr(heal_event, name, None) + + return store.record( + key, method=str(_field("method") or "unknown"), + coordinates=_field("coordinates"), description=_field("description"), + confidence=confidence, auto_threshold=auto_threshold) diff --git a/je_auto_control/utils/mcp_server/tools/_factories.py b/je_auto_control/utils/mcp_server/tools/_factories.py index b31e8396..25754291 100644 --- a/je_auto_control/utils/mcp_server/tools/_factories.py +++ b/je_auto_control/utils/mcp_server/tools/_factories.py @@ -3262,6 +3262,53 @@ def decision_table_tools() -> List[MCPTool]: ] +def locator_repair_tools() -> List[MCPTool]: + _DB = {"db": {"type": "string"}} + return [ + MCPTool( + name="ac_repair_record", + description=("Record a corrected locator from a successful heal " + "(method + coordinates/description). Auto-applies when " + "'confidence' >= 'auto_threshold' (default 0.9), else " + "queues a pending suggestion. Returns {id, status}."), + input_schema=schema( + {"key": {"type": "string"}, "method": {"type": "string"}, + "coordinates": {"type": "array", "items": {"type": "integer"}}, + "description": {"type": "string"}, + "confidence": {"type": "number"}, + "auto_threshold": {"type": "number"}, **_DB}, + ["key", "method"]), + handler=h.repair_record, + annotations=SIDE_EFFECT_ONLY, + ), + MCPTool( + name="ac_repair_resolved", + description=("Return the latest applied/approved corrected locator " + "for 'key' (or null) — the learned fix for reuse."), + input_schema=schema({"key": {"type": "string"}, **_DB}, ["key"]), + handler=h.repair_resolved, + annotations=READ_ONLY, + ), + MCPTool( + name="ac_repair_pending", + description="List locator-repair suggestions awaiting review. " + "Returns {pending}.", + input_schema=schema(dict(_DB)), + handler=h.repair_pending, + annotations=READ_ONLY, + ), + MCPTool( + name="ac_repair_approve", + description="Approve a pending locator-repair suggestion by id. " + "Returns {approved}.", + input_schema=schema({"suggestion_id": {"type": "string"}, **_DB}, + ["suggestion_id"]), + handler=h.repair_approve, + annotations=SIDE_EFFECT_ONLY, + ), + ] + + def unattended_tools() -> List[MCPTool]: return [ MCPTool( @@ -4325,7 +4372,7 @@ def media_assert_tools() -> List[MCPTool]: video_report_tools, fuzzy_tools, artifact_store_tools, image_dedup_tools, locale_tools, voice_tools, coordinate_space_tools, loop_guard_tools, process_mining_tools, asset_tools, events_tools, notify_channel_tools, - jsonpath_tools, saga_tools, decision_table_tools, + jsonpath_tools, saga_tools, decision_table_tools, locator_repair_tools, screen_record_tools, process_and_shell_tools, remote_desktop_tools, gamepad_tools, usb_passthrough_tools, assertion_tools, data_source_tools, diff --git a/je_auto_control/utils/mcp_server/tools/_handlers.py b/je_auto_control/utils/mcp_server/tools/_handlers.py index 3a34e8ac..9e9fe00a 100644 --- a/je_auto_control/utils/mcp_server/tools/_handlers.py +++ b/je_auto_control/utils/mcp_server/tools/_handlers.py @@ -1569,6 +1569,30 @@ def decision_table(spec, context): return {"result": evaluate_table(spec, context)} +def repair_record(key, method, coordinates=None, description=None, + confidence=1.0, auto_threshold=0.9, db=None): + from je_auto_control.utils.locator_repair import RepairStore + sug = RepairStore(db).record( + key, method=method, coordinates=coordinates, description=description, + confidence=confidence, auto_threshold=auto_threshold) + return {"id": sug.id, "status": sug.status} + + +def repair_resolved(key, db=None): + from je_auto_control.utils.locator_repair import RepairStore + return {"locator": RepairStore(db).resolved(key)} + + +def repair_pending(db=None): + from je_auto_control.utils.locator_repair import RepairStore + return {"pending": RepairStore(db).pending()} + + +def repair_approve(suggestion_id, db=None): + from je_auto_control.utils.locator_repair import RepairStore + return {"approved": RepairStore(db).approve(suggestion_id)} + + def vlm_locate(description: str, screen_region: Optional[List[int]] = None, model: Optional[str] = None) -> Optional[List[int]]: diff --git a/test/unit_test/headless/test_locator_repair_batch.py b/test/unit_test/headless/test_locator_repair_batch.py new file mode 100644 index 00000000..5ee0ee87 --- /dev/null +++ b/test/unit_test/headless/test_locator_repair_batch.py @@ -0,0 +1,103 @@ +"""Headless tests for self-healing locator write-back. Pure stdlib (JSON store), +no model/screen; no Qt imports.""" +import je_auto_control as ac +from je_auto_control.utils.locator_repair import RepairStore, repair_from_heal + + +def test_high_confidence_auto_applies(): + store = RepairStore() + sug = store.record("login", method="vlm", coordinates=[10, 20], + description="Login", confidence=0.95) + assert sug.status == "applied" + assert store.resolved("login") == {"method": "vlm", + "coordinates": [10, 20], + "description": "Login"} + + +def test_low_confidence_queues_pending(): + store = RepairStore() + sug = store.record("save", method="image", coordinates=[5, 5], + confidence=0.5) + assert sug.status == "pending" + assert store.resolved("save") is None # not usable until approved + assert [p["key"] for p in store.pending()] == ["save"] + + +def test_approve_makes_resolvable(): + store = RepairStore() + sug = store.record("save", method="image", coordinates=[5, 5], + confidence=0.5) + assert store.approve(sug.id) is True + assert store.resolved("save")["coordinates"] == [5, 5] + assert store.pending() == [] + + +def test_reject_keeps_unresolved(): + store = RepairStore() + sug = store.record("x", method="image", confidence=0.1) + assert store.reject(sug.id) is True + assert store.resolved("x") is None + assert store.approve(sug.id) is False # already decided + + +def test_latest_wins(): + store = RepairStore() + store.record("btn", method="image", coordinates=[1, 1], confidence=1.0) + store.record("btn", method="vlm", coordinates=[2, 2], confidence=1.0) + assert store.resolved("btn")["coordinates"] == [2, 2] + + +def test_repair_from_heal_object_and_dict(): + store = RepairStore() + + class _Heal: + method = "vlm" + coordinates = [7, 8] + description = "OK" + + s1 = repair_from_heal(_Heal(), "a", store=store, confidence=1.0) + s2 = repair_from_heal({"method": "image", "coordinates": [9, 9]}, "b", + store=store, confidence=1.0) + assert s1.coordinates == [7, 8] and s2.method == "image" + + +def test_persists_across_instances(tmp_path): + db = str(tmp_path / "repairs.json") + sug = RepairStore(db).record("k", method="vlm", coordinates=[1, 2], + confidence=0.4) + assert RepairStore(db).approve(sug.id) is True + assert RepairStore(db).resolved("k")["coordinates"] == [1, 2] + + +# --- wiring --------------------------------------------------------------- + +def test_executor_round_trip(tmp_path): + db = str(tmp_path / "r.json") + ac.execute_action([["AC_repair_record", + {"key": "login", "method": "vlm", + "coordinates": [3, 4], "confidence": 0.95, "db": db}]]) + rec = ac.execute_action([["AC_repair_resolved", + {"key": "login", "db": db}]]) + loc = next(v for v in rec.values() if isinstance(v, dict))["locator"] + assert loc["coordinates"] == [3, 4] + + +def test_wiring(): + known = ac.executor.known_commands() + assert {"AC_repair_record", "AC_repair_resolved", "AC_repair_pending", + "AC_repair_approve"} <= known + from je_auto_control.utils.mcp_server.tools import ( + build_default_tool_registry) + names = {t.name for t in build_default_tool_registry()} + assert {"ac_repair_record", "ac_repair_resolved", "ac_repair_pending", + "ac_repair_approve"} <= names + from je_auto_control.gui.script_builder.command_schema import _build_specs + cmds = {s.command for s in _build_specs()} + assert {"AC_repair_record", "AC_repair_resolved", "AC_repair_pending", + "AC_repair_approve"} <= cmds + + +def test_facade_exports(): + for attr in ("RepairStore", "RepairSuggestion", "repair_from_heal"): + assert hasattr(ac, attr) + assert attr in ac.__all__