Skip to content

Commit fdcf2eb

Browse files
committed
Add native-UI element repository and flow step debugger
1 parent a049e3a commit fdcf2eb

17 files changed

Lines changed: 710 additions & 0 deletions

File tree

README.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313

1414
## Table of Contents
1515

16+
- [What's new (2026-06-19) — Authoring & Debugging](#whats-new-2026-06-19--authoring--debugging)
1617
- [What's new (2026-06-19) — Test & Tooling Batch](#whats-new-2026-06-19--test--tooling-batch)
1718
- [What's new (2026-06-19) — Transactional Queue](#whats-new-2026-06-19--transactional-queue)
1819
- [What's new (2026-06-19) — Unattended Reliability](#whats-new-2026-06-19--unattended-reliability)
@@ -64,6 +65,13 @@
6465

6566
---
6667

68+
## What's new (2026-06-19) — Authoring & Debugging
69+
70+
Two pure-stdlib authoring-time tools, full stack (facade, `AC_*`, MCP, Script Builder). Full reference: [`docs/source/Eng/doc/new_features/v12_features_doc.rst`](docs/source/Eng/doc/new_features/v12_features_doc.rst).
71+
72+
- **Element repository**`ElementRepository` (`AC_element_save` / `AC_element_find` / `AC_element_click` / `AC_element_remove` / `AC_element_list`, `ac_element_*`): save native-UI locators under friendly names (object repository) and reuse them — `repo.click("login.submit")` instead of repeating name/role everywhere; a UI change is fixed in one place.
73+
- **Step debugger / tracer**`FlowDebugger` (breakpoints, `step`/`continue_`/`run_to_end`, live `variables()`) and `trace_actions` (`AC_debug_trace`, `ac_debug_trace`): step through an action list one command at a time with variables persisting across steps, or get a per-step `{index, command, result}` trace (with `dry_run` to plan without running).
74+
6775
## What's new (2026-06-19) — Test & Tooling Batch
6876

6977
Three pure-stdlib quality-of-life tools, full stack (facade, `AC_*`, MCP, Script Builder). Full reference: [`docs/source/Eng/doc/new_features/v11_features_doc.rst`](docs/source/Eng/doc/new_features/v11_features_doc.rst).

README/README_zh-CN.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212

1313
## 目录
1414

15+
- [本次更新 (2026-06-19) — 编写与调试](#本次更新-2026-06-19--编写与调试)
1516
- [本次更新 (2026-06-19) — 测试与工具三件套](#本次更新-2026-06-19--测试与工具三件套)
1617
- [本次更新 (2026-06-19) — 事务式工作队列](#本次更新-2026-06-19--事务式工作队列)
1718
- [本次更新 (2026-06-19) — 无人值守可靠性](#本次更新-2026-06-19--无人值守可靠性)
@@ -63,6 +64,13 @@
6364

6465
---
6566

67+
## 本次更新 (2026-06-19) — 编写与调试
68+
69+
两项纯标准库的编写期工具,走完整五层(facade、`AC_*`、MCP、Script Builder)。完整参考:[`docs/source/Zh/doc/new_features/v12_features_doc.rst`](../docs/source/Zh/doc/new_features/v12_features_doc.rst)
70+
71+
- **元素库**`ElementRepository`(`AC_element_save` / `AC_element_find` / `AC_element_click` / `AC_element_remove` / `AC_element_list``ac_element_*`):把原生 UI 定位器以友好名称存起来(object repository)重用——用 `repo.click("login.submit")` 取代到处重复 name/role;UI 变动只需改一处。
72+
- **步进调试器 / 追踪器**`FlowDebugger`(断点、`step`/`continue_`/`run_to_end`、实时 `variables()`)与 `trace_actions`(`AC_debug_trace``ac_debug_trace`):把动作列表一次跑一个指令、变量跨步保留,或获取每步 `{index, command, result}` 追踪(用 `dry_run` 只规划不执行)。
73+
6674
## 本次更新 (2026-06-19) — 测试与工具三件套
6775

6876
三项纯标准库的生产力工具,走完整五层(facade、`AC_*`、MCP、Script Builder)。完整参考:[`docs/source/Zh/doc/new_features/v11_features_doc.rst`](../docs/source/Zh/doc/new_features/v11_features_doc.rst)

README/README_zh-TW.md

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212

1313
## 目錄
1414

15+
- [本次更新 (2026-06-19) — 編寫與除錯](#本次更新-2026-06-19--編寫與除錯)
1516
- [本次更新 (2026-06-19) — 測試與工具三件套](#本次更新-2026-06-19--測試與工具三件套)
1617
- [本次更新 (2026-06-19) — 交易式工作佇列](#本次更新-2026-06-19--交易式工作佇列)
1718
- [本次更新 (2026-06-19) — 無人值守可靠性](#本次更新-2026-06-19--無人值守可靠性)
@@ -63,6 +64,13 @@
6364

6465
---
6566

67+
## 本次更新 (2026-06-19) — 編寫與除錯
68+
69+
兩項純標準庫的編寫期工具,走完整五層(facade、`AC_*`、MCP、Script Builder)。完整參考:[`docs/source/Zh/doc/new_features/v12_features_doc.rst`](../docs/source/Zh/doc/new_features/v12_features_doc.rst)
70+
71+
- **元素庫**`ElementRepository`(`AC_element_save` / `AC_element_find` / `AC_element_click` / `AC_element_remove` / `AC_element_list``ac_element_*`):把原生 UI 定位器以友善名稱存起來(object repository)重用——用 `repo.click("login.submit")` 取代到處重複 name/role;UI 變動只需改一處。
72+
- **步進除錯器 / 追蹤器**`FlowDebugger`(中斷點、`step`/`continue_`/`run_to_end`、即時 `variables()`)與 `trace_actions`(`AC_debug_trace``ac_debug_trace`):把動作清單一次跑一個指令、變數跨步保留,或取得每步 `{index, command, result}` 追蹤(用 `dry_run` 只規劃不執行)。
73+
6674
## 本次更新 (2026-06-19) — 測試與工具三件套
6775

6876
三項純標準庫的生產力工具,走完整五層(facade、`AC_*`、MCP、Script Builder)。完整參考:[`docs/source/Zh/doc/new_features/v11_features_doc.rst`](../docs/source/Zh/doc/new_features/v11_features_doc.rst)
Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
====================================================
2+
New Features (2026-06-19) — Authoring & Debugging
3+
====================================================
4+
5+
Two authoring-time tools, pure standard library and wired through the
6+
full stack (facade, ``AC_*`` executor commands, MCP tools, Script
7+
Builder): a native-UI **element repository** (object repository) and a
8+
**step debugger / tracer** for action lists.
9+
10+
.. contents::
11+
:local:
12+
:depth: 2
13+
14+
15+
Element repository
16+
=================
17+
18+
Save native-UI locators under friendly names once, reuse them everywhere
19+
— the classic RPA *object repository*. A flow references
20+
``"login.submit"`` instead of repeating ``name="Submit", role="button"``
21+
at every call site, and a UI change is fixed in one place::
22+
23+
from je_auto_control import ElementRepository
24+
25+
repo = ElementRepository("app.objects.json")
26+
repo.save("login.submit", name="Submit", role="button")
27+
repo.save("login.user", role="edit", app_name="MyApp")
28+
29+
repo.click("login.submit") # resolve + click the live element
30+
info = repo.find_info("login.user") # {found, name, role, center}
31+
32+
A locator is a small set of accessibility filters (``name`` / ``role`` /
33+
``app_name``); resolving finds the live element through the accessibility
34+
backend. Storage is a JSON file and works on any platform; resolution
35+
needs a platform accessibility backend.
36+
37+
Executor / MCP commands: ``AC_element_save`` / ``AC_element_find`` /
38+
``AC_element_click`` / ``AC_element_remove`` / ``AC_element_list`` (and
39+
the matching ``ac_element_*`` MCP tools).
40+
41+
42+
Step debugger and tracer
43+
=======================
44+
45+
Run an action list one command at a time with breakpoints, single-step,
46+
and live variable inspection. Stepping reuses one executor instance, so
47+
script variables (``${name}`` interpolation, ``AC_set_var`` …) persist
48+
across steps exactly as in a normal run::
49+
50+
from je_auto_control import FlowDebugger
51+
52+
dbg = FlowDebugger(actions, breakpoints=[3])
53+
dbg.continue_() # run until the breakpoint
54+
dbg.variables() # inspect live variables
55+
dbg.step() # one command at a time
56+
dbg.run_to_end()
57+
58+
The stateless one-shot form, :func:`trace_actions`, runs a list (or
59+
``dry_run`` to only plan it) and returns a per-step trace of
60+
``{index, command, result}`` — exposed as ``AC_debug_trace`` /
61+
``ac_debug_trace``::
62+
63+
from je_auto_control import trace_actions
64+
65+
plan = trace_actions(actions, dry_run=True) # plan without running
66+
trace = trace_actions(actions) # run and trace

docs/source/Eng/eng_index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ Comprehensive guides for all AutoControl features.
3434
doc/new_features/v9_features_doc
3535
doc/new_features/v10_features_doc
3636
doc/new_features/v11_features_doc
37+
doc/new_features/v12_features_doc
3738
doc/ocr_backends/ocr_backends_doc
3839
doc/observability/observability_doc
3940
doc/operations_layer/operations_layer_doc
Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
========================================
2+
新功能 (2026-06-19) — 編寫與除錯
3+
========================================
4+
5+
兩項編寫期工具,皆為純標準庫,並走完整五層(facade、``AC_*`` 執行器
6+
指令、MCP 工具、Script Builder):原生 UI 的**元素庫**(object
7+
repository)與動作清單的**步進除錯器 / 追蹤器**。
8+
9+
.. contents::
10+
:local:
11+
:depth: 2
12+
13+
14+
元素庫
15+
======
16+
17+
把原生 UI 定位器以友善名稱存一次、到處重用——這就是 RPA 經典的
18+
*object repository*。流程改以 ``"login.submit"`` 引用,而不必在每個
19+
呼叫點重複 ``name="Submit", role="button"``;UI 變動只需改一處::
20+
21+
from je_auto_control import ElementRepository
22+
23+
repo = ElementRepository("app.objects.json")
24+
repo.save("login.submit", name="Submit", role="button")
25+
repo.save("login.user", role="edit", app_name="MyApp")
26+
27+
repo.click("login.submit") # 解析並點擊實際元素
28+
info = repo.find_info("login.user") # {found, name, role, center}
29+
30+
定位器是一組小的 accessibility 過濾條件(``name`` / ``role`` /
31+
``app_name``);解析時透過 accessibility 後端找到實際元素。儲存為 JSON
32+
檔、跨平台可用;解析需要平台的 accessibility 後端。
33+
34+
執行器 / MCP 指令:``AC_element_save`` / ``AC_element_find`` /
35+
``AC_element_click`` / ``AC_element_remove`` / ``AC_element_list``
36+
(以及對應的 ``ac_element_*`` MCP 工具)。
37+
38+
39+
步進除錯器與追蹤器
40+
==================
41+
42+
把動作清單一次跑一個指令,具備中斷點、單步與即時變數檢視。步進時重用
43+
同一個執行器實例,因此腳本變數(``${name}`` 插值、``AC_set_var`` …)
44+
會像正常執行一樣在各步之間保留::
45+
46+
from je_auto_control import FlowDebugger
47+
48+
dbg = FlowDebugger(actions, breakpoints=[3])
49+
dbg.continue_() # 跑到中斷點
50+
dbg.variables() # 檢視即時變數
51+
dbg.step() # 一次一個指令
52+
dbg.run_to_end()
53+
54+
無狀態的一次性形式 :func:`trace_actions` 會跑完清單(或用 ``dry_run``
55+
只做規劃),回傳每步的 ``{index, command, result}`` 追蹤——對應
56+
``AC_debug_trace`` / ``ac_debug_trace``::
57+
58+
from je_auto_control import trace_actions
59+
60+
plan = trace_actions(actions, dry_run=True) # 只規劃不執行
61+
trace = trace_actions(actions) # 執行並追蹤

docs/source/Zh/zh_index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -34,6 +34,7 @@ AutoControl 所有功能的完整使用指南。
3434
doc/new_features/v9_features_doc
3535
doc/new_features/v10_features_doc
3636
doc/new_features/v11_features_doc
37+
doc/new_features/v12_features_doc
3738
doc/ocr_backends/ocr_backends_doc
3839
doc/observability/observability_doc
3940
doc/operations_layer/operations_layer_doc

je_auto_control/__init__.py

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,10 @@
121121
from je_auto_control.utils.mcp_registry import (
122122
build_server_manifest, write_server_manifest,
123123
)
124+
# Named locator repository (object repository) for native UI
125+
from je_auto_control.utils.element_repository import ElementRepository
126+
# Step-through debugger / tracer for action lists
127+
from je_auto_control.utils.flow_debugger import FlowDebugger, trace_actions
124128
# Background popup/interrupt watchdog (unattended automation)
125129
from je_auto_control.utils.watchdog import (
126130
PopupWatchdog, WatchdogRule, default_popup_watchdog,
@@ -525,6 +529,8 @@ def start_autocontrol_gui(*args, **kwargs):
525529
"generate_rows", "write_dataset",
526530
"rank_flows", "select_flows",
527531
"build_server_manifest", "write_server_manifest",
532+
"ElementRepository",
533+
"FlowDebugger", "trace_actions",
528534
# MCP server
529535
"AuditLogger", "HttpMCPServer", "MCPContent", "MCPPrompt",
530536
"MCPPromptArgument", "MCPResource", "MCPServer", "MCPTool",

je_auto_control/gui/script_builder/command_schema.py

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -654,6 +654,46 @@ def _add_misc_specs(specs: List[CommandSpec]) -> None:
654654
))
655655
_add_work_queue_specs(specs)
656656
_add_tooling_specs(specs)
657+
_add_authoring_specs(specs)
658+
659+
660+
def _add_authoring_specs(specs: List[CommandSpec]) -> None:
661+
path = FieldSpec("path", FieldType.FILE_PATH)
662+
key = FieldSpec("key", FieldType.STRING)
663+
specs.append(CommandSpec(
664+
"AC_element_save", "Native UI", "Element: Save Locator",
665+
fields=(path, key,
666+
FieldSpec("name", FieldType.STRING, optional=True),
667+
FieldSpec("role", FieldType.STRING, optional=True),
668+
FieldSpec("app_name", FieldType.STRING, optional=True)),
669+
description="Save a named native-UI locator (object repository).",
670+
))
671+
specs.append(CommandSpec(
672+
"AC_element_find", "Native UI", "Element: Find Saved",
673+
fields=(path, key),
674+
description="Resolve a saved locator to a live element summary.",
675+
))
676+
specs.append(CommandSpec(
677+
"AC_element_click", "Native UI", "Element: Click Saved",
678+
fields=(path, key),
679+
description="Click the element behind a saved locator.",
680+
))
681+
specs.append(CommandSpec(
682+
"AC_element_remove", "Native UI", "Element: Remove Saved",
683+
fields=(path, key),
684+
description="Delete a saved locator.",
685+
))
686+
specs.append(CommandSpec(
687+
"AC_element_list", "Native UI", "Element: List Saved",
688+
fields=(path,),
689+
description="List saved locator names in a repository file.",
690+
))
691+
specs.append(CommandSpec(
692+
"AC_debug_trace", "Flow", "Debug: Trace Actions",
693+
fields=(FieldSpec("dry_run", FieldType.BOOL, optional=True,
694+
default=False),),
695+
description="Run 'actions' (JSON view) and return a per-step trace.",
696+
))
657697

658698

659699
def _add_tooling_specs(specs: List[CommandSpec]) -> None:
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
"""Named locator repository (object repository) for native-UI elements."""
2+
from je_auto_control.utils.element_repository.element_repository import (
3+
ElementRepository,
4+
)
5+
6+
__all__ = ["ElementRepository"]

0 commit comments

Comments
 (0)