Skip to content

Commit f370336

Browse files
committed
Wire visual regression and the state-machine engine through all layers
Two headless cores existed but were never exposed past their package, in violation of the project's "every feature ships headless + AC_ + GUI" rule. Complete them: - Visual regression: facade re-export of take_golden / compare_to_golden / DiffResult / MaskRegion; AC_take_golden and AC_assert_visual executor commands (assert auto-creates the baseline on first run, saves a diff on mismatch, raises like other AC_assert_*); ac_take_golden / ac_assert_visual MCP tools; Script Builder entries. - State machine: facade re-export of run_state_machine / StateMachine / StateMachineError; AC_run_state_machine command; ac_run_state_machine MCP tool; Script Builder entry. Adds headless tests (PIL images / specs injected) and a v6 new-features reference page (EN + Traditional Chinese) plus README sections.
1 parent 2b02786 commit f370336

14 files changed

Lines changed: 472 additions & 1 deletion

File tree

README.md

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313

1414
## Table of Contents
1515

16+
- [What's new (2026-06-19)](#whats-new-2026-06-19)
1617
- [What's new (2026-06-18)](#whats-new-2026-06-18)
1718
- [What's new (2026-06-17)](#whats-new-2026-06-17)
1819
- [What's new (2026-06)](#whats-new-2026-06)
@@ -58,6 +59,16 @@
5859

5960
---
6061

62+
## What's new (2026-06-19)
63+
64+
Two headless cores that shipped without the rest of their stack are now
65+
first-class. Both gain a facade re-export, an `AC_*` executor command, an
66+
MCP tool, and a Script Builder entry, with headless tests. Full reference:
67+
[`docs/source/Eng/doc/new_features/v6_features_doc.rst`](docs/source/Eng/doc/new_features/v6_features_doc.rst).
68+
69+
- **Visual regression (golden images)**`take_golden` / `compare_to_golden` (`AC_take_golden` / `AC_assert_visual`): capture a baseline screenshot and fail when the screen drifts beyond a pixel tolerance, with a highlighted diff image and mask regions. `AC_assert_visual` auto-creates the baseline on first run. PIL-only.
70+
- **Finite-state machine**`run_state_machine` (`AC_run_state_machine`): drive a script as a declarative `{initial, states}` spec whose `on_enter` actions run through the executor and whose transitions fire on `after` / `if_var_eq` / predicate guards, bounded by `max_steps` / `global_timeout_s`.
71+
6172
## What's new (2026-06-18)
6273

6374
Eight headless capabilities that round out scripting, integration, and CI

README/README_zh-CN.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212

1313
## 目录
1414

15+
- [本次更新 (2026-06-19)](#本次更新-2026-06-19)
1516
- [本次更新 (2026-06-18)](#本次更新-2026-06-18)
1617
- [本次更新 (2026-06-17)](#本次更新-2026-06-17)
1718
- [本次更新 (2026-06)](#本次更新-2026-06)
@@ -57,6 +58,14 @@
5758

5859
---
5960

61+
## 本次更新 (2026-06-19)
62+
63+
两个早已存在、却没接上其余各层的 headless 核心,现在成为一级功能。两者都新增 facade re-export、`AC_*` 执行器指令、MCP 工具与 Script Builder 项目,并有 headless 测试。完整参考:
64+
[`docs/source/Eng/doc/new_features/v6_features_doc.rst`](../docs/source/Eng/doc/new_features/v6_features_doc.rst)
65+
66+
- **视觉回归(黄金图像)**`take_golden` / `compare_to_golden`(`AC_take_golden` / `AC_assert_visual`):捕获基准截图,画面偏离超过像素容差时判失败,并输出高亮差异图与遮罩区域。`AC_assert_visual` 首跑会自动建立基准。纯 PIL。
67+
- **有限状态机**`run_state_machine`(`AC_run_state_machine`):把脚本当成声明式 `{initial, states}` spec 驱动,`on_enter` 动作经执行器执行,transition 依 `after` / `if_var_eq` / predicate 触发,并以 `max_steps` / `global_timeout_s` 限制。
68+
6069
## 本次更新 (2026-06-18)
6170

6271
八项 headless 能力,补齐脚本化、集成与 CI 场景:真正的命令行界面、把录制转成代码,以及一级的 HTTP / SQL / Email / PDF / 等待步骤。每项都附带 headless API、`AC_*` 执行器指令、MCP 工具与可视化脚本构建器项目,并有 headless 测试(网络 / SMTP / PDF 后端均注入,不接触外部系统)。完整参考页:

README/README_zh-TW.md

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212

1313
## 目錄
1414

15+
- [本次更新 (2026-06-19)](#本次更新-2026-06-19)
1516
- [本次更新 (2026-06-18)](#本次更新-2026-06-18)
1617
- [本次更新 (2026-06-17)](#本次更新-2026-06-17)
1718
- [本次更新 (2026-06)](#本次更新-2026-06)
@@ -57,6 +58,14 @@
5758

5859
---
5960

61+
## 本次更新 (2026-06-19)
62+
63+
兩個早已存在、卻沒接上其餘各層的 headless 核心,現在成為一級功能。兩者都新增 facade re-export、`AC_*` 執行器指令、MCP 工具與 Script Builder 項目,並有 headless 測試。完整參考:
64+
[`docs/source/Zh/doc/new_features/v6_features_doc.rst`](../docs/source/Zh/doc/new_features/v6_features_doc.rst)
65+
66+
- **視覺回歸(黃金影像)**`take_golden` / `compare_to_golden`(`AC_take_golden` / `AC_assert_visual`):擷取基準截圖,畫面偏離超過像素容差時判失敗,並輸出標示差異圖與遮罩區域。`AC_assert_visual` 首跑會自動建立基準。純 PIL。
67+
- **有限狀態機**`run_state_machine`(`AC_run_state_machine`):把腳本當成宣告式 `{initial, states}` spec 驅動,`on_enter` 動作經執行器執行,transition 依 `after` / `if_var_eq` / predicate 觸發,並以 `max_steps` / `global_timeout_s` 限制。
68+
6069
## 本次更新 (2026-06-18)
6170

6271
八項 headless 能力,補齊腳本化、整合與 CI 情境:真正的命令列介面、把錄製轉成程式碼,以及一級的 HTTP / SQL / Email / PDF / 等待步驟。每項都附帶 headless API、`AC_*` 執行器指令、MCP 工具與視覺化腳本建構器項目,並有 headless 測試(網路 / SMTP / PDF 後端皆注入,不碰外部系統)。完整參考頁:
Lines changed: 70 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,70 @@
1+
=========================================================
2+
New Features (2026-06-19) — Visual Regression & FSM
3+
=========================================================
4+
5+
Two headless cores that already existed but were never wired through the
6+
rest of the stack are now first-class: **golden-image visual regression**
7+
and a **declarative finite-state-machine** runner. Both ship a facade
8+
re-export, an ``AC_*`` executor command, an MCP tool, and a Script Builder
9+
entry, with headless tests (PIL images / specs are injected, so nothing
10+
needs a real screen).
11+
12+
.. contents::
13+
:local:
14+
:depth: 2
15+
16+
17+
Visual regression (golden images)
18+
=================================
19+
20+
Capture a baseline image and later fail a run when the screen drifts from
21+
it — the screenshot equivalent of a snapshot test::
22+
23+
from je_auto_control import take_golden, compare_to_golden
24+
25+
take_golden("goldens/login.png") # establish baseline
26+
result = compare_to_golden("goldens/login.png", tolerance=0.5)
27+
if not result.matched:
28+
result.write_diff("goldens/login.diff.png")
29+
print(result.summary)
30+
31+
``compare_to_golden`` returns a ``DiffResult`` (``matched``, ``diff_pct``,
32+
``differing_pixels``, a highlighted ``diff_image``); ``tolerance`` is the
33+
percentage of pixels allowed to differ and ``per_pixel_threshold`` ignores
34+
small per-channel noise. ``MaskRegion`` excludes animated / volatile areas.
35+
PIL-only — no OpenCV / SciPy dependency.
36+
37+
Executor commands:
38+
39+
* ``AC_take_golden`` — capture and save a baseline (optional ``region``).
40+
* ``AC_assert_visual`` — compare the screen to a golden and raise on
41+
mismatch (saving an optional ``diff_path``). On the **first run** (golden
42+
missing) it captures the baseline and passes, unless
43+
``create_if_missing`` is false.
44+
45+
46+
Finite-state machine
47+
====================
48+
49+
Drive a script as a declarative state machine — clearer than nested
50+
loops / ifs for screen-flow automation::
51+
52+
from je_auto_control import run_state_machine
53+
54+
spec = {
55+
"initial": "login",
56+
"states": {
57+
"login": {"on_enter": [["AC_click_text", {"text": "Sign in"}]],
58+
"transitions": [{"go_to": "home", "after": 1.0}]},
59+
"home": {"final": True},
60+
},
61+
}
62+
result = run_state_machine(spec) # {final_state, steps, elapsed_s}
63+
64+
Each state's ``on_enter`` actions run through the executor; transitions
65+
fire on guards (``after`` a delay, ``if_var_eq``, or a caller predicate).
66+
``max_steps`` and ``global_timeout_s`` bound the run so it can't loop
67+
forever.
68+
69+
Executor command: ``AC_run_state_machine`` (the ``spec`` dict travels in
70+
JSON action files / the socket server / MCP unchanged).

docs/source/Eng/eng_index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ Comprehensive guides for all AutoControl features.
2828
doc/new_features/v3_features_doc
2929
doc/new_features/v4_features_doc
3030
doc/new_features/v5_features_doc
31+
doc/new_features/v6_features_doc
3132
doc/ocr_backends/ocr_backends_doc
3233
doc/observability/observability_doc
3334
doc/operations_layer/operations_layer_doc
Lines changed: 65 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
============================================
2+
新功能 (2026-06-19) — 視覺回歸與狀態機
3+
============================================
4+
5+
兩個早已存在、卻從未接上其餘各層的 headless 核心,現在成為一級功能:
6+
**黃金影像視覺回歸**與**宣告式有限狀態機**執行器。兩者都提供 facade
7+
re-export、``AC_*`` 執行器指令、MCP 工具與 Script Builder 項目,並有
8+
headless 測試(PIL 影像 / spec 以注入方式提供,完全不需真實螢幕)。
9+
10+
.. contents::
11+
:local:
12+
:depth: 2
13+
14+
15+
視覺回歸(黃金影像)
16+
====================
17+
18+
擷取基準圖,之後在畫面偏離時讓該次執行失敗——等同於截圖版的 snapshot
19+
測試::
20+
21+
from je_auto_control import take_golden, compare_to_golden
22+
23+
take_golden("goldens/login.png") # 建立基準
24+
result = compare_to_golden("goldens/login.png", tolerance=0.5)
25+
if not result.matched:
26+
result.write_diff("goldens/login.diff.png")
27+
print(result.summary)
28+
29+
``compare_to_golden`` 回傳 ``DiffResult``(``matched``、``diff_pct``、
30+
``differing_pixels``、標示差異的 ``diff_image``);``tolerance`` 是允許
31+
差異的像素百分比,``per_pixel_threshold`` 可忽略各通道的細微雜訊。
32+
``MaskRegion`` 可排除動畫 / 易變區域。純 PIL——不需 OpenCV / SciPy。
33+
34+
執行器指令:
35+
36+
* ``AC_take_golden`` — 擷取並存基準圖(可選 ``region``)。
37+
* ``AC_assert_visual`` — 把畫面與黃金影像比對,不符即拋例外(可存
38+
``diff_path``)。**首跑**(基準不存在)時會擷取基準並通過,除非
39+
``create_if_missing`` 設為 false。
40+
41+
42+
有限狀態機
43+
==========
44+
45+
把腳本當成宣告式狀態機來驅動——對於畫面流程自動化,比巢狀 loop / if
46+
更清楚::
47+
48+
from je_auto_control import run_state_machine
49+
50+
spec = {
51+
"initial": "login",
52+
"states": {
53+
"login": {"on_enter": [["AC_click_text", {"text": "Sign in"}]],
54+
"transitions": [{"go_to": "home", "after": 1.0}]},
55+
"home": {"final": True},
56+
},
57+
}
58+
result = run_state_machine(spec) # {final_state, steps, elapsed_s}
59+
60+
每個狀態的 ``on_enter`` 動作會透過執行器執行;transition 依 guard 觸發
61+
(延遲 ``after``、``if_var_eq`` 或呼叫端 predicate)。``max_steps`` 與
62+
``global_timeout_s`` 限制執行,使其不會無限迴圈。
63+
64+
執行器指令:``AC_run_state_machine``(``spec`` dict 可原樣經 JSON 動作檔 /
65+
socket server / MCP 傳遞)。

docs/source/Zh/zh_index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,7 @@ AutoControl 所有功能的完整使用指南。
2828
doc/new_features/v3_features_doc
2929
doc/new_features/v4_features_doc
3030
doc/new_features/v5_features_doc
31+
doc/new_features/v6_features_doc
3132
doc/ocr_backends/ocr_backends_doc
3233
doc/observability/observability_doc
3334
doc/operations_layer/operations_layer_doc

je_auto_control/__init__.py

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -152,6 +152,14 @@
152152
wait_until_pixel_changes, wait_until_port, wait_until_region_idle,
153153
wait_until_screen_stable, wait_until_window_closed,
154154
)
155+
# Visual regression (golden-image comparison)
156+
from je_auto_control.utils.visual_regression import (
157+
DiffResult, MaskRegion, compare_to_golden, image_difference, take_golden,
158+
)
159+
# Declarative finite-state-machine engine for action JSON
160+
from je_auto_control.utils.state_machine import (
161+
StateMachine, StateMachineError, run_state_machine,
162+
)
155163
# Assertion DSL (verify screen state; raise on mismatch)
156164
from je_auto_control.utils.assertion import (
157165
AssertionResult, GroupAssertionResult, assert_all, assert_any,
@@ -570,6 +578,10 @@ def start_autocontrol_gui(*args, **kwargs):
570578
"wait_until_region_idle", "wait_until_screen_stable",
571579
"wait_until_clipboard_changes", "wait_until_window_closed",
572580
"wait_until_file", "wait_until_port",
581+
# Visual regression + state machine
582+
"take_golden", "compare_to_golden", "image_difference",
583+
"DiffResult", "MaskRegion",
584+
"run_state_machine", "StateMachine", "StateMachineError",
573585
# Assertion DSL
574586
"AssertionResult", "assert_image", "assert_pixel",
575587
"assert_text", "assert_window", "assert_clipboard", "assert_process",

je_auto_control/gui/script_builder/command_schema.py

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -562,6 +562,29 @@ def _add_misc_specs(specs: List[CommandSpec]) -> None:
562562
"AC_shell_command", "Shell", "Shell Command",
563563
fields=(FieldSpec("shell_command", FieldType.STRING),),
564564
))
565+
specs.append(CommandSpec(
566+
"AC_take_golden", "Report", "Capture Golden Image",
567+
fields=(FieldSpec("path", FieldType.FILE_PATH),),
568+
description="Capture and save a baseline image for visual regression.",
569+
))
570+
specs.append(CommandSpec(
571+
"AC_assert_visual", "Report", "Assert Visual (Golden)",
572+
fields=(
573+
FieldSpec("golden_path", FieldType.FILE_PATH),
574+
FieldSpec("tolerance", FieldType.FLOAT, optional=True, default=0.0,
575+
min_value=0.0),
576+
FieldSpec("per_pixel_threshold", FieldType.INT, optional=True,
577+
default=16, min_value=0),
578+
FieldSpec("diff_path", FieldType.FILE_PATH, optional=True),
579+
),
580+
description=("Compare the screen to a golden image; first run creates "
581+
"the baseline. Use the JSON view for a region / masks."),
582+
))
583+
specs.append(CommandSpec(
584+
"AC_run_state_machine", "Flow", "Run State Machine",
585+
description=("Run a finite-state-machine; configure the 'spec' "
586+
"{initial, states} dict in the JSON view."),
587+
))
565588
specs.append(CommandSpec(
566589
"AC_shell_to_var", "Shell", "Shell Output into Variable",
567590
fields=(

je_auto_control/utils/executor/action_executor.py

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2159,6 +2159,49 @@ def _assert_pdf_text(path: str, text: str, present: bool = True,
21592159
raise_on_fail=bool(raise_on_fail))
21602160

21612161

2162+
def _take_golden(path: str, region: Optional[List[int]] = None) -> str:
2163+
"""Adapter: capture and save a golden/baseline image."""
2164+
from je_auto_control.utils.visual_regression import take_golden
2165+
return str(take_golden(path, region=region))
2166+
2167+
2168+
def _assert_visual(golden_path: str, region: Optional[List[int]] = None,
2169+
tolerance: float = 0.0, per_pixel_threshold: int = 16,
2170+
diff_path: Optional[str] = None,
2171+
create_if_missing: bool = True,
2172+
raise_on_fail: bool = True) -> Dict[str, Any]:
2173+
"""Adapter: compare the screen to a golden image (first run creates it)."""
2174+
import os
2175+
from je_auto_control.utils.exception.exceptions import (
2176+
AutoControlAssertionException,
2177+
)
2178+
from je_auto_control.utils.visual_regression import (
2179+
compare_to_golden, take_golden,
2180+
)
2181+
if create_if_missing and not os.path.exists(
2182+
os.path.expanduser(str(golden_path))):
2183+
take_golden(golden_path, region=region)
2184+
return {"created": True, "matched": True, "golden": str(golden_path)}
2185+
result = compare_to_golden(
2186+
golden_path, region=region, tolerance=float(tolerance),
2187+
per_pixel_threshold=int(per_pixel_threshold))
2188+
if diff_path and result.diff_image is not None:
2189+
result.write_diff(diff_path)
2190+
data = {"matched": result.matched, "diff_pct": result.diff_pct,
2191+
"differing_pixels": result.differing_pixels,
2192+
"total_pixels": result.total_pixels,
2193+
"tolerance_pct": result.tolerance_pct}
2194+
if not result.matched and raise_on_fail:
2195+
raise AutoControlAssertionException(result.summary)
2196+
return data
2197+
2198+
2199+
def _run_state_machine(spec: Any) -> Dict[str, Any]:
2200+
"""Adapter: run a finite-state-machine spec through the executor."""
2201+
from je_auto_control.utils.state_machine import run_state_machine
2202+
return run_state_machine(spec)
2203+
2204+
21622205
class Executor:
21632206
"""
21642207
Executor
@@ -2224,6 +2267,9 @@ def __init__(self):
22242267
"AC_generate_code": _generate_code,
22252268
"AC_send_email": _send_email,
22262269
"AC_assert_pdf_text": _assert_pdf_text,
2270+
"AC_take_golden": _take_golden,
2271+
"AC_assert_visual": _assert_visual,
2272+
"AC_run_state_machine": _run_state_machine,
22272273
"AC_http_request": http_request,
22282274

22292275
# Record 錄製

0 commit comments

Comments
 (0)