Skip to content

Commit 2cd59aa

Browse files
JE-ChenJE-Chen
committed
Add resilience primitives: RetryPolicy, CircuitBreaker, AC_circuit_call
1 parent 7fa3ed5 commit 2cd59aa

15 files changed

Lines changed: 393 additions & 1 deletion

File tree

‎README.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313

1414
## Table of Contents
1515

16+
- [What's new (2026-06-19) — Resilience Primitives](#whats-new-2026-06-19--resilience-primitives)
1617
- [What's new (2026-06-19) — Timed Input Macros](#whats-new-2026-06-19--timed-input-macros)
1718
- [What's new (2026-06-19) — Semantic Screen State](#whats-new-2026-06-19--semantic-screen-state)
1819
- [What's new (2026-06-19) — Set-of-Marks Overlay](#whats-new-2026-06-19--set-of-marks-overlay)
@@ -77,6 +78,13 @@
7778

7879
---
7980

81+
## What's new (2026-06-19) — Resilience Primitives
82+
83+
Reusable retry + circuit-breaker primitives. Full reference: [`docs/source/Eng/doc/new_features/v25_features_doc.rst`](docs/source/Eng/doc/new_features/v25_features_doc.rst).
84+
85+
- **RetryPolicy** — `RetryPolicy(...).run(fn)` / `retry_call(fn)`: retry on configured exceptions with exponential backoff (injectable sleep). (The existing `AC_retry` flow command already retries an action body; this is the reusable callable wrapper.)
86+
- **CircuitBreaker** — `CircuitBreaker` / `CircuitOpenError` (`AC_circuit_call`, `ac_circuit_call`): open after N consecutive failures, short-circuit until a reset timeout, then half-open — stops a retry storm hammering a downed dependency. Injectable clock; `AC_circuit_call` runs an action list through a named breaker.
87+
8088
## What's new (2026-06-19) — Timed Input Macros
8189

8290
Replay input with timing fidelity + a press-hold-release DSL, full stack. Full reference: [`docs/source/Eng/doc/new_features/v24_features_doc.rst`](docs/source/Eng/doc/new_features/v24_features_doc.rst).

‎README/README_zh-CN.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212

1313
## 目录
1414

15+
- [本次更新 (2026-06-19) — 韧性原语](#本次更新-2026-06-19--韧性原语)
1516
- [本次更新 (2026-06-19) — 计时输入宏](#本次更新-2026-06-19--计时输入宏)
1617
- [本次更新 (2026-06-19) — 语义屏幕状态](#本次更新-2026-06-19--语义屏幕状态)
1718
- [本次更新 (2026-06-19) — Set-of-Marks 叠图](#本次更新-2026-06-19--set-of-marks-叠图)
@@ -76,6 +77,13 @@
7677

7778
---
7879

80+
## 本次更新 (2026-06-19) — 韧性原语
81+
82+
可重用的 retry 与断路器原语。完整参考:[`docs/source/Zh/doc/new_features/v25_features_doc.rst`](../docs/source/Zh/doc/new_features/v25_features_doc.rst)。
83+
84+
- **RetryPolicy** — `RetryPolicy(...).run(fn)` / `retry_call(fn)`:在配置的异常上以指数退避重试(可注入 sleep)。(既有 `AC_retry` 流程指令已能对动作 body 重试;这是可重用的可调用包装器。)
85+
- **CircuitBreaker** — `CircuitBreaker` / `CircuitOpenError`(`AC_circuit_call`、`ac_circuit_call`):连续失败 N 次后打开、短路至重置超时、再半开——避免重试风暴打垮已故障依赖。可注入 clock;`AC_circuit_call` 让动作列表通过具名断路器执行。
86+
7987
## 本次更新 (2026-06-19) — 计时输入宏
8088

8189
以时间保真度重播输入 + 按住-放开 DSL,走完整五层。完整参考:[`docs/source/Zh/doc/new_features/v24_features_doc.rst`](../docs/source/Zh/doc/new_features/v24_features_doc.rst)。

‎README/README_zh-TW.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212

1313
## 目錄
1414

15+
- [本次更新 (2026-06-19) — 韌性原語](#本次更新-2026-06-19--韌性原語)
1516
- [本次更新 (2026-06-19) — 計時輸入巨集](#本次更新-2026-06-19--計時輸入巨集)
1617
- [本次更新 (2026-06-19) — 語意螢幕狀態](#本次更新-2026-06-19--語意螢幕狀態)
1718
- [本次更新 (2026-06-19) — Set-of-Marks 疊圖](#本次更新-2026-06-19--set-of-marks-疊圖)
@@ -76,6 +77,13 @@
7677

7778
---
7879

80+
## 本次更新 (2026-06-19) — 韌性原語
81+
82+
可重用的 retry 與斷路器原語。完整參考:[`docs/source/Zh/doc/new_features/v25_features_doc.rst`](../docs/source/Zh/doc/new_features/v25_features_doc.rst)。
83+
84+
- **RetryPolicy** — `RetryPolicy(...).run(fn)` / `retry_call(fn)`:在設定的例外上以指數退避重試(可注入 sleep)。(既有 `AC_retry` 流程指令已能對動作 body 重試;這是可重用的可呼叫包裝器。)
85+
- **CircuitBreaker** — `CircuitBreaker` / `CircuitOpenError`(`AC_circuit_call`、`ac_circuit_call`):連續失敗 N 次後開啟、短路至重置逾時、再半開——避免重試風暴打掛已故障依賴。可注入 clock;`AC_circuit_call` 讓動作清單透過具名斷路器執行。
86+
7987
## 本次更新 (2026-06-19) — 計時輸入巨集
8088

8189
以時間保真度重播輸入 + 按住-放開 DSL,走完整五層。完整參考:[`docs/source/Zh/doc/new_features/v24_features_doc.rst`](../docs/source/Zh/doc/new_features/v24_features_doc.rst)。

‎docs/source/Eng/doc/new_features/v25_features_doc.rst‎

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
==================================================
2+
New Features (2026-06-19) — Resilience Primitives
3+
==================================================
4+
5+
Reusable resilience primitives — a retry-with-backoff policy and a circuit
6+
breaker — plus an executor command that runs an action list through a named
7+
breaker. Pure standard library; both primitives take injectable
8+
``sleep`` / ``clock`` so they are unit-tested deterministically.
9+
10+
(The existing ``AC_retry`` flow command already retries an action *body*;
11+
this adds the reusable :class:`RetryPolicy` callable wrapper and the new
12+
:class:`CircuitBreaker`.)
13+
14+
.. contents::
15+
:local:
16+
:depth: 2
17+
18+
19+
RetryPolicy
20+
==========
21+
22+
::
23+
24+
from je_auto_control import RetryPolicy, retry_call
25+
26+
RetryPolicy(max_attempts=5, backoff=0.1, multiplier=2.0).run(flaky_fn)
27+
retry_call(flaky_fn, max_attempts=3) # convenience
28+
29+
Retries ``func`` on the configured ``exceptions`` with exponential backoff
30+
(``backoff * multiplier**n``, optionally capped by ``max_backoff``),
31+
re-raising the last error when attempts are exhausted.
32+
33+
34+
CircuitBreaker
35+
=============
36+
37+
::
38+
39+
from je_auto_control import CircuitBreaker, CircuitOpenError
40+
41+
breaker = CircuitBreaker(failure_threshold=5, reset_timeout=30.0)
42+
try:
43+
breaker.call(call_remote_service)
44+
except CircuitOpenError:
45+
... # short-circuited — the dependency is down
46+
47+
Opens after ``failure_threshold`` consecutive failures and short-circuits
48+
(raising :class:`CircuitOpenError`) until ``reset_timeout`` elapses, then
49+
half-opens for one trial; a success closes it. Stops a retry storm from
50+
hammering a downed dependency.
51+
52+
``AC_circuit_call`` / ``ac_circuit_call`` run an action list through a
53+
**named** breaker (state shared across calls), returning ``{state, record}``.

‎docs/source/Eng/eng_index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,7 @@ Comprehensive guides for all AutoControl features.
4747
doc/new_features/v22_features_doc
4848
doc/new_features/v23_features_doc
4949
doc/new_features/v24_features_doc
50+
doc/new_features/v25_features_doc
5051
doc/ocr_backends/ocr_backends_doc
5152
doc/observability/observability_doc
5253
doc/operations_layer/operations_layer_doc

‎docs/source/Zh/doc/new_features/v25_features_doc.rst‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
==========================================
2+
新功能 (2026-06-19) — 韌性原語
3+
==========================================
4+
5+
可重用的韌性原語——retry-with-backoff 策略與斷路器(circuit breaker)
6+
——並提供一個執行器指令,讓動作清單透過具名斷路器執行。純標準庫;兩個
7+
原語都接受可注入的 ``sleep`` / ``clock``,因此能做決定性單元測試。
8+
9+
(既有的 ``AC_retry`` 流程指令已能對動作 *body* 重試;本功能新增可重用的
10+
:class:`RetryPolicy` 可呼叫包裝器與全新的 :class:`CircuitBreaker`。)
11+
12+
.. contents::
13+
:local:
14+
:depth: 2
15+
16+
17+
RetryPolicy
18+
===========
19+
20+
::
21+
22+
from je_auto_control import RetryPolicy, retry_call
23+
24+
RetryPolicy(max_attempts=5, backoff=0.1, multiplier=2.0).run(flaky_fn)
25+
retry_call(flaky_fn, max_attempts=3) # 便利函式
26+
27+
在設定的 ``exceptions`` 上以指數退避重試 ``func``(``backoff *
28+
multiplier**n``,可用 ``max_backoff`` 上限夾限),嘗試耗盡後重新拋出最後
29+
一個錯誤。
30+
31+
32+
CircuitBreaker
33+
==============
34+
35+
::
36+
37+
from je_auto_control import CircuitBreaker, CircuitOpenError
38+
39+
breaker = CircuitBreaker(failure_threshold=5, reset_timeout=30.0)
40+
try:
41+
breaker.call(call_remote_service)
42+
except CircuitOpenError:
43+
... # 已短路——依賴掛了
44+
45+
連續失敗達 ``failure_threshold`` 次後開啟並短路(拋出
46+
:class:`CircuitOpenError`),直到 ``reset_timeout`` 過去後半開試一次;成功
47+
即關閉。可避免重試風暴持續打掛已故障的依賴。
48+
49+
``AC_circuit_call`` / ``ac_circuit_call`` 讓動作清單透過**具名**斷路器
50+
執行(狀態跨呼叫共享),回傳 ``{state, record}``。

‎docs/source/Zh/zh_index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,7 @@ AutoControl 所有功能的完整使用指南。
4747
doc/new_features/v22_features_doc
4848
doc/new_features/v23_features_doc
4949
doc/new_features/v24_features_doc
50+
doc/new_features/v25_features_doc
5051
doc/ocr_backends/ocr_backends_doc
5152
doc/observability/observability_doc
5253
doc/operations_layer/operations_layer_doc

‎je_auto_control/__init__.py‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -176,6 +176,10 @@
176176
)
177177
# Timed input replay + declarative input-sequence DSL
178178
from je_auto_control.utils.input_macro import replay_timeline, run_sequence
179+
# Resilience primitives (retry-with-backoff + circuit breaker)
180+
from je_auto_control.utils.resilience import (
181+
CircuitBreaker, CircuitOpenError, RetryPolicy, retry_call,
182+
)
179183
# Background popup/interrupt watchdog (unattended automation)
180184
from je_auto_control.utils.watchdog import (
181185
PopupWatchdog, WatchdogRule, default_popup_watchdog,
@@ -604,6 +608,7 @@ def start_autocontrol_gui(*args, **kwargs):
604608
"describe_screen", "diff_snapshots", "screen_changed", "snapshot",
605609
"snapshot_screen",
606610
"replay_timeline", "run_sequence",
611+
"CircuitBreaker", "CircuitOpenError", "RetryPolicy", "retry_call",
607612
# MCP server
608613
"AuditLogger", "HttpMCPServer", "MCPContent", "MCPPrompt",
609614
"MCPPromptArgument", "MCPResource", "MCPServer", "MCPTool",

‎je_auto_control/gui/script_builder/command_schema.py‎

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -664,6 +664,19 @@ def _add_misc_specs(specs: List[CommandSpec]) -> None:
664664
_add_set_of_marks_specs(specs)
665665
_add_screen_state_specs(specs)
666666
_add_input_macro_specs(specs)
667+
_add_resilience_specs(specs)
668+
669+
670+
def _add_resilience_specs(specs: List[CommandSpec]) -> None:
671+
specs.append(CommandSpec(
672+
"AC_circuit_call", "Flow", "Circuit Breaker Call",
673+
fields=(
674+
FieldSpec("name", FieldType.STRING),
675+
FieldSpec("threshold", FieldType.INT, optional=True, default=5),
676+
FieldSpec("reset_s", FieldType.FLOAT, optional=True, default=30.0),
677+
),
678+
description="Run 'actions' (JSON view) via a named circuit breaker.",
679+
))
667680

668681

669682
def _add_input_macro_specs(specs: List[CommandSpec]) -> None:

‎je_auto_control/utils/executor/action_executor.py‎

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2794,6 +2794,20 @@ def _input_sequence(steps: List[Dict[str, Any]]) -> Dict[str, Any]:
27942794
return {"log": run_sequence(steps)}
27952795

27962796

2797+
_CIRCUIT_BREAKERS: Dict[str, Any] = {}
2798+
2799+
2800+
def _circuit_call(name: str, actions: List[Any], threshold: int = 5,
2801+
reset_s: float = 30.0) -> Dict[str, Any]:
2802+
"""Adapter: run an action list through a named circuit breaker."""
2803+
from je_auto_control.utils.resilience import CircuitBreaker
2804+
breaker = _CIRCUIT_BREAKERS.setdefault(
2805+
name, CircuitBreaker(int(threshold), float(reset_s)))
2806+
record = breaker.call(
2807+
lambda: executor.execute_action(list(actions), raise_on_error=True))
2808+
return {"state": breaker.state, "record": record}
2809+
2810+
27972811
class Executor:
27982812
"""
27992813
Executor
@@ -3012,6 +3026,7 @@ def __init__(self):
30123026
"AC_describe_screen": _describe_screen,
30133027
"AC_replay_timeline": _replay_timeline,
30143028
"AC_input_sequence": _input_sequence,
3029+
"AC_circuit_call": _circuit_call,
30153030
"AC_a11y_record_start": _a11y_record_start,
30163031
"AC_a11y_record_stop": _a11y_record_stop,
30173032
"AC_a11y_record_events": _a11y_record_events,

0 commit comments

Comments
 (0)