Merge pull request #436 from Integration-Automation/feat/retry-budget-batch

Add retry_budget: deadline + jitter over resilience retries

Author: JE-Chen

WHATS_NEW.md

@@ -2,6 +2,12 @@
 
 ## What's new (2026-06-26)
 
+### Retry Budget — Deadline + Jitter
+
+Retry a flaky step bounded by a total time budget, with jittered backoff. Full reference: [`docs/source/Eng/doc/new_features/v209_features_doc.rst`](docs/source/Eng/doc/new_features/v209_features_doc.rst).
+
+- **`RetryBudget` / `run_with_budget` / `backoff_delay` / `jittered_delay`** (`AC_retry_delay`, `AC_plan_retry_delays`): `resilience.RetryPolicy` retries a fixed attempt count with plain exponential backoff — it can't express a *wall-clock deadline* ("give up after 30 s total, however many attempts that is") or *jitter* (randomized backoff so retrying workers don't resynchronize into a thundering herd). `RetryBudget` adds both: bounded by `max_attempts` *and/or* `deadline_s`, `run_with_budget` honours whichever is hit first and never sleeps past the deadline; delays use capped exponential backoff with a selectable `full`/`equal`/`none` jitter strategy. The randomness (`uniform`), clock and sleeper are all injectable, so every delay and giveup decision is deterministic in tests. First feature of the ROUND-15 input-fidelity lane. No `PySide6`.
+
 ### Live IME State for Safe CJK Entry
 
 Wait for the input method to commit before reading a Japanese/Chinese/Korean field. Full reference: [`docs/source/Eng/doc/new_features/v208_features_doc.rst`](docs/source/Eng/doc/new_features/v208_features_doc.rst).

docs/source/Eng/doc/new_features/v209_features_doc.rst

@@ -0,0 +1,56 @@
+Retry Budget — Deadline + Jitter
+================================
+
+:class:`resilience.RetryPolicy` retries a fixed number of attempts with plain
+exponential backoff. Two things it can't express are exactly what flaky,
+contended UI automation needs:
+
+* a **wall-clock deadline** — "keep retrying, but give up after 30 s total",
+  independent of how many attempts that takes; and
+* **jitter** — randomized backoff so many retrying workers don't resynchronize
+  into a thundering herd.
+
+``retry_budget`` adds both. :class:`RetryBudget` is bounded by ``max_attempts``
+*and / or* ``deadline_s``; :func:`run_with_budget` honours whichever is hit
+first and never sleeps past the deadline. Delays use capped exponential backoff
+with a selectable jitter strategy (``full`` / ``equal`` / ``none``). The
+randomness source (``uniform``), the clock and the sleeper are all injectable,
+so every delay and decision is deterministic in tests. Imports no ``PySide6``.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import RetryBudget, run_with_budget
+
+    budget = RetryBudget(max_attempts=8, deadline_s=30.0,
+                         base_delay_s=0.2, max_delay_s=5.0)
+
+    # Retry the click until it lands, capped at 8 tries OR 30 seconds total
+    run_with_budget(lambda: click_and_verify("Save"), budget)
+
+``RetryBudget`` is bounded by attempts and / or a deadline — set either to
+``None`` to bound only by the other. :func:`backoff_delay` (pure, no jitter) and
+:meth:`RetryBudget.plan` give the delay schedule for inspection:
+
+.. code-block:: python
+
+    RetryBudget(jitter="none").plan(4)   # [0.1, 0.2, 0.4, 0.8]
+
+For deterministic tests inject ``uniform`` / ``clock`` / ``sleep``:
+
+.. code-block:: python
+
+    run_with_budget(flaky, budget, clock=fake_clock, sleep=fake_sleep,
+                    uniform=lambda lo, hi: lo)   # always the low bound
+
+Executor commands
+-----------------
+
+``AC_retry_delay`` (``attempt`` / ``base`` / ``max_delay`` / ``multiplier`` /
+``jitter`` → ``{delay}``) and ``AC_plan_retry_delays`` (``attempts`` … →
+``{delays}``) expose the pure backoff schedule (``jitter`` defaults to ``none``
+for a deterministic result). They are the matching read-only ``ac_*`` MCP tools
+and Script Builder commands under **Flow**. :func:`run_with_budget` (which wraps
+a callable) is the Python-API surface.

docs/source/Zh/doc/new_features/v209_features_doc.rst

@@ -0,0 +1,49 @@
+重試預算——截止時間 + 抖動
+==========================
+
+:class:`resilience.RetryPolicy` 以固定次數搭配單純指數退避重試。有兩件它無法表達的事,正是不穩定、
+高競爭的 UI 自動化所需:
+
+* **掛鐘截止時間**——「持續重試,但總共超過 30 秒就放棄」,與嘗試了幾次無關;以及
+* **抖動(jitter)**——隨機化退避,讓眾多重試中的工作者不會重新同步成驚群效應。
+
+``retry_budget`` 兩者皆補上。:class:`RetryBudget` 由 ``max_attempts`` *與 / 或* ``deadline_s``
+界定;:func:`run_with_budget` 以先達到者為準,且絕不會睡過截止時間。延遲採用有上限的指數退避,
+搭配可選的抖動策略(``full`` / ``equal`` / ``none``)。隨機來源(``uniform``)、時鐘與睡眠器
+皆可注入,故每個延遲與決策在測試中都是確定的。不匯入 ``PySide6``。
+
+無頭 API
+--------
+
+.. code-block:: python
+
+    from je_auto_control import RetryBudget, run_with_budget
+
+    budget = RetryBudget(max_attempts=8, deadline_s=30.0,
+                         base_delay_s=0.2, max_delay_s=5.0)
+
+    # 重試點擊直到成功,上限為 8 次嘗試 或 總共 30 秒
+    run_with_budget(lambda: click_and_verify("Save"), budget)
+
+``RetryBudget`` 由嘗試次數與 / 或截止時間界定——把其一設為 ``None`` 即只以另一者界定。
+:func:`backoff_delay`(純函式,無抖動)與 :meth:`RetryBudget.plan` 提供延遲排程以供檢視:
+
+.. code-block:: python
+
+    RetryBudget(jitter="none").plan(4)   # [0.1, 0.2, 0.4, 0.8]
+
+確定性測試可注入 ``uniform`` / ``clock`` / ``sleep``:
+
+.. code-block:: python
+
+    run_with_budget(flaky, budget, clock=fake_clock, sleep=fake_sleep,
+                    uniform=lambda lo, hi: lo)   # 永遠取下界
+
+執行器指令
+----------
+
+``AC_retry_delay``(``attempt`` / ``base`` / ``max_delay`` / ``multiplier`` /
+``jitter`` → ``{delay}``)與 ``AC_plan_retry_delays``(``attempts`` … →
+``{delays}``)暴露純退避排程(``jitter`` 預設為 ``none`` 以得確定結果)。皆以對應的唯讀
+``ac_*`` MCP 工具及 Script Builder 指令(位於 **Flow** 分類下)形式提供。
+:func:`run_with_budget`(包裹一個 callable)則是 Python API 介面。

je_auto_control/__init__.py

@@ -115,6 +115,10 @@
     decode_conversion_mode, ime_state, is_composing,
     wait_for_composition_commit,
 )
+# Retry budget — deadline + jitter retries over a callable
+from je_auto_control.utils.retry_budget import (
+    RetryBudget, backoff_delay, jittered_delay, run_with_budget,
+)
 # Rich clipboard formats — RTF + CSV/TSV codecs and Windows get / set
 from je_auto_control.utils.clipboard_rich_formats import (
     build_rtf, csv_to_rows, get_clipboard_csv, get_clipboard_rtf, rows_to_csv,
@@ -1734,6 +1738,7 @@ def start_autocontrol_gui(*args, **kwargs):
     "wait_for_lock", "classify_lock_transitions",
     "ime_state", "is_composing", "wait_for_composition_commit",
     "decode_conversion_mode",
+    "RetryBudget", "run_with_budget", "backoff_delay", "jittered_delay",
     "build_rtf", "rtf_to_text", "rows_to_csv", "csv_to_rows",
     "set_clipboard_rtf", "get_clipboard_rtf",
     "set_clipboard_csv", "get_clipboard_csv",

je_auto_control/gui/script_builder/command_schema.py

@@ -4401,6 +4401,36 @@ def _add_work_queue_specs(specs: List[CommandSpec]) -> None:
         ),
         description="Decode an IMM32 conversion bitmask into named flags.",
     ))
+    specs.append(CommandSpec(
+        "AC_retry_delay", "Flow", "Retry Backoff Delay",
+        fields=(
+            FieldSpec("attempt", FieldType.INT, default=1,
+                      placeholder="1-based retry attempt"),
+            FieldSpec("base", FieldType.FLOAT, optional=True, default=0.1),
+            FieldSpec("max_delay", FieldType.FLOAT, optional=True,
+                      default=5.0),
+            FieldSpec("multiplier", FieldType.FLOAT, optional=True,
+                      default=2.0),
+            FieldSpec("jitter", FieldType.STRING, optional=True,
+                      default="none", placeholder="none / full / equal"),
+        ),
+        description="Capped exponential backoff delay before a retry attempt.",
+    ))
+    specs.append(CommandSpec(
+        "AC_plan_retry_delays", "Flow", "Plan Retry Delays",
+        fields=(
+            FieldSpec("attempts", FieldType.INT, default=5,
+                      placeholder="number of retries"),
+            FieldSpec("base", FieldType.FLOAT, optional=True, default=0.1),
+            FieldSpec("max_delay", FieldType.FLOAT, optional=True,
+                      default=5.0),
+            FieldSpec("multiplier", FieldType.FLOAT, optional=True,
+                      default=2.0),
+            FieldSpec("jitter", FieldType.STRING, optional=True,
+                      default="none", placeholder="none / full / equal"),
+        ),
+        description="The backoff delay schedule for the first N retries.",
+    ))
     specs.append(CommandSpec(
         "AC_normalize_ext", "Shell", "Normalize Extension",
         fields=(

je_auto_control/utils/executor/action_executor.py

@@ -2738,6 +2738,29 @@ def _decode_conversion_mode(flags: Any) -> Dict[str, Any]:
     return decode_conversion_mode(int(flags))
 
 
+def _make_retry_budget(base: Any, max_delay: Any, multiplier: Any,
+                       jitter: Any) -> Any:
+    """Build a RetryBudget from executor scalars (helper for the adapters)."""
+    from je_auto_control.utils.retry_budget import RetryBudget
+    return RetryBudget(base_delay_s=float(base), max_delay_s=float(max_delay),
+                       multiplier=float(multiplier), jitter=str(jitter))
+
+
+def _retry_delay(attempt: Any, base: Any = 0.1, max_delay: Any = 5.0,
+                 multiplier: Any = 2.0, jitter: Any = "none") -> Dict[str, Any]:
+    """Adapter: the (jittered) backoff delay before a retry attempt (pure)."""
+    budget = _make_retry_budget(base, max_delay, multiplier, jitter)
+    return {"delay": float(budget.next_delay(int(attempt)))}
+
+
+def _plan_retry_delays(attempts: Any, base: Any = 0.1, max_delay: Any = 5.0,
+                       multiplier: Any = 2.0, jitter: Any = "none"
+                       ) -> Dict[str, Any]:
+    """Adapter: the backoff delay schedule for the first N retries (pure)."""
+    budget = _make_retry_budget(base, max_delay, multiplier, jitter)
+    return {"delays": [float(d) for d in budget.plan(int(attempts))]}
+
+
 def _normalize_ext(target: str) -> Dict[str, Any]:
     """Adapter: the lowercased extension of a path / bare ext (pure)."""
     from je_auto_control.utils.file_assoc import normalize_ext
@@ -6760,6 +6783,8 @@ def __init__(self):
             "AC_is_composing": _is_composing,
             "AC_wait_for_composition_commit": _wait_for_composition_commit,
             "AC_decode_conversion_mode": _decode_conversion_mode,
+            "AC_retry_delay": _retry_delay,
+            "AC_plan_retry_delays": _plan_retry_delays,
             "AC_normalize_ext": _normalize_ext,
             "AC_file_association": _file_association,
             "AC_get_control_text": _get_control_text,

je_auto_control/utils/mcp_server/tools/_factories.py

@@ -1769,6 +1769,34 @@ def smart_wait_tools() -> List[MCPTool]:
             handler=h.wait_for_process,
             annotations=READ_ONLY,
         ),
+        MCPTool(
+            name="ac_retry_delay",
+            description=("Capped exponential backoff delay (seconds) before a "
+                         "given 1-based retry 'attempt'. 'jitter' is none / "
+                         "full / equal (default none). Returns {delay}."),
+            input_schema=schema({"attempt": {"type": "integer"},
+                                 "base": {"type": "number"},
+                                 "max_delay": {"type": "number"},
+                                 "multiplier": {"type": "number"},
+                                 "jitter": {"type": "string"}},
+                                required=["attempt"]),
+            handler=h.retry_delay,
+            annotations=READ_ONLY,
+        ),
+        MCPTool(
+            name="ac_plan_retry_delays",
+            description=("The backoff delay schedule (seconds) for the first "
+                         "'attempts' retries. 'jitter' none / full / equal "
+                         "(default none). Returns {delays}."),
+            input_schema=schema({"attempts": {"type": "integer"},
+                                 "base": {"type": "number"},
+                                 "max_delay": {"type": "number"},
+                                 "multiplier": {"type": "number"},
+                                 "jitter": {"type": "string"}},
+                                required=["attempts"]),
+            handler=h.plan_retry_delays,
+            annotations=READ_ONLY,
+        ),
     ]
 
 

je_auto_control/utils/mcp_server/tools/_handlers.py

@@ -668,6 +668,20 @@ def decode_conversion_mode(flags):
     return _decode_conversion_mode(flags)
 
 
+def retry_delay(attempt, base=0.1, max_delay=5.0, multiplier=2.0,
+                jitter="none"):
+    from je_auto_control.utils.executor.action_executor import _retry_delay
+    return _retry_delay(attempt, base, max_delay, multiplier, jitter)
+
+
+def plan_retry_delays(attempts, base=0.1, max_delay=5.0, multiplier=2.0,
+                      jitter="none"):
+    from je_auto_control.utils.executor.action_executor import (
+        _plan_retry_delays,
+    )
+    return _plan_retry_delays(attempts, base, max_delay, multiplier, jitter)
+
+
 def normalize_ext(target):
     from je_auto_control.utils.executor.action_executor import _normalize_ext
     return _normalize_ext(target)

je_auto_control/utils/retry_budget/__init__.py

@@ -0,0 +1,10 @@
+"""Retry budget: bound retries by a wall-clock deadline and full jitter."""
+from je_auto_control.utils.retry_budget.retry_budget import (
+    JITTER_EQUAL, JITTER_FULL, JITTER_NONE, RetryBudget, backoff_delay,
+    jittered_delay, run_with_budget,
+)
+
+__all__ = [
+    "RetryBudget", "run_with_budget", "backoff_delay", "jittered_delay",
+    "JITTER_FULL", "JITTER_EQUAL", "JITTER_NONE",
+]