Skip to content

Commit d52b463

Browse files
committed
Add SLO evaluation with error budget and burn-rate alerts
The framework emitted raw signals but had no operational layer turning them into an SLO. Add evaluate_slo (SLI + error budget), burn_rate, and multi-window multi-burn-rate alerts (Google SRE workbook tiers) over outcome records. Records are plain data and the clock is injectable, so it is fully deterministic. Wired through the facade, AC_evaluate_slo/ AC_burn_alerts executor commands, MCP tools and the Script Builder.
1 parent fd0c7e0 commit d52b463

15 files changed

Lines changed: 402 additions & 0 deletions

File tree

README.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313

1414
## Table of Contents
1515

16+
- [What's new (2026-06-21) — Service-Level Objectives (SLO)](#whats-new-2026-06-21--service-level-objectives-slo)
1617
- [What's new (2026-06-21) — Chaos Experiments](#whats-new-2026-06-21--chaos-experiments)
1718
- [What's new (2026-06-21) — JSON Contract & Snapshot Matching](#whats-new-2026-06-21--json-contract--snapshot-matching)
1819
- [What's new (2026-06-21) — SLSA Build Provenance](#whats-new-2026-06-21--slsa-build-provenance)
@@ -124,6 +125,12 @@
124125

125126
---
126127

128+
## What's new (2026-06-21) — Service-Level Objectives (SLO)
129+
130+
SLI, error budget and burn-rate alerts. Full reference: [`docs/source/Eng/doc/new_features/v72_features_doc.rst`](docs/source/Eng/doc/new_features/v72_features_doc.rst).
131+
132+
- **`evaluate_slo` / `burn_rate` / `burn_alerts` / `default_burn_rules`** (`AC_evaluate_slo`, `AC_burn_alerts`): the framework emitted raw signals but had no SLO layer. This computes the SLI over outcome records (`[{timestamp, ok}]`), the error budget against a target, and the **multi-window multi-burn-rate** alerts from the Google SRE workbook (page 14.4×@1h, 6×@6h; ticket 1×@3d — firing only when both windows exceed the threshold). Records are plain data, clock injectable, fully deterministic. Pure-stdlib.
133+
127134
## What's new (2026-06-21) — Chaos Experiments
128135

129136
Inject faults, verify the system holds. Full reference: [`docs/source/Eng/doc/new_features/v71_features_doc.rst`](docs/source/Eng/doc/new_features/v71_features_doc.rst).

README/README_zh-CN.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212

1313
## 目录
1414

15+
- [本次更新 (2026-06-21) — 服务等级目标(SLO)](#本次更新-2026-06-21--服务等级目标slo)
1516
- [本次更新 (2026-06-21) — 混沌实验](#本次更新-2026-06-21--混沌实验)
1617
- [本次更新 (2026-06-21) — JSON 合约与快照比对](#本次更新-2026-06-21--json-合约与快照比对)
1718
- [本次更新 (2026-06-21) — SLSA 构建来源证明](#本次更新-2026-06-21--slsa-构建来源证明)
@@ -123,6 +124,12 @@
123124

124125
---
125126

127+
## 本次更新 (2026-06-21) — 服务等级目标(SLO)
128+
129+
SLI、错误预算与燃烧率告警。完整参考:[`docs/source/Zh/doc/new_features/v72_features_doc.rst`](../docs/source/Zh/doc/new_features/v72_features_doc.rst)
130+
131+
- **`evaluate_slo` / `burn_rate` / `burn_alerts` / `default_burn_rules`**(`AC_evaluate_slo``AC_burn_alerts`):框架会发出原始信号却没有 SLO 层。本功能在结果记录(`[{timestamp, ok}]`)上计算 SLI、对目标计算错误预算,以及 Google SRE workbook 的**多窗口多燃烧率**告警(1h 达 14.4×、6h 达 6× 呼叫;3d 达 1× 开单 —— 只有当长短窗口双双超过阈值才触发)。记录为纯数据、时钟可注入、完全确定。纯标准库。
132+
126133
## 本次更新 (2026-06-21) — 混沌实验
127134

128135
注入故障、验证系统仍成立。完整参考:[`docs/source/Zh/doc/new_features/v71_features_doc.rst`](../docs/source/Zh/doc/new_features/v71_features_doc.rst)

README/README_zh-TW.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212

1313
## 目錄
1414

15+
- [本次更新 (2026-06-21) — 服務等級目標(SLO)](#本次更新-2026-06-21--服務等級目標slo)
1516
- [本次更新 (2026-06-21) — 混沌實驗](#本次更新-2026-06-21--混沌實驗)
1617
- [本次更新 (2026-06-21) — JSON 合約與快照比對](#本次更新-2026-06-21--json-合約與快照比對)
1718
- [本次更新 (2026-06-21) — SLSA 建置來源證明](#本次更新-2026-06-21--slsa-建置來源證明)
@@ -123,6 +124,12 @@
123124

124125
---
125126

127+
## 本次更新 (2026-06-21) — 服務等級目標(SLO)
128+
129+
SLI、錯誤預算與燃燒率警示。完整參考:[`docs/source/Zh/doc/new_features/v72_features_doc.rst`](../docs/source/Zh/doc/new_features/v72_features_doc.rst)
130+
131+
- **`evaluate_slo` / `burn_rate` / `burn_alerts` / `default_burn_rules`**(`AC_evaluate_slo``AC_burn_alerts`):框架會發出原始訊號卻沒有 SLO 層。本功能在結果紀錄(`[{timestamp, ok}]`)上計算 SLI、對目標計算錯誤預算,以及 Google SRE workbook 的**多視窗多燃燒率**警示(1h 達 14.4×、6h 達 6× 呼叫;3d 達 1× 開票 —— 只有當長短視窗雙雙超過門檻才觸發)。紀錄為純資料、時鐘可注入、完全具決定性。純標準函式庫。
132+
126133
## 本次更新 (2026-06-21) — 混沌實驗
127134

128135
注入故障、驗證系統仍成立。完整參考:[`docs/source/Zh/doc/new_features/v71_features_doc.rst`](../docs/source/Zh/doc/new_features/v71_features_doc.rst)
Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
Service-Level Objectives (SLO)
2+
==============================
3+
4+
The framework emits raw signals (``observability`` metrics, ``run_history``
5+
durations) but had no operational layer turning them into an SLO, an error
6+
budget, or burn-rate alerts. This adds that: compute the SLI over a window of
7+
outcome records, the error budget against a target, and the **multi-window
8+
multi-burn-rate** alerts from the Google SRE workbook.
9+
10+
Records are plain data (``[{"timestamp": float, "ok": bool}, ...]``) so the
11+
whole thing is offline and deterministic; the clock is injectable. Pure
12+
standard library; imports no ``PySide6``.
13+
14+
Headless API
15+
------------
16+
17+
.. code-block:: python
18+
19+
from je_auto_control import evaluate_slo, burn_alerts
20+
21+
report = evaluate_slo(records, target=0.99)
22+
# {"sli": 0.995, "good": 995, "total": 1000, "target": 0.99,
23+
# "budget_total": 10.0, "budget_remaining": 5.0,
24+
# "budget_remaining_fraction": 0.5, "burn_rate": 0.5}
25+
26+
for alert in burn_alerts(records, target=0.99):
27+
page_oncall(alert) # severity, threshold, long/short burn rates
28+
29+
``evaluate_slo`` computes the SLI (good / total), the error budget
30+
(``(1 - target) * total`` events) and the burn rate (``bad_rate / (1 -
31+
target)`` — 1.0 means spending budget exactly on pace, > 1 means too fast).
32+
``burn_rate`` is the bare number over a window. ``burn_alerts`` evaluates the
33+
canonical Google SRE tiers from :func:`default_burn_rules` — page at 14.4× over
34+
1h (and 5m), page at 6× over 6h (and 30m), ticket at 1× over 3d (and 6h) — and
35+
fires a tier only when **both** its long and short windows exceed the
36+
threshold, which gives fast reset and few false positives. Supply your own
37+
``rules`` (``BurnRule`` list) to customise.
38+
39+
Executor commands
40+
-----------------
41+
42+
``AC_evaluate_slo`` takes ``records`` (a list or JSON string), a ``target`` and
43+
optional ``window_s``, and returns the SLI/budget report. ``AC_burn_alerts``
44+
returns ``{alerts, firing}``. Both are exposed as MCP tools
45+
(``ac_evaluate_slo`` / ``ac_burn_alerts``) and as Script Builder commands under
46+
**Report**.

docs/source/Eng/eng_index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,6 +94,7 @@ Comprehensive guides for all AutoControl features.
9494
doc/new_features/v69_features_doc
9595
doc/new_features/v70_features_doc
9696
doc/new_features/v71_features_doc
97+
doc/new_features/v72_features_doc
9798
doc/ocr_backends/ocr_backends_doc
9899
doc/observability/observability_doc
99100
doc/operations_layer/operations_layer_doc
Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
服務等級目標(SLO)
2+
==================
3+
4+
框架會發出原始訊號(``observability`` 指標、``run_history`` 時長),但沒有把它們轉成 SLO、錯誤
5+
預算或燃燒率警示的運維層。本功能補上:在一段視窗的結果紀錄上計算 SLI、對目標計算錯誤預算,以及
6+
Google SRE workbook 的**多視窗多燃燒率**警示。
7+
8+
紀錄是純資料(``[{"timestamp": float, "ok": bool}, ...]``),因此整體離線且具決定性;時鐘可注入。
9+
純標準函式庫;不匯入 ``PySide6``。
10+
11+
無頭 API
12+
--------
13+
14+
.. code-block:: python
15+
16+
from je_auto_control import evaluate_slo, burn_alerts
17+
18+
report = evaluate_slo(records, target=0.99)
19+
# {"sli": 0.995, "good": 995, "total": 1000, "target": 0.99,
20+
# "budget_total": 10.0, "budget_remaining": 5.0,
21+
# "budget_remaining_fraction": 0.5, "burn_rate": 0.5}
22+
23+
for alert in burn_alerts(records, target=0.99):
24+
page_oncall(alert) # severity、threshold、long/short 燃燒率
25+
26+
``evaluate_slo`` 計算 SLI(good / total)、錯誤預算(``(1 - target) * total`` 個事件)與燃燒率
27+
(``bad_rate / (1 - target)`` —— 1.0 代表剛好按進度消耗預算,> 1 代表太快)。``burn_rate`` 是某
28+
視窗的純數字。``burn_alerts`` 評估 :func:`default_burn_rules` 的標準 Google SRE 分層 —— 1h(與
29+
5m)達 14.4× 呼叫、6h(與 30m)達 6× 呼叫、3d(與 6h)達 1× 開票 —— 且只有當某層的長視窗與短視窗
30+
**雙雙**超過門檻時才觸發,以取得快速重置與少量誤報。可傳入自訂的 ``rules``(``BurnRule`` 清單)。
31+
32+
執行器命令
33+
----------
34+
35+
``AC_evaluate_slo`` 接受 ``records``(清單或 JSON 字串)、``target`` 與選用的 ``window_s``,回傳
36+
SLI/預算報告。``AC_burn_alerts`` 回傳 ``{alerts, firing}``。兩者皆以 MCP 工具(``ac_evaluate_slo``
37+
/ ``ac_burn_alerts``)以及 Script Builder 中 **Report** 分類下的命令提供。

docs/source/Zh/zh_index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,6 +94,7 @@ AutoControl 所有功能的完整使用指南。
9494
doc/new_features/v69_features_doc
9595
doc/new_features/v70_features_doc
9696
doc/new_features/v71_features_doc
97+
doc/new_features/v72_features_doc
9798
doc/ocr_backends/ocr_backends_doc
9899
doc/observability/observability_doc
99100
doc/operations_layer/operations_layer_doc

je_auto_control/__init__.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -365,6 +365,10 @@
365365
ChaosExperiment, Fault, Probe, exception_fault, latency_fault,
366366
run_experiment,
367367
)
368+
# SLO: SLI, error budget and multi-window burn-rate alerts
369+
from je_auto_control.utils.slo import (
370+
BurnRule, burn_alerts, burn_rate, default_burn_rules, evaluate_slo,
371+
)
368372
# Background popup/interrupt watchdog (unattended automation)
369373
from je_auto_control.utils.watchdog import (
370374
PopupWatchdog, WatchdogRule, default_popup_watchdog,
@@ -860,6 +864,7 @@ def start_autocontrol_gui(*args, **kwargs):
860864
"MatchReport", "diff_json", "match_json", "normalize_json", "snapshot_json",
861865
"ChaosExperiment", "Fault", "Probe", "exception_fault", "latency_fault",
862866
"run_experiment",
867+
"BurnRule", "burn_alerts", "burn_rate", "default_burn_rules", "evaluate_slo",
863868
# MCP server
864869
"AuditLogger", "HttpMCPServer", "MCPContent", "MCPPrompt",
865870
"MCPPromptArgument", "MCPResource", "MCPServer", "MCPTool",

je_auto_control/gui/script_builder/command_schema.py

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1388,6 +1388,25 @@ def _add_misc_specs(specs: List[CommandSpec]) -> None:
13881388
),
13891389
description="Verify a JWT (alg allowlist + exp/nbf/aud); returns {ok, claims}.",
13901390
))
1391+
specs.append(CommandSpec(
1392+
"AC_evaluate_slo", "Report", "SLO: Evaluate (SLI + Error Budget)",
1393+
fields=(
1394+
FieldSpec("records", FieldType.STRING,
1395+
placeholder='[{"timestamp": 1700000000, "ok": true}]'),
1396+
FieldSpec("target", FieldType.FLOAT, placeholder="0.99"),
1397+
FieldSpec("window_s", FieldType.FLOAT, optional=True),
1398+
),
1399+
description="SLI + error budget for outcome records vs a target.",
1400+
))
1401+
specs.append(CommandSpec(
1402+
"AC_burn_alerts", "Report", "SLO: Burn-Rate Alerts",
1403+
fields=(
1404+
FieldSpec("records", FieldType.STRING,
1405+
placeholder='[{"timestamp": 1700000000, "ok": false}]'),
1406+
FieldSpec("target", FieldType.FLOAT, placeholder="0.99"),
1407+
),
1408+
description="Multi-window burn-rate alerts (Google SRE tiers).",
1409+
))
13911410
specs.append(CommandSpec(
13921411
"AC_run_chaos", "Flow", "Run Chaos Experiment",
13931412
fields=(

je_auto_control/utils/executor/action_executor.py

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2928,6 +2928,26 @@ def _rate_limit(name: str, rate: float = 1.0, capacity: float = 1.0,
29282928
"wait": round(bucket.time_until_available(float(n)), 4)}
29292929

29302930

2931+
def _evaluate_slo(records: Any, target: float,
2932+
window_s: Optional[float] = None) -> Dict[str, Any]:
2933+
"""Adapter: SLI + error budget for outcome records (list or JSON string)."""
2934+
import json
2935+
from je_auto_control.utils.slo import evaluate_slo
2936+
if isinstance(records, str):
2937+
records = json.loads(records)
2938+
return evaluate_slo(records, float(target), window_s=window_s)
2939+
2940+
2941+
def _burn_alerts(records: Any, target: float) -> Dict[str, Any]:
2942+
"""Adapter: multi-window burn-rate alerts for outcome records."""
2943+
import json
2944+
from je_auto_control.utils.slo import burn_alerts
2945+
if isinstance(records, str):
2946+
records = json.loads(records)
2947+
alerts = burn_alerts(records, float(target))
2948+
return {"alerts": alerts, "firing": bool(alerts)}
2949+
2950+
29312951
def _chaos_probe_call(actions: List[Any]) -> Any:
29322952
def call() -> bool:
29332953
executor.execute_action(list(actions), raise_on_error=True)
@@ -3965,6 +3985,8 @@ def __init__(self):
39653985
"AC_match_json": _match_json,
39663986
"AC_diff_json": _diff_json,
39673987
"AC_run_chaos": _run_chaos,
3988+
"AC_evaluate_slo": _evaluate_slo,
3989+
"AC_burn_alerts": _burn_alerts,
39683990
"AC_unified_diff": _unified_diff,
39693991
"AC_apply_unified": _apply_unified,
39703992
"AC_three_way_merge": _three_way_merge,

0 commit comments

Comments
 (0)