Skip to content

Commit 3cb667c

Browse files
committed
Add statistics and A/B significance testing
ab_locator ranks by raw success rate and run_history stores durations, but nothing computed percentiles or whether a difference is statistically significant. Add descriptive stats + percentiles, a two-proportion z-test with CI, Welch's t-test (exact t-distribution p-value via the incomplete beta, no SciPy), Cohen's d and a 2x2 chi-square. Normal CDF exact via erf; validated against textbook values. Wired through the facade, AC_describe_stats and AC_ab_significance executor commands, MCP tools and the Script Builder.
1 parent 6f8c26d commit 3cb667c

15 files changed

Lines changed: 529 additions & 1 deletion

File tree

README.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313

1414
## Table of Contents
1515

16+
- [What's new (2026-06-21) — Statistics & A/B Significance](#whats-new-2026-06-21--statistics--ab-significance)
1617
- [What's new (2026-06-21) — Full-Text Search (BM25)](#whats-new-2026-06-21--full-text-search-bm25)
1718
- [What's new (2026-06-21) — JSON Pointer, Patch & Merge Patch](#whats-new-2026-06-21--json-pointer-patch--merge-patch)
1819
- [What's new (2026-06-21) — Client-Side Rate Limiting](#whats-new-2026-06-21--client-side-rate-limiting)
@@ -117,6 +118,12 @@
117118

118119
---
119120

121+
## What's new (2026-06-21) — Statistics & A/B Significance
122+
123+
Decide whether a difference is real. Full reference: [`docs/source/Eng/doc/new_features/v65_features_doc.rst`](docs/source/Eng/doc/new_features/v65_features_doc.rst).
124+
125+
- **`describe` / `percentile` / `two_proportion_z_test` / `welch_t_test` / `cohens_d` / `chi_square_2x2`** (`AC_describe_stats`, `AC_ab_significance`): `ab_locator` ranks by raw success rate and `run_history` stores durations, but nothing computed percentiles or significance. This adds the analysis layer — summary stats + p50/p90/p95/p99, a two-proportion z-test (with CI), Welch's t-test (exact t-distribution p-value via the incomplete beta — no SciPy), Cohen's d, and a 2×2 chi-square. The normal CDF is exact via `math.erf`; validated against textbook values (incl. the chi²=z² identity). Pure-stdlib `math`+`statistics`.
126+
120127
## What's new (2026-06-21) — Full-Text Search (BM25)
121128

122129
Rank a document corpus by relevance. Full reference: [`docs/source/Eng/doc/new_features/v64_features_doc.rst`](docs/source/Eng/doc/new_features/v64_features_doc.rst).

README/README_zh-CN.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212

1313
## 目录
1414

15+
- [本次更新 (2026-06-21) — 统计与 A/B 显著性](#本次更新-2026-06-21--统计与-ab-显著性)
1516
- [本次更新 (2026-06-21) — 全文搜索(BM25)](#本次更新-2026-06-21--全文搜索bm25)
1617
- [本次更新 (2026-06-21) — JSON Pointer、Patch 与 Merge Patch](#本次更新-2026-06-21--json-pointerpatch-与-merge-patch)
1718
- [本次更新 (2026-06-21) — 客户端速率限制](#本次更新-2026-06-21--客户端速率限制)
@@ -116,6 +117,12 @@
116117

117118
---
118119

120+
## 本次更新 (2026-06-21) — 统计与 A/B 显著性
121+
122+
判断差异是否为真。完整参考:[`docs/source/Zh/doc/new_features/v65_features_doc.rst`](../docs/source/Zh/doc/new_features/v65_features_doc.rst)
123+
124+
- **`describe` / `percentile` / `two_proportion_z_test` / `welch_t_test` / `cohens_d` / `chi_square_2x2`**(`AC_describe_stats``AC_ab_significance`):`ab_locator` 以原始成功率排名,`run_history` 存储时长,但没有任何东西计算百分位或显著性。本功能补上分析层 —— 摘要统计 + p50/p90/p95/p99、双比例 z 检验(含置信区间)、Welch t 检验(以不完全 beta 取得精确 t 分布 p 值,免 SciPy)、Cohen's d,以及 2×2 卡方。正态 CDF 以 `math.erf` 精确计算;已对齐教科书数值(含 chi²=z² 恒等式)。纯标准库 `math`+`statistics`
125+
119126
## 本次更新 (2026-06-21) — 全文搜索(BM25)
120127

121128
依相关性对文档语料排名。完整参考:[`docs/source/Zh/doc/new_features/v64_features_doc.rst`](../docs/source/Zh/doc/new_features/v64_features_doc.rst)

README/README_zh-TW.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212

1313
## 目錄
1414

15+
- [本次更新 (2026-06-21) — 統計與 A/B 顯著性](#本次更新-2026-06-21--統計與-ab-顯著性)
1516
- [本次更新 (2026-06-21) — 全文搜尋(BM25)](#本次更新-2026-06-21--全文搜尋bm25)
1617
- [本次更新 (2026-06-21) — JSON Pointer、Patch 與 Merge Patch](#本次更新-2026-06-21--json-pointerpatch-與-merge-patch)
1718
- [本次更新 (2026-06-21) — 用戶端速率限制](#本次更新-2026-06-21--用戶端速率限制)
@@ -116,6 +117,12 @@
116117

117118
---
118119

120+
## 本次更新 (2026-06-21) — 統計與 A/B 顯著性
121+
122+
判斷差異是否為真。完整參考:[`docs/source/Zh/doc/new_features/v65_features_doc.rst`](../docs/source/Zh/doc/new_features/v65_features_doc.rst)
123+
124+
- **`describe` / `percentile` / `two_proportion_z_test` / `welch_t_test` / `cohens_d` / `chi_square_2x2`**(`AC_describe_stats``AC_ab_significance`):`ab_locator` 以原始成功率排名,`run_history` 儲存時長,但沒有任何東西計算百分位或顯著性。本功能補上分析層 —— 摘要統計 + p50/p90/p95/p99、雙比例 z 檢定(含信賴區間)、Welch t 檢定(以不完全 beta 取得精確 t 分布 p 值,免 SciPy)、Cohen's d,以及 2×2 卡方。常態 CDF 以 `math.erf` 精確計算;已對齊教科書數值(含 chi²=z² 恆等式)。純標準函式庫 `math`+`statistics`
125+
119126
## 本次更新 (2026-06-21) — 全文搜尋(BM25)
120127

121128
依相關性對文件語料排名。完整參考:[`docs/source/Zh/doc/new_features/v64_features_doc.rst`](../docs/source/Zh/doc/new_features/v64_features_doc.rst)
Lines changed: 51 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
Statistics & A/B Significance
2+
=============================
3+
4+
``ab_locator`` ranks strategies by raw success rate and ``run_history`` stores
5+
durations, but nothing computed percentiles or told you whether a difference is
6+
*statistically significant* rather than noise. This adds the analysis layer:
7+
summary statistics, a two-proportion z-test, Welch's t-test, Cohen's d, and a
8+
2x2 chi-square test.
9+
10+
The normal CDF is exact via ``math.erf``; the t-distribution p-value uses the
11+
regularized incomplete beta function, so results match reference
12+
implementations without SciPy. Pure standard library; imports no ``PySide6``.
13+
14+
Headless API
15+
------------
16+
17+
.. code-block:: python
18+
19+
from je_auto_control import (
20+
describe, percentile, two_proportion_z_test, welch_t_test, cohens_d)
21+
22+
describe([12.0, 9.5, 14.2, 11.1])
23+
# {"n": 4, "min": 9.5, "max": 14.2, "mean": 11.7, "stdev": ...,
24+
# "p50": ..., "p90": ..., "p95": ..., "p99": ...}
25+
26+
# Did variant B convert better than A? (90/200 vs 110/200)
27+
result = two_proportion_z_test(90, 200, 110, 200)
28+
# {"z": 2.0, "p_value": 0.0455, "significant": True,
29+
# "diff": 0.1, "ci_low": ..., "ci_high": ...}
30+
31+
# Continuous metric (e.g. latencies): is B different from A?
32+
welch_t_test(a_samples, b_samples) # {t, df, p_value, significant, ci}
33+
cohens_d(a_samples, b_samples) # effect size
34+
35+
``percentile`` supports linear interpolation (default) or nearest-rank;
36+
``describe`` adds p50/p90/p95/p99 to the usual moments. ``two_proportion_z_test``
37+
uses the pooled standard error for the test and the unpooled SE for the
38+
confidence interval (the textbook convention). ``welch_t_test`` reports the
39+
Welch–Satterthwaite degrees of freedom and an exact t-distribution p-value.
40+
``chi_square_2x2`` gives the df=1 chi-square (which equals the z-test's ````
41+
for the same table). These pair naturally with ``ab_locator`` counts and
42+
``run_history`` durations.
43+
44+
Executor commands
45+
-----------------
46+
47+
``AC_describe_stats`` takes ``values`` (a numeric list or JSON string) and
48+
returns the summary dict. ``AC_ab_significance`` takes ``a_conv`` / ``a_n`` /
49+
``b_conv`` / ``b_n`` and returns the two-proportion z-test result. Both are
50+
exposed as MCP tools (``ac_describe_stats`` / ``ac_ab_significance``) and as
51+
Script Builder commands under **Data**.

docs/source/Eng/eng_index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,7 @@ Comprehensive guides for all AutoControl features.
8787
doc/new_features/v62_features_doc
8888
doc/new_features/v63_features_doc
8989
doc/new_features/v64_features_doc
90+
doc/new_features/v65_features_doc
9091
doc/ocr_backends/ocr_backends_doc
9192
doc/observability/observability_doc
9293
doc/operations_layer/operations_layer_doc
Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
統計與 A/B 顯著性
2+
=================
3+
4+
``ab_locator`` 以原始成功率對策略排名,``run_history`` 儲存執行時長,但沒有任何東西計算百分位,
5+
也無法告訴你某個差異是*統計上顯著*還是雜訊。本功能補上分析層:摘要統計、雙比例 z 檢定、Welch
6+
t 檢定、Cohen's d,以及 2x2 卡方檢定。
7+
8+
常態 CDF 以 ``math.erf`` 精確計算;t 分布的 p 值使用正規化不完全 beta 函數,因此結果不需 SciPy
9+
即可對齊參考實作。純標準函式庫;不匯入 ``PySide6``。
10+
11+
無頭 API
12+
--------
13+
14+
.. code-block:: python
15+
16+
from je_auto_control import (
17+
describe, percentile, two_proportion_z_test, welch_t_test, cohens_d)
18+
19+
describe([12.0, 9.5, 14.2, 11.1])
20+
# {"n": 4, "min": 9.5, "max": 14.2, "mean": 11.7, "stdev": ...,
21+
# "p50": ..., "p90": ..., "p95": ..., "p99": ...}
22+
23+
# B 變體的轉換率有比 A 好嗎?(90/200 vs 110/200)
24+
result = two_proportion_z_test(90, 200, 110, 200)
25+
# {"z": 2.0, "p_value": 0.0455, "significant": True,
26+
# "diff": 0.1, "ci_low": ..., "ci_high": ...}
27+
28+
# 連續型指標(例如延遲):B 與 A 是否不同?
29+
welch_t_test(a_samples, b_samples) # {t, df, p_value, significant, ci}
30+
cohens_d(a_samples, b_samples) # 效果量
31+
32+
``percentile`` 支援線性內插(預設)或最近秩;``describe`` 在常見動差之外加上 p50/p90/p95/p99。
33+
``two_proportion_z_test`` 在檢定時使用合併標準誤、在信賴區間時使用未合併標準誤(教科書慣例)。
34+
``welch_t_test`` 回報 Welch–Satterthwaite 自由度與精確的 t 分布 p 值。``chi_square_2x2`` 給出
35+
df=1 的卡方(對同一張表等於 z 檢定的 ````)。這些與 ``ab_locator`` 的計數及 ``run_history`` 的
36+
時長自然搭配。
37+
38+
執行器命令
39+
----------
40+
41+
``AC_describe_stats`` 接受 ``values``(數值清單或 JSON 字串),回傳摘要字典。
42+
``AC_ab_significance`` 接受 ``a_conv`` / ``a_n`` / ``b_conv`` / ``b_n``,回傳雙比例 z 檢定結果。
43+
兩者皆以 MCP 工具(``ac_describe_stats`` / ``ac_ab_significance``)以及 Script Builder 中
44+
**Data** 分類下的命令提供。

docs/source/Zh/zh_index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -87,6 +87,7 @@ AutoControl 所有功能的完整使用指南。
8787
doc/new_features/v62_features_doc
8888
doc/new_features/v63_features_doc
8989
doc/new_features/v64_features_doc
90+
doc/new_features/v65_features_doc
9091
doc/ocr_backends/ocr_backends_doc
9192
doc/observability/observability_doc
9293
doc/operations_layer/operations_layer_doc

je_auto_control/__init__.py

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -333,6 +333,11 @@
333333
from je_auto_control.utils.search_index import (
334334
SearchHit, SearchIndex, search_documents, tokenize,
335335
)
336+
# Descriptive statistics and A/B significance testing
337+
from je_auto_control.utils.stats import (
338+
chi_square_2x2, cohens_d, describe, normal_cdf, percentile,
339+
two_proportion_z_test, welch_t_test,
340+
)
336341
# Background popup/interrupt watchdog (unattended automation)
337342
from je_auto_control.utils.watchdog import (
338343
PopupWatchdog, WatchdogRule, default_popup_watchdog,
@@ -816,6 +821,8 @@ def start_autocontrol_gui(*args, **kwargs):
816821
"make_patch", "merge_patch", "remove_pointer", "resolve_pointer",
817822
"set_pointer",
818823
"SearchHit", "SearchIndex", "search_documents", "tokenize",
824+
"chi_square_2x2", "cohens_d", "describe", "normal_cdf", "percentile",
825+
"two_proportion_z_test", "welch_t_test",
819826
# MCP server
820827
"AuditLogger", "HttpMCPServer", "MCPContent", "MCPPrompt",
821828
"MCPPromptArgument", "MCPResource", "MCPServer", "MCPTool",

je_auto_control/gui/script_builder/command_schema.py

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1155,6 +1155,24 @@ def _add_misc_specs(specs: List[CommandSpec]) -> None:
11551155
),
11561156
description="Validate JSON against a JSON Schema; returns {ok, errors}.",
11571157
))
1158+
specs.append(CommandSpec(
1159+
"AC_describe_stats", "Data", "Describe Statistics",
1160+
fields=(
1161+
FieldSpec("values", FieldType.STRING,
1162+
placeholder="[12.0, 9.5, 14.2, 11.1]"),
1163+
),
1164+
description="Summary stats + percentiles of a numeric list.",
1165+
))
1166+
specs.append(CommandSpec(
1167+
"AC_ab_significance", "Data", "A/B Significance (z-test)",
1168+
fields=(
1169+
FieldSpec("a_conv", FieldType.INT, placeholder="90"),
1170+
FieldSpec("a_n", FieldType.INT, placeholder="200"),
1171+
FieldSpec("b_conv", FieldType.INT, placeholder="110"),
1172+
FieldSpec("b_n", FieldType.INT, placeholder="200"),
1173+
),
1174+
description="Two-proportion z-test; returns {z, p_value, significant, ci}.",
1175+
))
11581176
specs.append(CommandSpec(
11591177
"AC_search_documents", "Data", "Full-Text Search (BM25)",
11601178
fields=(

je_auto_control/utils/executor/action_executor.py

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2928,6 +2928,22 @@ def _rate_limit(name: str, rate: float = 1.0, capacity: float = 1.0,
29282928
"wait": round(bucket.time_until_available(float(n)), 4)}
29292929

29302930

2931+
def _describe_stats(values: Any) -> Dict[str, Any]:
2932+
"""Adapter: summary statistics + percentiles of a numeric list (or JSON)."""
2933+
import json
2934+
from je_auto_control.utils.stats import describe
2935+
if isinstance(values, str):
2936+
values = json.loads(values)
2937+
return describe(values)
2938+
2939+
2940+
def _ab_significance(a_conv: int, a_n: int, b_conv: int,
2941+
b_n: int) -> Dict[str, Any]:
2942+
"""Adapter: two-proportion z-test on A/B conversion counts."""
2943+
from je_auto_control.utils.stats import two_proportion_z_test
2944+
return two_proportion_z_test(int(a_conv), int(a_n), int(b_conv), int(b_n))
2945+
2946+
29312947
def _search_documents(docs: Any, query: str, top_k: int = 10,
29322948
mode: str = "bm25") -> Dict[str, Any]:
29332949
"""Adapter: BM25/TF-IDF search a {doc_id: text} corpus (dict or JSON str)."""
@@ -3793,6 +3809,8 @@ def __init__(self):
37933809
"AC_jwt_decode": _jwt_decode,
37943810
"AC_rate_limit": _rate_limit,
37953811
"AC_search_documents": _search_documents,
3812+
"AC_describe_stats": _describe_stats,
3813+
"AC_ab_significance": _ab_significance,
37963814
"AC_resolve_pointer": _resolve_pointer,
37973815
"AC_apply_json_patch": _apply_json_patch,
37983816
"AC_make_json_patch": _make_json_patch,

0 commit comments

Comments
 (0)