Add statistics and A/B significance testing

README.md

@@ -13,6 +13,7 @@
 
 ## Table of Contents
 
+- [What's new (2026-06-21) — Statistics & A/B Significance](#whats-new-2026-06-21--statistics--ab-significance)
 - [What's new (2026-06-21) — Full-Text Search (BM25)](#whats-new-2026-06-21--full-text-search-bm25)
 - [What's new (2026-06-21) — JSON Pointer, Patch & Merge Patch](#whats-new-2026-06-21--json-pointer-patch--merge-patch)
 - [What's new (2026-06-21) — Client-Side Rate Limiting](#whats-new-2026-06-21--client-side-rate-limiting)
@@ -117,6 +118,12 @@
 
 ---
 
+## What's new (2026-06-21) — Statistics & A/B Significance
+
+Decide whether a difference is real. Full reference: [`docs/source/Eng/doc/new_features/v65_features_doc.rst`](docs/source/Eng/doc/new_features/v65_features_doc.rst).
+
+- **`describe` / `percentile` / `two_proportion_z_test` / `welch_t_test` / `cohens_d` / `chi_square_2x2`** (`AC_describe_stats`, `AC_ab_significance`): `ab_locator` ranks by raw success rate and `run_history` stores durations, but nothing computed percentiles or significance. This adds the analysis layer — summary stats + p50/p90/p95/p99, a two-proportion z-test (with CI), Welch's t-test (exact t-distribution p-value via the incomplete beta — no SciPy), Cohen's d, and a 2×2 chi-square. The normal CDF is exact via `math.erf`; validated against textbook values (incl. the chi²=z² identity). Pure-stdlib `math`+`statistics`.
+
 ## What's new (2026-06-21) — Full-Text Search (BM25)
 
 Rank a document corpus by relevance. Full reference: [`docs/source/Eng/doc/new_features/v64_features_doc.rst`](docs/source/Eng/doc/new_features/v64_features_doc.rst).

README/README_zh-CN.md

@@ -12,6 +12,7 @@
 
 ## 目录
 
+- [本次更新 (2026-06-21) — 统计与 A/B 显著性](#本次更新-2026-06-21--统计与-ab-显著性)
 - [本次更新 (2026-06-21) — 全文搜索(BM25)](#本次更新-2026-06-21--全文搜索bm25)
 - [本次更新 (2026-06-21) — JSON Pointer、Patch 与 Merge Patch](#本次更新-2026-06-21--json-pointerpatch-与-merge-patch)
 - [本次更新 (2026-06-21) — 客户端速率限制](#本次更新-2026-06-21--客户端速率限制)
@@ -116,6 +117,12 @@
 
 ---
 
+## 本次更新 (2026-06-21) — 统计与 A/B 显著性
+
+判断差异是否为真。完整参考:[`docs/source/Zh/doc/new_features/v65_features_doc.rst`](../docs/source/Zh/doc/new_features/v65_features_doc.rst)。
+
+- **`describe` / `percentile` / `two_proportion_z_test` / `welch_t_test` / `cohens_d` / `chi_square_2x2`**(`AC_describe_stats`、`AC_ab_significance`):`ab_locator` 以原始成功率排名,`run_history` 存储时长,但没有任何东西计算百分位或显著性。本功能补上分析层 —— 摘要统计 + p50/p90/p95/p99、双比例 z 检验(含置信区间)、Welch t 检验(以不完全 beta 取得精确 t 分布 p 值,免 SciPy)、Cohen's d,以及 2×2 卡方。正态 CDF 以 `math.erf` 精确计算;已对齐教科书数值(含 chi²=z² 恒等式)。纯标准库 `math`+`statistics`。
+
 ## 本次更新 (2026-06-21) — 全文搜索(BM25)
 
 依相关性对文档语料排名。完整参考:[`docs/source/Zh/doc/new_features/v64_features_doc.rst`](../docs/source/Zh/doc/new_features/v64_features_doc.rst)。

README/README_zh-TW.md

@@ -12,6 +12,7 @@
 
 ## 目錄
 
+- [本次更新 (2026-06-21) — 統計與 A/B 顯著性](#本次更新-2026-06-21--統計與-ab-顯著性)
 - [本次更新 (2026-06-21) — 全文搜尋(BM25)](#本次更新-2026-06-21--全文搜尋bm25)
 - [本次更新 (2026-06-21) — JSON Pointer、Patch 與 Merge Patch](#本次更新-2026-06-21--json-pointerpatch-與-merge-patch)
 - [本次更新 (2026-06-21) — 用戶端速率限制](#本次更新-2026-06-21--用戶端速率限制)
@@ -116,6 +117,12 @@
 
 ---
 
+## 本次更新 (2026-06-21) — 統計與 A/B 顯著性
+
+判斷差異是否為真。完整參考:[`docs/source/Zh/doc/new_features/v65_features_doc.rst`](../docs/source/Zh/doc/new_features/v65_features_doc.rst)。
+
+- **`describe` / `percentile` / `two_proportion_z_test` / `welch_t_test` / `cohens_d` / `chi_square_2x2`**(`AC_describe_stats`、`AC_ab_significance`):`ab_locator` 以原始成功率排名,`run_history` 儲存時長,但沒有任何東西計算百分位或顯著性。本功能補上分析層 —— 摘要統計 + p50/p90/p95/p99、雙比例 z 檢定(含信賴區間)、Welch t 檢定(以不完全 beta 取得精確 t 分布 p 值,免 SciPy)、Cohen's d,以及 2×2 卡方。常態 CDF 以 `math.erf` 精確計算;已對齊教科書數值(含 chi²=z² 恆等式)。純標準函式庫 `math`+`statistics`。
+
 ## 本次更新 (2026-06-21) — 全文搜尋(BM25)
 
 依相關性對文件語料排名。完整參考:[`docs/source/Zh/doc/new_features/v64_features_doc.rst`](../docs/source/Zh/doc/new_features/v64_features_doc.rst)。

docs/source/Eng/doc/new_features/v65_features_doc.rst

@@ -0,0 +1,51 @@
+Statistics & A/B Significance
+=============================
+
+``ab_locator`` ranks strategies by raw success rate and ``run_history`` stores
+durations, but nothing computed percentiles or told you whether a difference is
+*statistically significant* rather than noise. This adds the analysis layer:
+summary statistics, a two-proportion z-test, Welch's t-test, Cohen's d, and a
+2x2 chi-square test.
+
+The normal CDF is exact via ``math.erf``; the t-distribution p-value uses the
+regularized incomplete beta function, so results match reference
+implementations without SciPy. Pure standard library; imports no ``PySide6``.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import (
+        describe, percentile, two_proportion_z_test, welch_t_test, cohens_d)
+
+    describe([12.0, 9.5, 14.2, 11.1])
+    # {"n": 4, "min": 9.5, "max": 14.2, "mean": 11.7, "stdev": ...,
+    #  "p50": ..., "p90": ..., "p95": ..., "p99": ...}
+
+    # Did variant B convert better than A? (90/200 vs 110/200)
+    result = two_proportion_z_test(90, 200, 110, 200)
+    # {"z": 2.0, "p_value": 0.0455, "significant": True,
+    #  "diff": 0.1, "ci_low": ..., "ci_high": ...}
+
+    # Continuous metric (e.g. latencies): is B different from A?
+    welch_t_test(a_samples, b_samples)        # {t, df, p_value, significant, ci}
+    cohens_d(a_samples, b_samples)            # effect size
+
+``percentile`` supports linear interpolation (default) or nearest-rank;
+``describe`` adds p50/p90/p95/p99 to the usual moments. ``two_proportion_z_test``
+uses the pooled standard error for the test and the unpooled SE for the
+confidence interval (the textbook convention). ``welch_t_test`` reports the
+Welch–Satterthwaite degrees of freedom and an exact t-distribution p-value.
+``chi_square_2x2`` gives the df=1 chi-square (which equals the z-test's ``z²``
+for the same table). These pair naturally with ``ab_locator`` counts and
+``run_history`` durations.
+
+Executor commands
+-----------------
+
+``AC_describe_stats`` takes ``values`` (a numeric list or JSON string) and
+returns the summary dict. ``AC_ab_significance`` takes ``a_conv`` / ``a_n`` /
+``b_conv`` / ``b_n`` and returns the two-proportion z-test result. Both are
+exposed as MCP tools (``ac_describe_stats`` / ``ac_ab_significance``) and as
+Script Builder commands under **Data**.

docs/source/Eng/eng_index.rst

@@ -87,6 +87,7 @@ Comprehensive guides for all AutoControl features.
    doc/new_features/v62_features_doc
    doc/new_features/v63_features_doc
    doc/new_features/v64_features_doc
+   doc/new_features/v65_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

docs/source/Zh/doc/new_features/v65_features_doc.rst

@@ -0,0 +1,44 @@
+統計與 A/B 顯著性
+=================
+
+``ab_locator`` 以原始成功率對策略排名,``run_history`` 儲存執行時長,但沒有任何東西計算百分位,
+也無法告訴你某個差異是*統計上顯著*還是雜訊。本功能補上分析層:摘要統計、雙比例 z 檢定、Welch
+t 檢定、Cohen's d,以及 2x2 卡方檢定。
+
+常態 CDF 以 ``math.erf`` 精確計算;t 分布的 p 值使用正規化不完全 beta 函數,因此結果不需 SciPy
+即可對齊參考實作。純標準函式庫;不匯入 ``PySide6``。
+
+無頭 API
+--------
+
+.. code-block:: python
+
+    from je_auto_control import (
+        describe, percentile, two_proportion_z_test, welch_t_test, cohens_d)
+
+    describe([12.0, 9.5, 14.2, 11.1])
+    # {"n": 4, "min": 9.5, "max": 14.2, "mean": 11.7, "stdev": ...,
+    #  "p50": ..., "p90": ..., "p95": ..., "p99": ...}
+
+    # B 變體的轉換率有比 A 好嗎?(90/200 vs 110/200)
+    result = two_proportion_z_test(90, 200, 110, 200)
+    # {"z": 2.0, "p_value": 0.0455, "significant": True,
+    #  "diff": 0.1, "ci_low": ..., "ci_high": ...}
+
+    # 連續型指標(例如延遲):B 與 A 是否不同?
+    welch_t_test(a_samples, b_samples)        # {t, df, p_value, significant, ci}
+    cohens_d(a_samples, b_samples)            # 效果量
+
+``percentile`` 支援線性內插(預設)或最近秩;``describe`` 在常見動差之外加上 p50/p90/p95/p99。
+``two_proportion_z_test`` 在檢定時使用合併標準誤、在信賴區間時使用未合併標準誤(教科書慣例)。
+``welch_t_test`` 回報 Welch–Satterthwaite 自由度與精確的 t 分布 p 值。``chi_square_2x2`` 給出
+df=1 的卡方(對同一張表等於 z 檢定的 ``z²``)。這些與 ``ab_locator`` 的計數及 ``run_history`` 的
+時長自然搭配。
+
+執行器命令
+----------
+
+``AC_describe_stats`` 接受 ``values``(數值清單或 JSON 字串),回傳摘要字典。
+``AC_ab_significance`` 接受 ``a_conv`` / ``a_n`` / ``b_conv`` / ``b_n``,回傳雙比例 z 檢定結果。
+兩者皆以 MCP 工具(``ac_describe_stats`` / ``ac_ab_significance``)以及 Script Builder 中
+**Data** 分類下的命令提供。

docs/source/Zh/zh_index.rst

@@ -87,6 +87,7 @@ AutoControl 所有功能的完整使用指南。
    doc/new_features/v62_features_doc
    doc/new_features/v63_features_doc
    doc/new_features/v64_features_doc
+   doc/new_features/v65_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

je_auto_control/__init__.py

@@ -333,6 +333,11 @@
 from je_auto_control.utils.search_index import (
     SearchHit, SearchIndex, search_documents, tokenize,
 )
+# Descriptive statistics and A/B significance testing
+from je_auto_control.utils.stats import (
+    chi_square_2x2, cohens_d, describe, normal_cdf, percentile,
+    two_proportion_z_test, welch_t_test,
+)
 # Background popup/interrupt watchdog (unattended automation)
 from je_auto_control.utils.watchdog import (
     PopupWatchdog, WatchdogRule, default_popup_watchdog,
@@ -816,6 +821,8 @@ def start_autocontrol_gui(*args, **kwargs):
     "make_patch", "merge_patch", "remove_pointer", "resolve_pointer",
     "set_pointer",
     "SearchHit", "SearchIndex", "search_documents", "tokenize",
+    "chi_square_2x2", "cohens_d", "describe", "normal_cdf", "percentile",
+    "two_proportion_z_test", "welch_t_test",
     # MCP server
     "AuditLogger", "HttpMCPServer", "MCPContent", "MCPPrompt",
     "MCPPromptArgument", "MCPResource", "MCPServer", "MCPTool",

je_auto_control/gui/script_builder/command_schema.py

@@ -1155,6 +1155,24 @@ def _add_misc_specs(specs: List[CommandSpec]) -> None:
         ),
         description="Validate JSON against a JSON Schema; returns {ok, errors}.",
     ))
+    specs.append(CommandSpec(
+        "AC_describe_stats", "Data", "Describe Statistics",
+        fields=(
+            FieldSpec("values", FieldType.STRING,
+                      placeholder="[12.0, 9.5, 14.2, 11.1]"),
+        ),
+        description="Summary stats + percentiles of a numeric list.",
+    ))
+    specs.append(CommandSpec(
+        "AC_ab_significance", "Data", "A/B Significance (z-test)",
+        fields=(
+            FieldSpec("a_conv", FieldType.INT, placeholder="90"),
+            FieldSpec("a_n", FieldType.INT, placeholder="200"),
+            FieldSpec("b_conv", FieldType.INT, placeholder="110"),
+            FieldSpec("b_n", FieldType.INT, placeholder="200"),
+        ),
+        description="Two-proportion z-test; returns {z, p_value, significant, ci}.",
+    ))
     specs.append(CommandSpec(
         "AC_search_documents", "Data", "Full-Text Search (BM25)",
         fields=(

je_auto_control/utils/executor/action_executor.py

@@ -2928,6 +2928,22 @@ def _rate_limit(name: str, rate: float = 1.0, capacity: float = 1.0,
             "wait": round(bucket.time_until_available(float(n)), 4)}
 
 
+def _describe_stats(values: Any) -> Dict[str, Any]:
+    """Adapter: summary statistics + percentiles of a numeric list (or JSON)."""
+    import json
+    from je_auto_control.utils.stats import describe
+    if isinstance(values, str):
+        values = json.loads(values)
+    return describe(values)
+
+
+def _ab_significance(a_conv: int, a_n: int, b_conv: int,
+                     b_n: int) -> Dict[str, Any]:
+    """Adapter: two-proportion z-test on A/B conversion counts."""
+    from je_auto_control.utils.stats import two_proportion_z_test
+    return two_proportion_z_test(int(a_conv), int(a_n), int(b_conv), int(b_n))
+
+
 def _search_documents(docs: Any, query: str, top_k: int = 10,
                       mode: str = "bm25") -> Dict[str, Any]:
     """Adapter: BM25/TF-IDF search a {doc_id: text} corpus (dict or JSON str)."""
@@ -3793,6 +3809,8 @@ def __init__(self):
             "AC_jwt_decode": _jwt_decode,
             "AC_rate_limit": _rate_limit,
             "AC_search_documents": _search_documents,
+            "AC_describe_stats": _describe_stats,
+            "AC_ab_significance": _ab_significance,
             "AC_resolve_pointer": _resolve_pointer,
             "AC_apply_json_patch": _apply_json_patch,
             "AC_make_json_patch": _make_json_patch,

je_auto_control/utils/mcp_server/tools/_factories.py

@@ -3343,6 +3343,32 @@ def rate_limit_tools() -> List[MCPTool]:
     ]
 
 
+def stats_tools() -> List[MCPTool]:
+    return [
+        MCPTool(
+            name="ac_describe_stats",
+            description=("Summary statistics + percentiles (n/min/max/mean/"
+                         "stdev/variance/p50/p90/p95/p99) of a numeric "
+                         "'values' list."),
+            input_schema=schema({"values": {"type": "array"}}, ["values"]),
+            handler=h.describe_stats,
+            annotations=READ_ONLY,
+        ),
+        MCPTool(
+            name="ac_ab_significance",
+            description=("Two-proportion z-test on A/B conversion counts. "
+                         "Returns {z, p_value, significant, diff, ci_low, "
+                         "ci_high}."),
+            input_schema=schema(
+                {"a_conv": {"type": "integer"}, "a_n": {"type": "integer"},
+                 "b_conv": {"type": "integer"}, "b_n": {"type": "integer"}},
+                ["a_conv", "a_n", "b_conv", "b_n"]),
+            handler=h.ab_significance,
+            annotations=READ_ONLY,
+        ),
+    ]
+
+
 def search_index_tools() -> List[MCPTool]:
     return [
         MCPTool(
@@ -4601,7 +4627,7 @@ def media_assert_tools() -> List[MCPTool]:
     process_mining_tools, asset_tools, events_tools, notify_channel_tools,
     jsonpath_tools, json_schema_tools, vuln_scan_tools, vex_tools,
     license_policy_tools, jwt_tools, rate_limit_tools, json_patch_tools,
-    search_index_tools,
+    search_index_tools, stats_tools,
     saga_tools, decision_table_tools, locator_repair_tools,
     pii_text_tools, sarif_tools,
     screen_record_tools,

je_auto_control/utils/mcp_server/tools/_handlers.py

@@ -1637,6 +1637,16 @@ def search_documents(docs, query, top_k=10, mode="bm25"):
     return {"hits": [{"doc_id": h.doc_id, "score": h.score} for h in hits]}
 
 
+def describe_stats(values):
+    from je_auto_control.utils.stats import describe
+    return describe(values)
+
+
+def ab_significance(a_conv, a_n, b_conv, b_n):
+    from je_auto_control.utils.stats import two_proportion_z_test
+    return two_proportion_z_test(int(a_conv), int(a_n), int(b_conv), int(b_n))
+
+
 def run_saga(steps):
     from je_auto_control.utils.saga import run_saga as _run
     result = _run(steps)

je_auto_control/utils/stats/__init__.py

@@ -0,0 +1,10 @@
+"""Descriptive statistics and A/B significance testing (pure stdlib)."""
+from je_auto_control.utils.stats.stats import (
+    chi_square_2x2, cohens_d, describe, normal_cdf, percentile,
+    two_proportion_z_test, welch_t_test,
+)
+
+__all__ = [
+    "chi_square_2x2", "cohens_d", "describe", "normal_cdf", "percentile",
+    "two_proportion_z_test", "welch_t_test",
+]