|
| 1 | +Statistics & A/B Significance |
| 2 | +============================= |
| 3 | + |
| 4 | +``ab_locator`` ranks strategies by raw success rate and ``run_history`` stores |
| 5 | +durations, but nothing computed percentiles or told you whether a difference is |
| 6 | +*statistically significant* rather than noise. This adds the analysis layer: |
| 7 | +summary statistics, a two-proportion z-test, Welch's t-test, Cohen's d, and a |
| 8 | +2x2 chi-square test. |
| 9 | + |
| 10 | +The normal CDF is exact via ``math.erf``; the t-distribution p-value uses the |
| 11 | +regularized incomplete beta function, so results match reference |
| 12 | +implementations without SciPy. Pure standard library; imports no ``PySide6``. |
| 13 | + |
| 14 | +Headless API |
| 15 | +------------ |
| 16 | + |
| 17 | +.. code-block:: python |
| 18 | +
|
| 19 | + from je_auto_control import ( |
| 20 | + describe, percentile, two_proportion_z_test, welch_t_test, cohens_d) |
| 21 | +
|
| 22 | + describe([12.0, 9.5, 14.2, 11.1]) |
| 23 | + # {"n": 4, "min": 9.5, "max": 14.2, "mean": 11.7, "stdev": ..., |
| 24 | + # "p50": ..., "p90": ..., "p95": ..., "p99": ...} |
| 25 | +
|
| 26 | + # Did variant B convert better than A? (90/200 vs 110/200) |
| 27 | + result = two_proportion_z_test(90, 200, 110, 200) |
| 28 | + # {"z": 2.0, "p_value": 0.0455, "significant": True, |
| 29 | + # "diff": 0.1, "ci_low": ..., "ci_high": ...} |
| 30 | +
|
| 31 | + # Continuous metric (e.g. latencies): is B different from A? |
| 32 | + welch_t_test(a_samples, b_samples) # {t, df, p_value, significant, ci} |
| 33 | + cohens_d(a_samples, b_samples) # effect size |
| 34 | +
|
| 35 | +``percentile`` supports linear interpolation (default) or nearest-rank; |
| 36 | +``describe`` adds p50/p90/p95/p99 to the usual moments. ``two_proportion_z_test`` |
| 37 | +uses the pooled standard error for the test and the unpooled SE for the |
| 38 | +confidence interval (the textbook convention). ``welch_t_test`` reports the |
| 39 | +Welch–Satterthwaite degrees of freedom and an exact t-distribution p-value. |
| 40 | +``chi_square_2x2`` gives the df=1 chi-square (which equals the z-test's ``z²`` |
| 41 | +for the same table). These pair naturally with ``ab_locator`` counts and |
| 42 | +``run_history`` durations. |
| 43 | + |
| 44 | +Executor commands |
| 45 | +----------------- |
| 46 | + |
| 47 | +``AC_describe_stats`` takes ``values`` (a numeric list or JSON string) and |
| 48 | +returns the summary dict. ``AC_ab_significance`` takes ``a_conv`` / ``a_n`` / |
| 49 | +``b_conv`` / ``b_n`` and returns the two-proportion z-test result. Both are |
| 50 | +exposed as MCP tools (``ac_describe_stats`` / ``ac_ab_significance``) and as |
| 51 | +Script Builder commands under **Data**. |
0 commit comments