Skip to content

Commit 59767cf

Browse files
authored
Merge pull request #412 from Integration-Automation/feat/flake-cluster-batch
Add flake_cluster: cluster tests that flake together (co-failure Jaccard)
2 parents 20ec65f + 3fd8b68 commit 59767cf

11 files changed

Lines changed: 364 additions & 0 deletions

File tree

WHATS_NEW.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,11 @@
11
# What's New — AutoControl
22

3+
## What's new (2026-06-25) — Flaky-Test Co-Failure Clustering
4+
5+
Find the tests that flake *together* — and the shared root cause behind them. Full reference: [`docs/source/Eng/doc/new_features/v193_features_doc.rst`](docs/source/Eng/doc/new_features/v193_features_doc.rst).
6+
7+
- **`cofailure_pairs` / `failure_clusters`** (`AC_cofailure_pairs`, `AC_failure_clusters`): flaky tests are rarely independent — a wobbly fixture or noisy dependency makes a *group* fail in the same runs (~75% of flaky tests cluster). Ranking tests one-by-one by flip rate misses that. This measures how often each pair of tests fails in the *same* runs (Jaccard over their failing-run sets) and groups tests above a threshold into connected clusters with a cohesion score — so you chase one root cause instead of N symptoms. Input is a list of runs, each the test names that failed in it. Pure stdlib. No `PySide6`.
8+
39
## What's new (2026-06-25) — Run-Trace Diff (what changed between two executions)
410

511
See exactly what changed between a passing run and a failing one. Full reference: [`docs/source/Eng/doc/new_features/v192_features_doc.rst`](docs/source/Eng/doc/new_features/v192_features_doc.rst).
Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
Flaky-Test Co-Failure Clustering
2+
================================
3+
4+
Flaky tests are rarely independent: a wobbly shared fixture, a slow dependency or
5+
a noisy environment makes a *group* of tests fail in the same runs (research finds
6+
~75% of flaky tests fall into co-failure clusters). Ranking tests one-by-one by
7+
flip rate misses that shared root cause. ``flake_cluster`` measures how often each
8+
pair of tests fails in the *same* runs — Jaccard similarity over the set of runs
9+
each failed in — and groups tests whose co-failure exceeds a threshold, so you can
10+
chase one root cause instead of N symptoms.
11+
12+
* :func:`cofailure_pairs` — test pairs that fail together above a threshold,
13+
* :func:`failure_clusters` — connected clusters of co-failing tests with a
14+
cohesion score (mean pairwise Jaccard).
15+
16+
Input is a list of runs, each a collection of the test names that failed in that
17+
run. Pure standard library; no device, no ``PySide6``.
18+
19+
Headless API
20+
------------
21+
22+
.. code-block:: python
23+
24+
from je_auto_control import failure_clusters, cofailure_pairs
25+
26+
runs = [["test_a", "test_b"], # both failed in this run
27+
["test_a", "test_b"],
28+
["test_c"],
29+
["test_a", "test_b", "test_c"]]
30+
31+
failure_clusters(runs, threshold=0.6)
32+
# [{"tests": ["test_a", "test_b"], "size": 2, "cohesion": 1.0}]
33+
34+
cofailure_pairs(runs, threshold=0.6)
35+
# [{"tests": ["test_a", "test_b"], "jaccard": 1.0, "co_failures": 3}]
36+
37+
``threshold`` is the minimum co-failure Jaccard to link two tests; ``min_size``
38+
(default ``2``) drops singletons so only genuine clusters surface. Clusters come
39+
back largest / most cohesive first.
40+
41+
Executor commands
42+
-----------------
43+
44+
``AC_failure_clusters`` (``runs`` / ``threshold`` / ``min_size``) and
45+
``AC_cofailure_pairs`` (``runs`` / ``threshold``). They are exposed as read-only
46+
``ac_*`` MCP tools and as Script Builder commands under **Testing**.
Lines changed: 41 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,41 @@
1+
不穩定測試的共同失敗分群
2+
========================
3+
4+
不穩定(flaky)測試很少是獨立的:搖晃的共用 fixture、緩慢的相依、或吵雜的環境,會讓*一群*測試在
5+
相同的執行中一起失敗(研究發現約 75% 的 flaky 測試落在共同失敗的群集裡)。逐一以翻轉率排名測試
6+
會錯過這個共同根因。``flake_cluster`` 量測每對測試多常在*相同*執行中失敗——即各自失敗的執行集合
7+
之間的 Jaccard 相似度——並把共同失敗超過門檻的測試分群,讓你能追一個根因,而非 N 個症狀。
8+
9+
* :func:`cofailure_pairs` ——共同失敗超過門檻的測試對,
10+
* :func:`failure_clusters` ——共同失敗測試的連通群集,附凝聚度分數(群內平均成對 Jaccard)。
11+
12+
輸入是一份執行清單,每個元素為該次執行中失敗的測試名稱集合。純標準庫;不涉及裝置,不匯入
13+
``PySide6``。
14+
15+
無頭 API
16+
--------
17+
18+
.. code-block:: python
19+
20+
from je_auto_control import failure_clusters, cofailure_pairs
21+
22+
runs = [["test_a", "test_b"], # 這次執行兩者皆失敗
23+
["test_a", "test_b"],
24+
["test_c"],
25+
["test_a", "test_b", "test_c"]]
26+
27+
failure_clusters(runs, threshold=0.6)
28+
# [{"tests": ["test_a", "test_b"], "size": 2, "cohesion": 1.0}]
29+
30+
cofailure_pairs(runs, threshold=0.6)
31+
# [{"tests": ["test_a", "test_b"], "jaccard": 1.0, "co_failures": 3}]
32+
33+
``threshold`` 是連結兩測試所需的最小共同失敗 Jaccard;``min_size``(預設 ``2``)會丟棄單例,
34+
讓只有真正的群集浮現。群集以最大 / 最凝聚者在前回傳。
35+
36+
執行器指令
37+
----------
38+
39+
``AC_failure_clusters``(``runs`` / ``threshold`` / ``min_size``)與
40+
``AC_cofailure_pairs``(``runs`` / ``threshold``)。皆以唯讀 ``ac_*`` MCP 工具及 Script Builder
41+
指令(位於 **Testing** 分類下)形式提供。

je_auto_control/__init__.py

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -94,6 +94,8 @@
9494
)
9595
# Run-trace diff (LCS-aligned: added/removed steps, status flips, regressions)
9696
from je_auto_control.utils.run_diff import diff_runs, summarize_run_diff
97+
# Flaky-test co-failure clustering (Jaccard over shared failing runs)
98+
from je_auto_control.utils.flake_cluster import cofailure_pairs, failure_clusters
9799
# VLM element locator (headless)
98100
from je_auto_control.utils.vision import (
99101
VLMNotAvailableError, click_by_description, locate_by_description,
@@ -1673,6 +1675,7 @@ def start_autocontrol_gui(*args, **kwargs):
16731675
"saliency_map", "salient_regions", "most_salient",
16741676
"normalize_error", "failure_signature", "group_failures",
16751677
"diff_runs", "summarize_run_diff",
1678+
"cofailure_pairs", "failure_clusters",
16761679
# VLM locator
16771680
"VLMNotAvailableError", "locate_by_description", "click_by_description",
16781681
"verify_description",

je_auto_control/gui/script_builder/command_schema.py

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2736,6 +2736,25 @@ def _add_audit_specs(specs: List[CommandSpec]) -> None:
27362736
),
27372737
description="LCS-align two run step-traces: added/removed/flips/regress.",
27382738
))
2739+
specs.append(CommandSpec(
2740+
"AC_failure_clusters", "Testing", "Cluster Co-Failing Tests",
2741+
fields=(
2742+
FieldSpec("runs", FieldType.STRING,
2743+
placeholder='[["test_a", "test_b"], ["test_a", "test_b"]]'),
2744+
FieldSpec("threshold", FieldType.FLOAT, optional=True, default=0.5),
2745+
FieldSpec("min_size", FieldType.INT, optional=True, default=2),
2746+
),
2747+
description="Cluster tests that flake together (co-failure Jaccard).",
2748+
))
2749+
specs.append(CommandSpec(
2750+
"AC_cofailure_pairs", "Testing", "Co-Failing Test Pairs",
2751+
fields=(
2752+
FieldSpec("runs", FieldType.STRING,
2753+
placeholder='[["test_a", "test_b"]]'),
2754+
FieldSpec("threshold", FieldType.FLOAT, optional=True, default=0.5),
2755+
),
2756+
description="Test pairs that fail together above a Jaccard threshold.",
2757+
))
27392758
specs.append(CommandSpec(
27402759
"AC_scan_secrets", "Tools", "Scan for Hardcoded Secrets",
27412760
description="Scan 'data' (JSON view) for hardcoded secrets that "

je_auto_control/utils/executor/action_executor.py

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4380,6 +4380,28 @@ def _diff_runs(before: Any, after: Any, key: str = "name",
43804380
return {**diff, "summary": summarize_run_diff(diff)}
43814381

43824382

4383+
def _failure_clusters(runs: Any, threshold: Any = 0.5,
4384+
min_size: Any = 2) -> Dict[str, Any]:
4385+
"""Adapter: cluster tests that fail together (co-failure Jaccard)."""
4386+
import json
4387+
from je_auto_control.utils.flake_cluster import failure_clusters
4388+
if isinstance(runs, str):
4389+
runs = json.loads(runs)
4390+
clusters = failure_clusters(runs, threshold=float(threshold),
4391+
min_size=int(min_size))
4392+
return {"clusters": clusters, "count": len(clusters)}
4393+
4394+
4395+
def _cofailure_pairs(runs: Any, threshold: Any = 0.5) -> Dict[str, Any]:
4396+
"""Adapter: test pairs that fail together above a Jaccard threshold."""
4397+
import json
4398+
from je_auto_control.utils.flake_cluster import cofailure_pairs
4399+
if isinstance(runs, str):
4400+
runs = json.loads(runs)
4401+
pairs = cofailure_pairs(runs, threshold=float(threshold))
4402+
return {"pairs": pairs, "count": len(pairs)}
4403+
4404+
43834405
def _image_histogram(source: Any = None, bins: Any = 32, space: str = "hsv",
43844406
region: Any = None) -> Dict[str, Any]:
43854407
"""Adapter: per-channel colour histogram of an image / the screen."""
@@ -6611,6 +6633,8 @@ def __init__(self):
66116633
"AC_failure_signature": _failure_signature,
66126634
"AC_group_failures": _group_failures,
66136635
"AC_diff_runs": _diff_runs,
6636+
"AC_failure_clusters": _failure_clusters,
6637+
"AC_cofailure_pairs": _cofailure_pairs,
66146638
"AC_image_histogram": _image_histogram,
66156639
"AC_histogram_changed": _histogram_changed,
66166640
"AC_changed_regions": _changed_regions,
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
"""Cluster tests that flake together by co-failure Jaccard similarity."""
2+
from je_auto_control.utils.flake_cluster.flake_cluster import (
3+
cofailure_pairs, failure_clusters,
4+
)
5+
6+
__all__ = ["cofailure_pairs", "failure_clusters"]
Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
"""Cluster tests that flake *together* by co-failure similarity.
2+
3+
Flaky tests are rarely independent: a wobbly shared fixture, a slow dependency or
4+
a noisy environment makes a *group* of tests fail in the same runs (research finds
5+
~75% of flaky tests fall into co-failure clusters). Ranking tests one-by-one by
6+
flip rate misses that shared root cause. ``flake_cluster`` measures how often each
7+
pair of tests fails in the *same* runs (Jaccard similarity over the set of runs
8+
each failed in) and groups tests whose co-failure exceeds a threshold into
9+
clusters — so you can chase one root cause instead of N symptoms.
10+
11+
Input is a list of runs, each a collection of the test names that failed in that
12+
run. Pure standard library; no device, no ``PySide6``.
13+
"""
14+
from itertools import combinations
15+
from typing import Any, Dict, List, Sequence, Set
16+
17+
18+
def _set_jaccard(left: Set[int], right: Set[int]) -> float:
19+
union = left | right
20+
return len(left & right) / len(union) if union else 0.0
21+
22+
23+
def _fail_runs(runs: Sequence[Sequence[str]]) -> Dict[str, Set[int]]:
24+
"""Map each test name to the set of run indices in which it failed."""
25+
fails: Dict[str, Set[int]] = {}
26+
for index, run in enumerate(runs):
27+
for test in set(run):
28+
fails.setdefault(str(test), set()).add(index)
29+
return fails
30+
31+
32+
def cofailure_pairs(runs: Sequence[Sequence[str]], *,
33+
threshold: float = 0.5) -> List[Dict[str, Any]]:
34+
"""Return test pairs whose co-failure Jaccard meets ``threshold``.
35+
36+
Each entry is ``{tests:[a,b], jaccard, co_failures}``, most similar first.
37+
"""
38+
fails = _fail_runs(runs)
39+
pairs = []
40+
for left, right in combinations(sorted(fails), 2):
41+
score = _set_jaccard(fails[left], fails[right])
42+
if score >= float(threshold):
43+
pairs.append({"tests": [left, right], "jaccard": round(score, 3),
44+
"co_failures": len(fails[left] & fails[right])})
45+
pairs.sort(key=lambda pair: pair["jaccard"], reverse=True)
46+
return pairs
47+
48+
49+
def _connected_components(nodes: Sequence[str],
50+
adjacency: Dict[str, Set[str]]) -> List[List[str]]:
51+
seen: Set[str] = set()
52+
components = []
53+
for node in nodes:
54+
if node in seen:
55+
continue
56+
stack, component = [node], []
57+
while stack:
58+
current = stack.pop()
59+
if current in seen:
60+
continue
61+
seen.add(current)
62+
component.append(current)
63+
stack.extend(adjacency[current] - seen)
64+
components.append(component)
65+
return components
66+
67+
68+
def _cohesion(component: Sequence[str], fails: Dict[str, Set[int]]) -> float:
69+
scores = [_set_jaccard(fails[a], fails[b])
70+
for a, b in combinations(component, 2)]
71+
return round(sum(scores) / len(scores), 3) if scores else 1.0
72+
73+
74+
def failure_clusters(runs: Sequence[Sequence[str]], *, threshold: float = 0.5,
75+
min_size: int = 2) -> List[Dict[str, Any]]:
76+
"""Group tests that fail together into co-failure clusters.
77+
78+
Builds a graph linking test pairs whose co-failure Jaccard meets
79+
``threshold``, then returns its connected components of at least ``min_size``
80+
tests as ``[{tests, size, cohesion}]`` (largest / most cohesive first).
81+
``cohesion`` is the mean pairwise Jaccard within the cluster.
82+
"""
83+
fails = _fail_runs(runs)
84+
tests = sorted(fails)
85+
adjacency: Dict[str, Set[str]] = {test: set() for test in tests}
86+
for left, right in combinations(tests, 2):
87+
if _set_jaccard(fails[left], fails[right]) >= float(threshold):
88+
adjacency[left].add(right)
89+
adjacency[right].add(left)
90+
clusters = []
91+
for component in _connected_components(tests, adjacency):
92+
if len(component) >= int(min_size):
93+
clusters.append({"tests": sorted(component), "size": len(component),
94+
"cohesion": _cohesion(component, fails)})
95+
clusters.sort(key=lambda cluster: (cluster["size"], cluster["cohesion"]),
96+
reverse=True)
97+
return clusters

je_auto_control/utils/mcp_server/tools/_factories.py

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -7691,6 +7691,35 @@ def flakiness_tools() -> List[MCPTool]:
76917691
handler=h.diff_runs,
76927692
annotations=READ_ONLY,
76937693
),
7694+
MCPTool(
7695+
name="ac_failure_clusters",
7696+
description=("Cluster tests that flake TOGETHER: 'runs' is a list of "
7697+
"runs, each the list of test names that failed in that "
7698+
"run. Groups tests whose co-failure Jaccard >= "
7699+
"'threshold'. Returns {clusters:[{tests,size,cohesion}], "
7700+
"count} — chase one root cause, not N symptoms."),
7701+
input_schema=schema({
7702+
"runs": {"type": "array",
7703+
"items": {"type": "array", "items": {"type": "string"}}},
7704+
"threshold": {"type": "number"},
7705+
"min_size": {"type": "integer"}},
7706+
required=["runs"]),
7707+
handler=h.failure_clusters,
7708+
annotations=READ_ONLY,
7709+
),
7710+
MCPTool(
7711+
name="ac_cofailure_pairs",
7712+
description=("Test pairs that fail together above a Jaccard "
7713+
"'threshold' over shared failing runs: "
7714+
"{pairs:[{tests,jaccard,co_failures}], count}."),
7715+
input_schema=schema({
7716+
"runs": {"type": "array",
7717+
"items": {"type": "array", "items": {"type": "string"}}},
7718+
"threshold": {"type": "number"}},
7719+
required=["runs"]),
7720+
handler=h.cofailure_pairs,
7721+
annotations=READ_ONLY,
7722+
),
76947723
]
76957724

76967725

je_auto_control/utils/mcp_server/tools/_handlers.py

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2558,6 +2558,16 @@ def diff_runs(before, after, key="name", regress_factor=1.5):
25582558
return _diff_runs(before, after, key, regress_factor)
25592559

25602560

2561+
def failure_clusters(runs, threshold=0.5, min_size=2):
2562+
from je_auto_control.utils.executor.action_executor import _failure_clusters
2563+
return _failure_clusters(runs, threshold, min_size)
2564+
2565+
2566+
def cofailure_pairs(runs, threshold=0.5):
2567+
from je_auto_control.utils.executor.action_executor import _cofailure_pairs
2568+
return _cofailure_pairs(runs, threshold)
2569+
2570+
25612571
def image_histogram(source=None, bins=32, space="hsv", region=None):
25622572
from je_auto_control.utils.executor.action_executor import _image_histogram
25632573
return _image_histogram(source, bins, space, region)

0 commit comments

Comments
 (0)