Skip to content

Commit 481bbee

Browse files
authored
Merge pull request #307 from Integration-Automation/feat/text-similarity-batch
Add string-distance similarity metrics
2 parents 438decc + 8afa606 commit 481bbee

15 files changed

Lines changed: 389 additions & 1 deletion

File tree

README.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313

1414
## Table of Contents
1515

16+
- [What's new (2026-06-22) — String-Distance Similarity Metrics](#whats-new-2026-06-22--string-distance-similarity-metrics)
1617
- [What's new (2026-06-22) — Time-Series Transforms](#whats-new-2026-06-22--time-series-transforms)
1718
- [What's new (2026-06-22) — Unicode Text Normalisation & Slugify](#whats-new-2026-06-22--unicode-text-normalisation--slugify)
1819
- [What's new (2026-06-22) — JSON-Schema Compatibility Checking](#whats-new-2026-06-22--json-schema-compatibility-checking)
@@ -151,6 +152,12 @@
151152

152153
---
153154

155+
## What's new (2026-06-22) — String-Distance Similarity Metrics
156+
157+
Match typos and reordered tokens. Full reference: [`docs/source/Eng/doc/new_features/v99_features_doc.rst`](docs/source/Eng/doc/new_features/v99_features_doc.rst).
158+
159+
- **`levenshtein` / `damerau_levenshtein` / `jaro` / `jaro_winkler` / `jaccard` / `dice` / `similarity`** (`AC_text_similarity`): `fuzzy` exposed only difflib's gestalt ratio. This adds the edit-distance and token-set metrics it lacks — Jaro-Winkler (standard for short labels), Damerau (transposition-aware), and char-n-gram Jaccard/Dice — plus a unified `similarity()` that normalizes every metric to `[0, 1]`. Pairs with `normalize_text`. Pure-stdlib, deterministic.
160+
154161
## What's new (2026-06-22) — Time-Series Transforms
155162

156163
Turn counters into rates; downsample and resample. Full reference: [`docs/source/Eng/doc/new_features/v98_features_doc.rst`](docs/source/Eng/doc/new_features/v98_features_doc.rst).

README/README_zh-CN.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212

1313
## 目录
1414

15+
- [本次更新 (2026-06-22) — 字符串距离相似度量](#本次更新-2026-06-22--字符串距离相似度量)
1516
- [本次更新 (2026-06-22) — 时间序列变换](#本次更新-2026-06-22--时间序列变换)
1617
- [本次更新 (2026-06-22) — Unicode 文本规范化与 Slug](#本次更新-2026-06-22--unicode-文本规范化与-slug)
1718
- [本次更新 (2026-06-22) — JSON-Schema 兼容性检查](#本次更新-2026-06-22--json-schema-兼容性检查)
@@ -150,6 +151,12 @@
150151

151152
---
152153

154+
## 本次更新 (2026-06-22) — 字符串距离相似度量
155+
156+
匹配打字错误与重排 token。完整参考:[`docs/source/Zh/doc/new_features/v99_features_doc.rst`](../docs/source/Zh/doc/new_features/v99_features_doc.rst)
157+
158+
- **`levenshtein` / `damerau_levenshtein` / `jaro` / `jaro_winkler` / `jaccard` / `dice` / `similarity`**(`AC_text_similarity`):`fuzzy` 只提供 difflib 的 gestalt ratio。本功能补上它缺少的编辑距离与 token 集合度量 —— Jaro-Winkler(短标签标准)、Damerau(转置感知)、字符 n-gram Jaccard/Dice —— 并提供统一的 `similarity()` 把每个度量规范化到 `[0, 1]`。可搭配 `normalize_text`。纯标准库、确定。
159+
153160
## 本次更新 (2026-06-22) — 时间序列变换
154161

155162
把计数器转成速率;降采样与重采样。完整参考:[`docs/source/Zh/doc/new_features/v98_features_doc.rst`](../docs/source/Zh/doc/new_features/v98_features_doc.rst)

README/README_zh-TW.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212

1313
## 目錄
1414

15+
- [本次更新 (2026-06-22) — 字串距離相似度量](#本次更新-2026-06-22--字串距離相似度量)
1516
- [本次更新 (2026-06-22) — 時間序列轉換](#本次更新-2026-06-22--時間序列轉換)
1617
- [本次更新 (2026-06-22) — Unicode 文字正規化與 Slug](#本次更新-2026-06-22--unicode-文字正規化與-slug)
1718
- [本次更新 (2026-06-22) — JSON-Schema 相容性檢查](#本次更新-2026-06-22--json-schema-相容性檢查)
@@ -150,6 +151,12 @@
150151

151152
---
152153

154+
## 本次更新 (2026-06-22) — 字串距離相似度量
155+
156+
比對打字錯誤與重排 token。完整參考:[`docs/source/Zh/doc/new_features/v99_features_doc.rst`](../docs/source/Zh/doc/new_features/v99_features_doc.rst)
157+
158+
- **`levenshtein` / `damerau_levenshtein` / `jaro` / `jaro_winkler` / `jaccard` / `dice` / `similarity`**(`AC_text_similarity`):`fuzzy` 只提供 difflib 的 gestalt ratio。本功能補上它缺少的編輯距離與 token 集合度量 —— Jaro-Winkler(短標籤標準)、Damerau(轉置感知)、字元 n-gram Jaccard/Dice —— 並提供統一的 `similarity()` 把每個度量正規化到 `[0, 1]`。可搭配 `normalize_text`。純標準函式庫、具決定性。
159+
153160
## 本次更新 (2026-06-22) — 時間序列轉換
154161

155162
把計數器轉成速率;降採樣與重採樣。完整參考:[`docs/source/Zh/doc/new_features/v98_features_doc.rst`](../docs/source/Zh/doc/new_features/v98_features_doc.rst)
Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
String-Distance Similarity Metrics
2+
==================================
3+
4+
``fuzzy`` exposes only difflib's gestalt ratio. This adds the edit-distance and
5+
token-set metrics it lacks — Levenshtein / Damerau-Levenshtein, Jaro and
6+
Jaro-Winkler (the standard for short names and labels), and character-n-gram
7+
Jaccard / Dice — for better matching of typos and reordered tokens, especially
8+
from OCR.
9+
10+
Pure standard library; imports no ``PySide6``. Every function is pure (two
11+
strings in, a number out), so it is fully deterministic in CI. Pair with
12+
``normalize_text`` to make matches accent- and form-insensitive first.
13+
14+
Headless API
15+
------------
16+
17+
.. code-block:: python
18+
19+
from je_auto_control import (
20+
levenshtein, damerau_levenshtein, jaro_winkler, jaccard, dice,
21+
similarity, normalize_text,
22+
)
23+
24+
levenshtein("kitten", "sitting") # 3
25+
damerau_levenshtein("ab", "ba") # 1 (transposition)
26+
jaro_winkler("MARTHA", "MARHTA") # ~0.961
27+
jaccard("night", "nacht", n=2) # char-bigram overlap
28+
29+
# normalised [0, 1] score for any metric (edit distance -> 1 - d/max_len):
30+
similarity(normalize_text(a), normalize_text(b), metric="jaro_winkler")
31+
32+
``levenshtein`` / ``damerau_levenshtein`` return integer edit distances (the
33+
latter counting an adjacent transposition as one edit). ``jaro`` / ``jaro_winkler``
34+
and ``jaccard`` / ``dice`` return ``[0, 1]`` similarities. ``similarity`` is the
35+
unified entry point — it returns the Jaro/Jaccard/Dice metrics directly and
36+
converts edit distances to ``1 - distance / max_len`` so every metric is
37+
comparable on the same scale.
38+
39+
Executor command
40+
----------------
41+
42+
``AC_text_similarity`` returns ``{score}`` for two strings ``a`` / ``b`` and an
43+
optional ``metric``. It is exposed as the MCP tool ``ac_text_similarity`` and as
44+
a Script Builder command under **Data**.

docs/source/Eng/eng_index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,7 @@ Comprehensive guides for all AutoControl features.
121121
doc/new_features/v96_features_doc
122122
doc/new_features/v97_features_doc
123123
doc/new_features/v98_features_doc
124+
doc/new_features/v99_features_doc
124125
doc/ocr_backends/ocr_backends_doc
125126
doc/observability/observability_doc
126127
doc/operations_layer/operations_layer_doc
Lines changed: 37 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,37 @@
1+
字串距離相似度量
2+
==============
3+
4+
``fuzzy`` 只提供 difflib 的 gestalt ratio。本功能補上它缺少的編輯距離與 token 集合度量 ——
5+
Levenshtein / Damerau-Levenshtein、Jaro 與 Jaro-Winkler(短名稱/標籤的標準),以及字元 n-gram 的
6+
Jaccard / Dice —— 更適合比對打字錯誤與重排 token,尤其來自 OCR。
7+
8+
純標準函式庫;不匯入 ``PySide6``。每個函式皆為純函式(輸入兩個字串、輸出數字),因此在 CI 中完全
9+
具決定性。可先搭配 ``normalize_text`` 讓比對對重音與形式不敏感。
10+
11+
無頭 API
12+
--------
13+
14+
.. code-block:: python
15+
16+
from je_auto_control import (
17+
levenshtein, damerau_levenshtein, jaro_winkler, jaccard, dice,
18+
similarity, normalize_text,
19+
)
20+
21+
levenshtein("kitten", "sitting") # 3
22+
damerau_levenshtein("ab", "ba") # 1(轉置)
23+
jaro_winkler("MARTHA", "MARHTA") # ~0.961
24+
jaccard("night", "nacht", n=2) # 字元 bigram 重疊
25+
26+
# 任一度量的正規化 [0, 1] 分數(編輯距離 → 1 - d/max_len):
27+
similarity(normalize_text(a), normalize_text(b), metric="jaro_winkler")
28+
29+
``levenshtein`` / ``damerau_levenshtein`` 回傳整數編輯距離(後者把相鄰轉置算作一次編輯)。``jaro`` /
30+
``jaro_winkler`` 與 ``jaccard`` / ``dice`` 回傳 ``[0, 1]`` 相似度。``similarity`` 是統一入口 —— 直接回傳
31+
Jaro/Jaccard/Dice,並把編輯距離轉成 ``1 - distance / max_len``,讓所有度量在同一尺度上可比較。
32+
33+
執行器命令
34+
----------
35+
36+
``AC_text_similarity`` 對兩個字串 ``a`` / ``b`` 與選用的 ``metric`` 回傳 ``{score}``。它以 MCP 工具
37+
``ac_text_similarity`` 以及 Script Builder 中 **Data** 分類下的命令提供。

docs/source/Zh/zh_index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -121,6 +121,7 @@ AutoControl 所有功能的完整使用指南。
121121
doc/new_features/v96_features_doc
122122
doc/new_features/v97_features_doc
123123
doc/new_features/v98_features_doc
124+
doc/new_features/v99_features_doc
124125
doc/ocr_backends/ocr_backends_doc
125126
doc/observability/observability_doc
126127
doc/operations_layer/operations_layer_doc

je_auto_control/__init__.py

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -257,6 +257,11 @@
257257
from je_auto_control.utils.text_normalize import (
258258
deaccent, fold_whitespace, normalize_quotes, normalize_text, slugify,
259259
)
260+
# String-distance metrics (Levenshtein / Jaro-Winkler / Jaccard / Dice)
261+
from je_auto_control.utils.text_similarity import (
262+
damerau_levenshtein, dice, jaccard, jaro, jaro_winkler, levenshtein,
263+
similarity,
264+
)
260265
# S3-compatible artifact store (optional boto3, injectable client)
261266
from je_auto_control.utils.artifact_store import (
262267
S3ArtifactStore, configure_default_store, get_default_store,
@@ -928,6 +933,8 @@ def start_autocontrol_gui(*args, **kwargs):
928933
"fuzzy_best_match", "fuzzy_dedupe", "fuzzy_matches", "fuzzy_ratio",
929934
"deaccent", "fold_whitespace", "normalize_quotes", "normalize_text",
930935
"slugify",
936+
"damerau_levenshtein", "dice", "jaccard", "jaro", "jaro_winkler",
937+
"levenshtein", "similarity",
931938
"S3ArtifactStore", "configure_default_store", "get_default_store",
932939
"set_default_store",
933940
"average_hash", "dedupe_images", "dhash", "hamming_distance",

je_auto_control/gui/script_builder/command_schema.py

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1668,6 +1668,16 @@ def _add_resilience_specs(specs: List[CommandSpec]) -> None:
16681668
),
16691669
description="Produce an ASCII slug (de-accent, lowercase, join).",
16701670
))
1671+
specs.append(CommandSpec(
1672+
"AC_text_similarity", "Data", "Text: Similarity",
1673+
fields=(
1674+
FieldSpec("a", FieldType.STRING, placeholder="login"),
1675+
FieldSpec("b", FieldType.STRING, placeholder="lgoin"),
1676+
FieldSpec("metric", FieldType.STRING, optional=True,
1677+
placeholder="jaro_winkler | levenshtein | jaccard | dice"),
1678+
),
1679+
description="Normalised string similarity (Jaro-Winkler / edit / Jaccard).",
1680+
))
16711681
specs.append(CommandSpec(
16721682
"AC_spans_to_otlp", "Report", "OTLP: Export Spans",
16731683
fields=(

je_auto_control/utils/executor/action_executor.py

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3152,6 +3152,13 @@ def _slugify(text: str, sep: str = "-") -> Dict[str, Any]:
31523152
return {"slug": slugify(text, sep=sep)}
31533153

31543154

3155+
def _text_similarity(a: str, b: str,
3156+
metric: str = "jaro_winkler") -> Dict[str, Any]:
3157+
"""Adapter: normalised string similarity for the chosen metric."""
3158+
from je_auto_control.utils.text_similarity import similarity
3159+
return {"score": similarity(a, b, metric=metric)}
3160+
3161+
31553162
def _canonical_log(fields: Any) -> Dict[str, Any]:
31563163
"""Adapter: build a canonical log line from a fields dict."""
31573164
import json
@@ -4461,6 +4468,7 @@ def __init__(self):
44614468
"AC_spans_to_otlp": _spans_to_otlp,
44624469
"AC_normalize_text": _normalize_text,
44634470
"AC_slugify": _slugify,
4471+
"AC_text_similarity": _text_similarity,
44644472
"AC_validate_config": _validate_config,
44654473
"AC_resolve_ref": _resolve_ref,
44664474
"AC_resolve_refs": _resolve_refs,

0 commit comments

Comments
 (0)