Skip to content

Commit a29ec8b

Browse files
JE-ChenJE-Chen
committed
Add Unicode text normalisation and slugify
fuzzy and search_index.tokenize only lowercase and OCR find_text_matches only .lower()+substring, so the same text in different Unicode forms (NFC/NFD), accents, or smart quotes compares unequal. Add normalize_text (NFKC + casefold + whitespace fold), deaccent, normalize_quotes, fold_whitespace, and slugify — the canonicalisation layer to run before matching. Wired through facade, executor (AC_normalize_text / AC_slugify), MCP, and the Script Builder with a headless test batch and EN/Zh docs.
1 parent 426d1cb commit a29ec8b

15 files changed

Lines changed: 321 additions & 0 deletions

File tree

‎README.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313

1414
## Table of Contents
1515

16+
- [What's new (2026-06-22) — Unicode Text Normalisation & Slugify](#whats-new-2026-06-22--unicode-text-normalisation--slugify)
1617
- [What's new (2026-06-22) — JSON-Schema Compatibility Checking](#whats-new-2026-06-22--json-schema-compatibility-checking)
1718
- [What's new (2026-06-22) — Typed Configuration Schema](#whats-new-2026-06-22--typed-configuration-schema)
1819
- [What's new (2026-06-22) — OTLP/JSON Span Export](#whats-new-2026-06-22--otlpjson-span-export)
@@ -149,6 +150,12 @@
149150

150151
---
151152

153+
## What's new (2026-06-22) — Unicode Text Normalisation & Slugify
154+
155+
Canonicalize text before fuzzy/search/OCR matching. Full reference: [`docs/source/Eng/doc/new_features/v97_features_doc.rst`](docs/source/Eng/doc/new_features/v97_features_doc.rst).
156+
157+
- **`normalize_text` / `deaccent` / `slugify` / `normalize_quotes` / `fold_whitespace`** (`AC_normalize_text`, `AC_slugify`): `fuzzy` and `search_index.tokenize` only lowercase and OCR matching only `.lower()`+substring, so `"Café"` (NFC) vs `"Café"` (NFD) vs `"cafe"` compare unequal. This adds the missing canonicalization layer (NFKC + casefold + whitespace fold, accent stripping, smart-quote mapping, ASCII slugs). Pure-stdlib (`unicodedata`), deterministic.
158+
152159
## What's new (2026-06-22) — JSON-Schema Compatibility Checking
153160

154161
Classify schema changes as backward/forward/full. Full reference: [`docs/source/Eng/doc/new_features/v96_features_doc.rst`](docs/source/Eng/doc/new_features/v96_features_doc.rst).

‎README/README_zh-CN.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212

1313
## 目录
1414

15+
- [本次更新 (2026-06-22) — Unicode 文本规范化与 Slug](#本次更新-2026-06-22--unicode-文本规范化与-slug)
1516
- [本次更新 (2026-06-22) — JSON-Schema 兼容性检查](#本次更新-2026-06-22--json-schema-兼容性检查)
1617
- [本次更新 (2026-06-22) — 具类型的配置结构](#本次更新-2026-06-22--具类型的配置结构)
1718
- [本次更新 (2026-06-22) — OTLP/JSON Span 导出](#本次更新-2026-06-22--otlpjson-span-导出)
@@ -148,6 +149,12 @@
148149

149150
---
150151

152+
## 本次更新 (2026-06-22) — Unicode 文本规范化与 Slug
153+
154+
在 fuzzy/search/OCR 匹配前规范化文本。完整参考:[`docs/source/Zh/doc/new_features/v97_features_doc.rst`](../docs/source/Zh/doc/new_features/v97_features_doc.rst)。
155+
156+
- **`normalize_text` / `deaccent` / `slugify` / `normalize_quotes` / `fold_whitespace`**(`AC_normalize_text`、`AC_slugify`):`fuzzy` 与 `search_index.tokenize` 只做小写,OCR 匹配只做 `.lower()`+子串,因此 `"Café"`(NFC)、`"Café"`(NFD)、`"cafe"` 会匹配不相等。本功能补上缺少的规范化层(NFKC + casefold + 空白折叠、去重音、智能引号映射、ASCII slug)。纯标准库(`unicodedata`)、确定。
157+
151158
## 本次更新 (2026-06-22) — JSON-Schema 兼容性检查
152159

153160
把结构变更分类为 backward/forward/full。完整参考:[`docs/source/Zh/doc/new_features/v96_features_doc.rst`](../docs/source/Zh/doc/new_features/v96_features_doc.rst)。

‎README/README_zh-TW.md‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212

1313
## 目錄
1414

15+
- [本次更新 (2026-06-22) — Unicode 文字正規化與 Slug](#本次更新-2026-06-22--unicode-文字正規化與-slug)
1516
- [本次更新 (2026-06-22) — JSON-Schema 相容性檢查](#本次更新-2026-06-22--json-schema-相容性檢查)
1617
- [本次更新 (2026-06-22) — 具型別的設定結構](#本次更新-2026-06-22--具型別的設定結構)
1718
- [本次更新 (2026-06-22) — OTLP/JSON Span 匯出](#本次更新-2026-06-22--otlpjson-span-匯出)
@@ -148,6 +149,12 @@
148149

149150
---
150151

152+
## 本次更新 (2026-06-22) — Unicode 文字正規化與 Slug
153+
154+
在 fuzzy/search/OCR 比對前正規化文字。完整參考:[`docs/source/Zh/doc/new_features/v97_features_doc.rst`](../docs/source/Zh/doc/new_features/v97_features_doc.rst)。
155+
156+
- **`normalize_text` / `deaccent` / `slugify` / `normalize_quotes` / `fold_whitespace`**(`AC_normalize_text`、`AC_slugify`):`fuzzy` 與 `search_index.tokenize` 只做小寫,OCR 比對只做 `.lower()`+子字串,因此 `"Café"`(NFC)、`"Café"`(NFD)、`"cafe"` 會比對不相等。本功能補上缺少的正規化層(NFKC + casefold + 空白折疊、去重音、智慧引號對應、ASCII slug)。純標準函式庫(`unicodedata`)、具決定性。
157+
151158
## 本次更新 (2026-06-22) — JSON-Schema 相容性檢查
152159

153160
把結構變更分類為 backward/forward/full。完整參考:[`docs/source/Zh/doc/new_features/v96_features_doc.rst`](../docs/source/Zh/doc/new_features/v96_features_doc.rst)。

‎docs/source/Eng/doc/new_features/v97_features_doc.rst‎

Lines changed: 40 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,40 @@
1+
Unicode Text Normalisation & Slugify
2+
====================================
3+
4+
``fuzzy`` and ``search_index.tokenize`` only lowercase, and OCR
5+
``find_text_matches`` only ``.lower()`` + substring — so ``"Café"`` (NFC) versus
6+
``"Café"`` (NFD) versus OCR ``"cafe"`` compare unequal. This adds the
7+
canonicalisation layer they should run before matching.
8+
9+
Pure standard library (``unicodedata`` / ``re``); imports no ``PySide6``. Every
10+
function is pure (text in, text out), so it is fully deterministic in CI.
11+
12+
Headless API
13+
------------
14+
15+
.. code-block:: python
16+
17+
from je_auto_control import (
18+
normalize_text, deaccent, slugify, normalize_quotes, fold_whitespace,
19+
)
20+
21+
normalize_text("CAFÉ Menu") # "café menu" (NFKC + casefold + ws)
22+
deaccent("résumé") # "resume"
23+
slugify("Café Menu! 2026") # "cafe-menu-2026"
24+
normalize_quotes("“Hi” — it’s…") # '"Hi" - it\'s...'
25+
26+
``normalize_text`` applies a Unicode ``form`` (default ``NFKC``), optional
27+
casefolding, and whitespace folding, so the same text in different code-point
28+
forms compares equal. ``deaccent`` strips combining marks; ``fold_whitespace``
29+
collapses runs to single spaces; ``normalize_quotes`` maps smart quotes, dashes,
30+
ellipsis and NBSP to ASCII; ``slugify`` produces an ASCII slug (de-accent,
31+
lowercase, join alphanumeric runs with a separator). Run ``normalize_text``
32+
before fuzzy/search/OCR matching to make matches accent- and form-insensitive.
33+
34+
Executor commands
35+
-----------------
36+
37+
``AC_normalize_text`` returns ``{text}`` (with optional ``form`` / ``casefold``
38+
/ ``collapse_ws``); ``AC_slugify`` returns ``{slug}``. Both are exposed as MCP
39+
tools (``ac_normalize_text`` / ``ac_slugify``) and as Script Builder commands
40+
under **Data**.

‎docs/source/Eng/eng_index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -119,6 +119,7 @@ Comprehensive guides for all AutoControl features.
119119
doc/new_features/v94_features_doc
120120
doc/new_features/v95_features_doc
121121
doc/new_features/v96_features_doc
122+
doc/new_features/v97_features_doc
122123
doc/ocr_backends/ocr_backends_doc
123124
doc/observability/observability_doc
124125
doc/operations_layer/operations_layer_doc

‎docs/source/Zh/doc/new_features/v97_features_doc.rst‎

Lines changed: 35 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,35 @@
1+
Unicode 文字正規化與 Slug
2+
========================
3+
4+
``fuzzy`` 與 ``search_index.tokenize`` 只做小寫化,OCR ``find_text_matches`` 只做 ``.lower()`` + 子字串
5+
比對 —— 因此 ``"Café"``(NFC)、``"Café"``(NFD)與 OCR 的 ``"cafe"`` 會比對不相等。本功能補上它們在比對
6+
前應執行的正規化層。
7+
8+
純標準函式庫(``unicodedata`` / ``re``);不匯入 ``PySide6``。每個函式皆為純函式(輸入文字、輸出文字),
9+
因此在 CI 中完全具決定性。
10+
11+
無頭 API
12+
--------
13+
14+
.. code-block:: python
15+
16+
from je_auto_control import (
17+
normalize_text, deaccent, slugify, normalize_quotes, fold_whitespace,
18+
)
19+
20+
normalize_text("CAFÉ Menu") # "café menu"(NFKC + casefold + 空白)
21+
deaccent("résumé") # "resume"
22+
slugify("Café Menu! 2026") # "cafe-menu-2026"
23+
normalize_quotes("“Hi” — it’s…") # '"Hi" - it\'s...'
24+
25+
``normalize_text`` 套用 Unicode ``form``(預設 ``NFKC``)、選用 casefold 與空白折疊,讓不同碼點形式的相同
26+
文字比對相等。``deaccent`` 去除組合附加符號;``fold_whitespace`` 把連續空白收成單一空格;``normalize_quotes``
27+
把智慧引號、破折號、省略號與 NBSP 對應成 ASCII;``slugify`` 產生 ASCII slug(去重音、小寫、以分隔符連接
28+
英數段)。在 fuzzy/search/OCR 比對前先執行 ``normalize_text`` 可讓比對對重音與形式不敏感。
29+
30+
執行器命令
31+
----------
32+
33+
``AC_normalize_text`` 回傳 ``{text}``(可選 ``form`` / ``casefold`` / ``collapse_ws``);``AC_slugify`` 回傳
34+
``{slug}``。兩者皆以 MCP 工具(``ac_normalize_text`` / ``ac_slugify``)以及 Script Builder 中 **Data** 分類下
35+
的命令提供。

‎docs/source/Zh/zh_index.rst‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -119,6 +119,7 @@ AutoControl 所有功能的完整使用指南。
119119
doc/new_features/v94_features_doc
120120
doc/new_features/v95_features_doc
121121
doc/new_features/v96_features_doc
122+
doc/new_features/v97_features_doc
122123
doc/ocr_backends/ocr_backends_doc
123124
doc/observability/observability_doc
124125
doc/operations_layer/operations_layer_doc

‎je_auto_control/__init__.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -253,6 +253,10 @@
253253
from je_auto_control.utils.fuzzy import (
254254
fuzzy_best_match, fuzzy_dedupe, fuzzy_matches, fuzzy_ratio,
255255
)
256+
# Unicode text normalisation + slugify (canonicalise before matching)
257+
from je_auto_control.utils.text_normalize import (
258+
deaccent, fold_whitespace, normalize_quotes, normalize_text, slugify,
259+
)
256260
# S3-compatible artifact store (optional boto3, injectable client)
257261
from je_auto_control.utils.artifact_store import (
258262
S3ArtifactStore, configure_default_store, get_default_store,
@@ -917,6 +921,8 @@ def start_autocontrol_gui(*args, **kwargs):
917921
"VideoStep", "build_overlay_plan", "render_overlay_frame",
918922
"write_step_video",
919923
"fuzzy_best_match", "fuzzy_dedupe", "fuzzy_matches", "fuzzy_ratio",
924+
"deaccent", "fold_whitespace", "normalize_quotes", "normalize_text",
925+
"slugify",
920926
"S3ArtifactStore", "configure_default_store", "get_default_store",
921927
"set_default_store",
922928
"average_hash", "dedupe_images", "dhash", "hamming_distance",

‎je_auto_control/gui/script_builder/command_schema.py‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1648,6 +1648,26 @@ def _add_resilience_specs(specs: List[CommandSpec]) -> None:
16481648
),
16491649
description="Build a canonical wide-event log line (rendered as JSON).",
16501650
))
1651+
specs.append(CommandSpec(
1652+
"AC_normalize_text", "Data", "Text: Normalize (Unicode)",
1653+
fields=(
1654+
FieldSpec("text", FieldType.STRING, placeholder="Café Menu"),
1655+
FieldSpec("form", FieldType.STRING, optional=True,
1656+
placeholder="NFKC"),
1657+
FieldSpec("casefold", FieldType.BOOL, optional=True, default=True),
1658+
FieldSpec("collapse_ws", FieldType.BOOL, optional=True,
1659+
default=True),
1660+
),
1661+
description="Unicode-normalise text (form + casefold + ws fold).",
1662+
))
1663+
specs.append(CommandSpec(
1664+
"AC_slugify", "Data", "Text: Slugify",
1665+
fields=(
1666+
FieldSpec("text", FieldType.STRING, placeholder="Café Menu!"),
1667+
FieldSpec("sep", FieldType.STRING, optional=True, placeholder="-"),
1668+
),
1669+
description="Produce an ASCII slug (de-accent, lowercase, join).",
1670+
))
16511671
specs.append(CommandSpec(
16521672
"AC_spans_to_otlp", "Report", "OTLP: Export Spans",
16531673
fields=(

‎je_auto_control/utils/executor/action_executor.py‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3138,6 +3138,20 @@ def _baggage_parse(header: str) -> Dict[str, Any]:
31383138
return {"items": parse_baggage(header).to_dict()}
31393139

31403140

3141+
def _normalize_text(text: str, form: str = "NFKC", casefold: Any = True,
3142+
collapse_ws: Any = True) -> Dict[str, Any]:
3143+
"""Adapter: Unicode-normalise text into {text}."""
3144+
from je_auto_control.utils.text_normalize import normalize_text
3145+
return {"text": normalize_text(text, form=form, casefold=bool(casefold),
3146+
collapse_ws=bool(collapse_ws))}
3147+
3148+
3149+
def _slugify(text: str, sep: str = "-") -> Dict[str, Any]:
3150+
"""Adapter: produce an ASCII slug from text."""
3151+
from je_auto_control.utils.text_normalize import slugify
3152+
return {"slug": slugify(text, sep=sep)}
3153+
3154+
31413155
def _canonical_log(fields: Any) -> Dict[str, Any]:
31423156
"""Adapter: build a canonical log line from a fields dict."""
31433157
import json
@@ -4424,6 +4438,8 @@ def __init__(self):
44244438
"AC_baggage_format": _baggage_format,
44254439
"AC_canonical_log": _canonical_log,
44264440
"AC_spans_to_otlp": _spans_to_otlp,
4441+
"AC_normalize_text": _normalize_text,
4442+
"AC_slugify": _slugify,
44274443
"AC_validate_config": _validate_config,
44284444
"AC_resolve_ref": _resolve_ref,
44294445
"AC_resolve_refs": _resolve_refs,

0 commit comments

Comments
 (0)