Skip to content

Commit 74475fe

Browse files
authored
Merge pull request #317 from Integration-Automation/feat/confusables-batch
Add confusable / homoglyph detection for Unicode spoofing
2 parents d6a079d + 95a4ef6 commit 74475fe

15 files changed

Lines changed: 362 additions & 1 deletion

File tree

README.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,7 @@
1313

1414
## Table of Contents
1515

16+
- [What's new (2026-06-22) — Confusable / Homoglyph Detection](#whats-new-2026-06-22--confusable--homoglyph-detection)
1617
- [What's new (2026-06-22) — Locale-Aware String Collation](#whats-new-2026-06-22--locale-aware-string-collation)
1718
- [What's new (2026-06-22) — Transactional Outbox](#whats-new-2026-06-22--transactional-outbox)
1819
- [What's new (2026-06-22) — Optimistic-Concurrency Versioned Store](#whats-new-2026-06-22--optimistic-concurrency-versioned-store)
@@ -161,6 +162,12 @@
161162

162163
---
163164

165+
## What's new (2026-06-22) — Confusable / Homoglyph Detection
166+
167+
Catch Unicode visual spoofing (IDN-homograph phishing, lookalike labels). Full reference: [`docs/source/Eng/doc/new_features/v109_features_doc.rst`](docs/source/Eng/doc/new_features/v109_features_doc.rst).
168+
169+
- **`confusable_skeleton` / `is_confusable` / `detect_homoglyphs` / `is_mixed_script` / `scripts_of`** (`AC_confusable_scan`, `AC_confusable_compare`): a Cyrillic `"а"` is pixel-for-pixel a Latin `"a"`, so `"pаypal"` reads as `"paypal"` yet compares unequal. Following Unicode TR39, this folds confusables to a prototype skeleton (strings match when skeletons match) and flags mixed-script tokens. Pure-stdlib (`unicodedata`), deterministic.
170+
164171
## What's new (2026-06-22) — Locale-Aware String Collation
165172

166173
Sort strings the way a reader of the language expects. Full reference: [`docs/source/Eng/doc/new_features/v108_features_doc.rst`](docs/source/Eng/doc/new_features/v108_features_doc.rst).

README/README_zh-CN.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212

1313
## 目录
1414

15+
- [本次更新 (2026-06-22) — 易混淆字符 / 同形异义字检测](#本次更新-2026-06-22--易混淆字符--同形异义字检测)
1516
- [本次更新 (2026-06-22) — 区域感知字符串排序](#本次更新-2026-06-22--区域感知字符串排序)
1617
- [本次更新 (2026-06-22) — 事务型 Outbox](#本次更新-2026-06-22--事务型-outbox)
1718
- [本次更新 (2026-06-22) — 乐观并发版本存储](#本次更新-2026-06-22--乐观并发版本存储)
@@ -164,6 +165,12 @@
164165

165166
平滑噪声值序列。完整参考:[`docs/source/Zh/doc/new_features/v102_features_doc.rst`](../docs/source/Zh/doc/new_features/v102_features_doc.rst)
166167

168+
## 本次更新 (2026-06-22) — 易混淆字符 / 同形异义字检测
169+
170+
抓出 Unicode 视觉仿冒(IDN 同形异义字钓鱼、仿冒标签)。完整参考:[`docs/source/Zh/doc/new_features/v109_features_doc.rst`](../docs/source/Zh/doc/new_features/v109_features_doc.rst)
171+
172+
- **`confusable_skeleton` / `is_confusable` / `detect_homoglyphs` / `is_mixed_script` / `scripts_of`**(`AC_confusable_scan``AC_confusable_compare`):西里尔字母 `"а"` 与拉丁字母 `"a"` 在像素上相同,因此 `"pаypal"` 读来是 `"paypal"` 却比较不相等。参照 Unicode TR39,本功能将易混淆字折叠为原型骨架(骨架相同即相符),并标记混用文字系统的令牌。纯标准库(`unicodedata`)、确定。
173+
167174
## 本次更新 (2026-06-22) — 区域感知字符串排序
168175

169176
依某语言读者的期望排序字符串。完整参考:[`docs/source/Zh/doc/new_features/v108_features_doc.rst`](../docs/source/Zh/doc/new_features/v108_features_doc.rst)

README/README_zh-TW.md

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@
1212

1313
## 目錄
1414

15+
- [本次更新 (2026-06-22) — 易混淆字元 / 同形異義字偵測](#本次更新-2026-06-22--易混淆字元--同形異義字偵測)
1516
- [本次更新 (2026-06-22) — 地區感知字串排序](#本次更新-2026-06-22--地區感知字串排序)
1617
- [本次更新 (2026-06-22) — 交易型 Outbox](#本次更新-2026-06-22--交易型-outbox)
1718
- [本次更新 (2026-06-22) — 樂觀並行版本儲存](#本次更新-2026-06-22--樂觀並行版本儲存)
@@ -164,6 +165,12 @@
164165

165166
平滑雜訊值序列。完整參考:[`docs/source/Zh/doc/new_features/v102_features_doc.rst`](../docs/source/Zh/doc/new_features/v102_features_doc.rst)
166167

168+
## 本次更新 (2026-06-22) — 易混淆字元 / 同形異義字偵測
169+
170+
抓出 Unicode 視覺仿冒(IDN 同形異義字釣魚、仿冒標籤)。完整參考:[`docs/source/Zh/doc/new_features/v109_features_doc.rst`](../docs/source/Zh/doc/new_features/v109_features_doc.rst)
171+
172+
- **`confusable_skeleton` / `is_confusable` / `detect_homoglyphs` / `is_mixed_script` / `scripts_of`**(`AC_confusable_scan``AC_confusable_compare`):西里爾字母 `"а"` 與拉丁字母 `"a"` 在像素上相同,因此 `"pаypal"` 讀來是 `"paypal"` 卻比較不相等。參照 Unicode TR39,本功能將易混淆字折疊為原型骨架(骨架相同即相符),並標記混用文字系統的權杖。純標準函式庫(`unicodedata`)、具決定性。
173+
167174
## 本次更新 (2026-06-22) — 地區感知字串排序
168175

169176
依某語言讀者的期望排序字串。完整參考:[`docs/source/Zh/doc/new_features/v108_features_doc.rst`](../docs/source/Zh/doc/new_features/v108_features_doc.rst)
Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
Confusable / Homoglyph Detection
2+
================================
3+
4+
``secrets_scan`` finds secret-shaped tokens and ``guardrail`` screens text for
5+
prompt injection, but nothing catches *visual* spoofing: a Cyrillic ``"а"``
6+
(U+0430) is pixel-for-pixel a Latin ``"a"`` (U+0061), so ``"pаypal"`` (with a
7+
Cyrillic ``а``) reads as ``"paypal"`` to a human yet compares unequal — the basis
8+
of IDN-homograph phishing and lookalike UI labels.
9+
10+
Following the idea of Unicode TR39, this folds confusable characters to a
11+
prototype *skeleton* (two strings are confusable when their skeletons match) and
12+
flags strings that mix scripts. Pure standard library (``unicodedata``); imports
13+
no ``PySide6``. Every function is pure, so it is fully deterministic in CI.
14+
15+
Headless API
16+
------------
17+
18+
.. code-block:: python
19+
20+
from je_auto_control import (
21+
confusable_skeleton, is_confusable, detect_homoglyphs,
22+
is_mixed_script, scripts_of,
23+
)
24+
25+
confusable_skeleton("pаypal") # 'paypal' (Cyrillic а -> a)
26+
is_confusable("pаypal", "paypal") # True
27+
detect_homoglyphs("pаypal") # [{'index': 1, 'char': 'а', 'prototype': 'a'}]
28+
is_mixed_script("pаypal") # True (Latin + Cyrillic)
29+
scripts_of("pаypal") # {'LATIN', 'CYRILLIC'}
30+
31+
``confusable_skeleton`` NFKC-normalises (folding fullwidth, ligatures and math
32+
alphanumerics) then maps each remaining cross-script lookalike to its Latin
33+
prototype. ``is_confusable`` is true only for *distinct* strings with equal
34+
skeletons. ``detect_homoglyphs`` returns the offending characters with their
35+
position and prototype. ``scripts_of`` / ``is_mixed_script`` classify characters
36+
by Unicode block (ignoring digits, punctuation and spaces) so a single mixed-
37+
script token can be flagged on its own.
38+
39+
Executor commands
40+
-----------------
41+
42+
``AC_confusable_scan`` returns ``{skeleton, homoglyphs, mixed_script, scripts}``
43+
for one string; ``AC_confusable_compare`` returns ``{confusable}`` for a pair.
44+
Both are exposed as MCP tools (``ac_confusable_scan`` / ``ac_confusable_compare``)
45+
and as Script Builder commands under **Data**.

docs/source/Eng/eng_index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -131,6 +131,7 @@ Comprehensive guides for all AutoControl features.
131131
doc/new_features/v106_features_doc
132132
doc/new_features/v107_features_doc
133133
doc/new_features/v108_features_doc
134+
doc/new_features/v109_features_doc
134135
doc/ocr_backends/ocr_backends_doc
135136
doc/observability/observability_doc
136137
doc/operations_layer/operations_layer_doc
Lines changed: 38 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,38 @@
1+
易混淆字元 / 同形異義字偵測
2+
==========================
3+
4+
``secrets_scan`` 找出疑似機密的權杖、``guardrail`` 篩檢提示注入,但沒有任何功能能抓出*視覺*仿冒:西里爾字母
5+
``"а"``(U+0430)與拉丁字母 ``"a"``(U+0061)在像素上完全相同,因此 ``"pаypal"``(其中 ``а`` 為西里爾字母)
6+
對人類讀來就是 ``"paypal"``,但比較卻不相等——這正是 IDN 同形異義字釣魚與仿冒 UI 標籤的根源。
7+
8+
本功能參照 Unicode TR39 的概念,將易混淆字元折疊為原型*骨架*(skeleton)(兩字串骨架相同即為易混淆),並標記
9+
混用多種文字系統的字串。純標準函式庫(``unicodedata``);不匯入 ``PySide6``。每個函式皆為純函式,因此在 CI 中
10+
完全具決定性。
11+
12+
無頭 API
13+
--------
14+
15+
.. code-block:: python
16+
17+
from je_auto_control import (
18+
confusable_skeleton, is_confusable, detect_homoglyphs,
19+
is_mixed_script, scripts_of,
20+
)
21+
22+
confusable_skeleton("pаypal") # 'paypal' (西里爾 а -> a)
23+
is_confusable("pаypal", "paypal") # True
24+
detect_homoglyphs("pаypal") # [{'index': 1, 'char': 'а', 'prototype': 'a'}]
25+
is_mixed_script("pаypal") # True (拉丁 + 西里爾)
26+
scripts_of("pаypal") # {'LATIN', 'CYRILLIC'}
27+
28+
``confusable_skeleton`` 先以 NFKC 正規化(折疊全形、連字與數學英數字),再將每個剩餘的跨文字系統仿冒字對映到
29+
其拉丁原型。``is_confusable`` 僅在兩個*不同*字串骨架相同時為真。``detect_homoglyphs`` 回傳有問題的字元連同其
30+
位置與原型。``scripts_of`` / ``is_mixed_script`` 依 Unicode 區塊將字元分類(忽略數字、標點與空白),因此可單獨
31+
標記一個混用文字系統的權杖。
32+
33+
執行器命令
34+
----------
35+
36+
``AC_confusable_scan`` 對單一字串回傳 ``{skeleton, homoglyphs, mixed_script, scripts}``;``AC_confusable_compare``
37+
對一組字串回傳 ``{confusable}``。兩者皆以 MCP 工具(``ac_confusable_scan`` / ``ac_confusable_compare``)以及
38+
Script Builder 中 **Data** 分類下的命令提供。

docs/source/Zh/zh_index.rst

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -131,6 +131,7 @@ AutoControl 所有功能的完整使用指南。
131131
doc/new_features/v106_features_doc
132132
doc/new_features/v107_features_doc
133133
doc/new_features/v108_features_doc
134+
doc/new_features/v109_features_doc
134135
doc/ocr_backends/ocr_backends_doc
135136
doc/observability/observability_doc
136137
doc/operations_layer/operations_layer_doc

je_auto_control/__init__.py

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -218,6 +218,11 @@
218218
collation_key, sort_strings,
219219
)
220220
from je_auto_control.utils.locale_collation import compare as collation_compare
221+
# Confusable / homoglyph detection (Unicode-spoofing skeletons)
222+
from je_auto_control.utils.confusables import (
223+
detect_homoglyphs, is_confusable, is_mixed_script, scripts_of,
224+
)
225+
from je_auto_control.utils.confusables import skeleton as confusable_skeleton
221226
# CI workflow annotations (GitHub Actions)
222227
from je_auto_control.utils.ci_annotations import (
223228
emit_annotations, format_annotation,
@@ -951,6 +956,11 @@ def start_autocontrol_gui(*args, **kwargs):
951956
"collation_key",
952957
"collation_compare",
953958
"sort_strings",
959+
"confusable_skeleton",
960+
"detect_homoglyphs",
961+
"is_confusable",
962+
"is_mixed_script",
963+
"scripts_of",
954964
"emit_annotations", "format_annotation",
955965
"ClipboardHistory", "default_clipboard_history",
956966
"analyze_heal_log", "heal_stats", "scan_secrets",

je_auto_control/gui/script_builder/command_schema.py

Lines changed: 15 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2090,6 +2090,21 @@ def _add_resilience_specs(specs: List[CommandSpec]) -> None:
20902090
),
20912091
description="Locale-aware compare; returns order -1/0/1.",
20922092
))
2093+
specs.append(CommandSpec(
2094+
"AC_confusable_scan", "Data", "Text: Confusable Scan",
2095+
fields=(
2096+
FieldSpec("text", FieldType.STRING, placeholder="pаypal.com"),
2097+
),
2098+
description="Homoglyph / mixed-script spoofing report for a string.",
2099+
))
2100+
specs.append(CommandSpec(
2101+
"AC_confusable_compare", "Data", "Text: Confusable Compare",
2102+
fields=(
2103+
FieldSpec("first", FieldType.STRING, placeholder="paypal"),
2104+
FieldSpec("second", FieldType.STRING, placeholder="pаypal"),
2105+
),
2106+
description="Whether two strings share the same confusable skeleton.",
2107+
))
20932108
specs.append(CommandSpec(
20942109
"AC_diff_rows", "Data", "Dataset Diff: Rows by Key",
20952110
fields=(
Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
"""Confusable / homoglyph detection (Unicode-spoofing skeletons)."""
2+
from je_auto_control.utils.confusables.confusables import (
3+
detect_homoglyphs, is_confusable, is_mixed_script, scripts_of, skeleton,
4+
)
5+
6+
__all__ = [
7+
"detect_homoglyphs", "is_confusable", "is_mixed_script", "scripts_of",
8+
"skeleton",
9+
]

0 commit comments

Comments
 (0)