Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 7 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,7 @@

## Table of Contents

- [What's new (2026-06-20) — Text PII Detection & Redaction](#whats-new-2026-06-20--text-pii-detection--redaction)
- [What's new (2026-06-20) — Self-Healing Locator Write-Back](#whats-new-2026-06-20--self-healing-locator-write-back)
- [What's new (2026-06-20) — DMN-Style Decision Tables](#whats-new-2026-06-20--dmn-style-decision-tables)
- [What's new (2026-06-20) — Saga / Compensating Rollback](#whats-new-2026-06-20--saga--compensating-rollback)
Expand Down Expand Up @@ -107,6 +108,12 @@

---

## What's new (2026-06-20) — Text PII Detection & Redaction

Mask PII in text before it leaks. Full reference: [`docs/source/Eng/doc/new_features/v55_features_doc.rst`](docs/source/Eng/doc/new_features/v55_features_doc.rst).

- **`detect_pii` / `redact_pii_text`** (`AC_detect_pii` / `AC_redact_pii`, `ac_*`): image redaction existed but text (OCR, clipboard, LLM I/O, logs) had no string-level PII handling. This detects emails / phones / SSNs / credit cards / IPv4 / IBANs over plain text and redacts with `label` / `mask` / `partial` / `hash`. Overlapping spans dedupe (a card isn't also a phone); patterns are backtracking-safe. Pure-stdlib `re`+`hashlib`.

## What's new (2026-06-20) — Self-Healing Locator Write-Back

Persist corrected locators so heals aren't forgotten. Full reference: [`docs/source/Eng/doc/new_features/v54_features_doc.rst`](docs/source/Eng/doc/new_features/v54_features_doc.rst).
Expand Down
7 changes: 7 additions & 0 deletions README/README_zh-CN.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@

## 目录

- [本次更新 (2026-06-20) — 文本 PII 检测与遮蔽](#本次更新-2026-06-20--文本-pii-检测与遮蔽)
- [本次更新 (2026-06-20) — 自我修复定位器回写](#本次更新-2026-06-20--自我修复定位器回写)
- [本次更新 (2026-06-20) — DMN 式决策表](#本次更新-2026-06-20--dmn-式决策表)
- [本次更新 (2026-06-20) — Saga / 补偿回滚](#本次更新-2026-06-20--saga--补偿回滚)
Expand Down Expand Up @@ -106,6 +107,12 @@

---

## 本次更新 (2026-06-20) — 文本 PII 检测与遮蔽

在文本泄漏前遮蔽 PII。完整参考:[`docs/source/Zh/doc/new_features/v55_features_doc.rst`](../docs/source/Zh/doc/new_features/v55_features_doc.rst)。

- **`detect_pii` / `redact_pii_text`**(`AC_detect_pii` / `AC_redact_pii`、`ac_*`):图像遮蔽已存在,但文本(OCR、剪贴板、LLM I/O、日志)无字符串级 PII 处理。本功能在纯文本上检测邮件/电话/SSN/信用卡/IPv4/IBAN 并以 `label`/`mask`/`partial`/`hash` 遮蔽。重叠区段会去重(卡号不会同时是电话);模式无回溯风险。纯标准库 `re`+`hashlib`。

## 本次更新 (2026-06-20) — 自我修复定位器回写

保存修正定位器,使修复不被遗忘。完整参考:[`docs/source/Zh/doc/new_features/v54_features_doc.rst`](../docs/source/Zh/doc/new_features/v54_features_doc.rst)。
Expand Down
7 changes: 7 additions & 0 deletions README/README_zh-TW.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@

## 目錄

- [本次更新 (2026-06-20) — 文字 PII 偵測與遮蔽](#本次更新-2026-06-20--文字-pii-偵測與遮蔽)
- [本次更新 (2026-06-20) — 自我修復定位器回寫](#本次更新-2026-06-20--自我修復定位器回寫)
- [本次更新 (2026-06-20) — DMN 式決策表](#本次更新-2026-06-20--dmn-式決策表)
- [本次更新 (2026-06-20) — Saga / 補償回溯](#本次更新-2026-06-20--saga--補償回溯)
Expand Down Expand Up @@ -106,6 +107,12 @@

---

## 本次更新 (2026-06-20) — 文字 PII 偵測與遮蔽

在文字洩漏前遮蔽 PII。完整參考:[`docs/source/Zh/doc/new_features/v55_features_doc.rst`](../docs/source/Zh/doc/new_features/v55_features_doc.rst)。

- **`detect_pii` / `redact_pii_text`**(`AC_detect_pii` / `AC_redact_pii`、`ac_*`):影像遮蔽已存在,但文字(OCR、剪貼簿、LLM I/O、日誌)無字串層級 PII 處理。本功能在純文字上偵測電子郵件/電話/SSN/信用卡/IPv4/IBAN 並以 `label`/`mask`/`partial`/`hash` 遮蔽。重疊區段會去重(卡號不會同時是電話);樣式無回溯風險。純標準函式庫 `re`+`hashlib`。

## 本次更新 (2026-06-20) — 自我修復定位器回寫

保存修正定位器,使修復不被遺忘。完整參考:[`docs/source/Zh/doc/new_features/v54_features_doc.rst`](../docs/source/Zh/doc/new_features/v54_features_doc.rst)。
Expand Down
48 changes: 48 additions & 0 deletions docs/source/Eng/doc/new_features/v55_features_doc.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
Text PII Detection & Redaction
==============================

The image-redaction module blurs PII in screenshots, but text scraped from a UI,
OCR, the clipboard, an LLM prompt/response, or a log line had no string-level
equivalent — so PII could leak into action records, audit logs, or a model call.
``detect_pii`` / ``redact_pii_text`` find and mask emails, phone numbers, SSNs,
credit-card numbers, IPv4 addresses, and IBANs over plain text.

Patterns are deliberately simple (no nested quantifiers → no catastrophic
backtracking). Pure standard library (``re`` + ``hashlib``); imports no
``PySide6``.

Headless API
------------

.. code-block:: python

from je_auto_control import detect_pii, redact_pii_text

detect_pii("mail a@b.com, ip 10.0.0.1")
# -> [PIIFinding(kind='email', value='a@b.com', start=5, end=12), ...]

redact_pii_text("contact a@b.com") # -> "contact [email]"
redact_pii_text("a@b.com", mode="mask") # -> "*******"
redact_pii_text("4111111111111111", mode="partial") # -> "************1111"
redact_pii_text("a@b.com", mode="hash") # -> "[email:fb98d44a]"

detect_pii(text, kinds=["email", "phone"]) # restrict detectors

``detect_pii`` returns non-overlapping findings sorted by position (the earlier,
then longer, match wins — so a credit-card number is not also flagged as a
phone). ``redact_pii_text`` modes: ``label`` (``[email]``), ``mask`` (``****``),
``partial`` (keep last 4), ``hash`` (``[email:<digest>]``).

Executor commands
-----------------

================================ ===================================================
Command Effect
================================ ===================================================
``AC_detect_pii`` ``{findings}`` — PII spans in text.
``AC_redact_pii`` ``{text}`` — text with PII masked.
================================ ===================================================

``kinds`` accepts a list or JSON-string list. The same operations are exposed as
MCP tools (``ac_detect_pii`` / ``ac_redact_pii``) and as Script Builder commands
under **Data**.
1 change: 1 addition & 0 deletions docs/source/Eng/eng_index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ Comprehensive guides for all AutoControl features.
doc/new_features/v52_features_doc
doc/new_features/v53_features_doc
doc/new_features/v54_features_doc
doc/new_features/v55_features_doc
doc/ocr_backends/ocr_backends_doc
doc/observability/observability_doc
doc/operations_layer/operations_layer_doc
Expand Down
44 changes: 44 additions & 0 deletions docs/source/Zh/doc/new_features/v55_features_doc.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
文字 PII 偵測與遮蔽
==================

影像遮蔽模組會在螢幕截圖中模糊 PII,但從 UI、OCR、剪貼簿、LLM 提示/回應或日誌行擷取的
*文字*卻沒有字串層級的對應 —— 因此 PII 可能洩漏進動作紀錄、稽核日誌或一次模型呼叫。
``detect_pii`` / ``redact_pii_text`` 可在純文字上找出並遮蔽電子郵件、電話號碼、SSN、信
用卡號、IPv4 位址與 IBAN。

樣式刻意保持簡單(無巢狀量詞 → 無災難性回溯)。純標準函式庫(``re`` + ``hashlib``);不匯
入 ``PySide6``。

無頭 API
--------

.. code-block:: python

from je_auto_control import detect_pii, redact_pii_text

detect_pii("mail a@b.com, ip 10.0.0.1")
# -> [PIIFinding(kind='email', value='a@b.com', start=5, end=12), ...]

redact_pii_text("contact a@b.com") # -> "contact [email]"
redact_pii_text("a@b.com", mode="mask") # -> "*******"
redact_pii_text("4111111111111111", mode="partial") # -> "************1111"
redact_pii_text("a@b.com", mode="hash") # -> "[email:fb98d44a]"

detect_pii(text, kinds=["email", "phone"]) # 限定偵測器

``detect_pii`` 回傳依位置排序、互不重疊的結果(較前、再較長的匹配勝出 —— 因此信用卡號不
會同時被標記為電話)。``redact_pii_text`` 模式:``label``(``[email]``)、``mask``
(``****``)、``partial``(保留末 4 碼)、``hash``(``[email:<digest>]``)。

執行器指令
----------

================================ ===================================================
指令 效果
================================ ===================================================
``AC_detect_pii`` ``{findings}`` —— 文字中的 PII 區段。
``AC_redact_pii`` ``{text}`` —— 已遮蔽 PII 的文字。
================================ ===================================================

``kinds`` 接受清單或 JSON 字串清單。相同操作亦提供為 MCP 工具(``ac_detect_pii`` /
``ac_redact_pii``),以及 Script Builder 中 **Data** 分類下的指令。
1 change: 1 addition & 0 deletions docs/source/Zh/zh_index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -77,6 +77,7 @@ AutoControl 所有功能的完整使用指南。
doc/new_features/v52_features_doc
doc/new_features/v53_features_doc
doc/new_features/v54_features_doc
doc/new_features/v55_features_doc
doc/ocr_backends/ocr_backends_doc
doc/observability/observability_doc
doc/operations_layer/operations_layer_doc
Expand Down
5 changes: 5 additions & 0 deletions je_auto_control/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -290,6 +290,10 @@
from je_auto_control.utils.locator_repair import (
RepairStore, RepairSuggestion, repair_from_heal,
)
# Text PII detection / redaction (free-text emails, phones, SSNs, cards, …)
from je_auto_control.utils.pii_text import (
PIIFinding, detect_pii, redact_pii_text,
)
# Background popup/interrupt watchdog (unattended automation)
from je_auto_control.utils.watchdog import (
PopupWatchdog, WatchdogRule, default_popup_watchdog,
Expand Down Expand Up @@ -756,6 +760,7 @@ def start_autocontrol_gui(*args, **kwargs):
"Saga", "SagaResult", "run_saga",
"DecisionTable", "Rule", "evaluate_table",
"RepairStore", "RepairSuggestion", "repair_from_heal",
"PIIFinding", "detect_pii", "redact_pii_text",
# MCP server
"AuditLogger", "HttpMCPServer", "MCPContent", "MCPPrompt",
"MCPPromptArgument", "MCPResource", "MCPServer", "MCPTool",
Expand Down
22 changes: 22 additions & 0 deletions je_auto_control/gui/script_builder/command_schema.py
Original file line number Diff line number Diff line change
Expand Up @@ -1202,6 +1202,28 @@ def _add_misc_specs(specs: List[CommandSpec]) -> None:
),
description="Approve a pending locator-repair suggestion.",
))
specs.append(CommandSpec(
"AC_detect_pii", "Data", "PII: Detect",
fields=(
FieldSpec("text", FieldType.STRING),
FieldSpec("kinds", FieldType.STRING, optional=True,
placeholder='["email", "phone"]'),
),
description="Detect PII spans (email/phone/ssn/card/ip/iban) in text.",
))
specs.append(CommandSpec(
"AC_redact_pii", "Data", "PII: Redact",
fields=(
FieldSpec("text", FieldType.STRING),
FieldSpec("kinds", FieldType.STRING, optional=True,
placeholder='["email"]'),
FieldSpec("mode", FieldType.ENUM, optional=True, default="label",
choices=("label", "mask", "partial", "hash")),
FieldSpec("mask_char", FieldType.STRING, optional=True,
default="*"),
),
description="Redact PII in text (label/mask/partial/hash).",
))
specs.append(CommandSpec(
"AC_generate_sop", "Report", "Generate SOP Document",
fields=(
Expand Down
19 changes: 19 additions & 0 deletions je_auto_control/utils/executor/action_executor.py
Original file line number Diff line number Diff line change
Expand Up @@ -3373,6 +3373,23 @@ def _repair_approve(suggestion_id: str,
return {"approved": RepairStore(db).approve(suggestion_id)}


def _detect_pii(text: str, kinds: Any = None) -> Dict[str, Any]:
"""Adapter: detect PII spans in text."""
from je_auto_control.utils.pii_text import detect_pii
findings = detect_pii(text, kinds=_coerce_list(kinds) if kinds else None)
return {"findings": [{"kind": f.kind, "value": f.value,
"start": f.start, "end": f.end} for f in findings]}


def _redact_pii(text: str, kinds: Any = None, mode: str = "label",
mask_char: str = "*") -> Dict[str, Any]:
"""Adapter: redact PII in text (label/mask/partial/hash)."""
from je_auto_control.utils.pii_text import redact_pii_text
return {"text": redact_pii_text(
text, kinds=_coerce_list(kinds) if kinds else None, mode=mode,
mask_char=mask_char)}


class Executor:
"""
Executor
Expand Down Expand Up @@ -3661,6 +3678,8 @@ def __init__(self):
"AC_repair_resolved": _repair_resolved,
"AC_repair_pending": _repair_pending,
"AC_repair_approve": _repair_approve,
"AC_detect_pii": _detect_pii,
"AC_redact_pii": _redact_pii,
"AC_a11y_record_start": _a11y_record_start,
"AC_a11y_record_stop": _a11y_record_stop,
"AC_a11y_record_events": _a11y_record_events,
Expand Down
30 changes: 30 additions & 0 deletions je_auto_control/utils/mcp_server/tools/_factories.py
Original file line number Diff line number Diff line change
Expand Up @@ -3309,6 +3309,35 @@ def locator_repair_tools() -> List[MCPTool]:
]


def pii_text_tools() -> List[MCPTool]:
_KINDS = {"type": "array", "items": {"type": "string"}}
return [
MCPTool(
name="ac_detect_pii",
description=("Detect PII spans (email/phone/ssn/credit_card/ipv4/"
"iban) in free text. Optional 'kinds' filter. Returns "
"{findings:[{kind, value, start, end}]}."),
input_schema=schema(
{"text": {"type": "string"}, "kinds": _KINDS}, ["text"]),
handler=h.detect_pii,
annotations=READ_ONLY,
),
MCPTool(
name="ac_redact_pii",
description=("Redact PII in text. 'mode' is label ([email]) / mask "
"(****) / partial (keep last 4) / hash. Returns "
"{text}."),
input_schema=schema(
{"text": {"type": "string"}, "kinds": _KINDS,
"mode": {"type": "string",
"enum": ["label", "mask", "partial", "hash"]},
"mask_char": {"type": "string"}}, ["text"]),
handler=h.redact_pii,
annotations=READ_ONLY,
),
]


def unattended_tools() -> List[MCPTool]:
return [
MCPTool(
Expand Down Expand Up @@ -4373,6 +4402,7 @@ def media_assert_tools() -> List[MCPTool]:
locale_tools, voice_tools, coordinate_space_tools, loop_guard_tools,
process_mining_tools, asset_tools, events_tools, notify_channel_tools,
jsonpath_tools, saga_tools, decision_table_tools, locator_repair_tools,
pii_text_tools,
screen_record_tools,
process_and_shell_tools, remote_desktop_tools, gamepad_tools,
usb_passthrough_tools, assertion_tools, data_source_tools,
Expand Down
13 changes: 13 additions & 0 deletions je_auto_control/utils/mcp_server/tools/_handlers.py
Original file line number Diff line number Diff line change
Expand Up @@ -1593,6 +1593,19 @@ def repair_approve(suggestion_id, db=None):
return {"approved": RepairStore(db).approve(suggestion_id)}


def detect_pii(text, kinds=None):
from je_auto_control.utils.pii_text import detect_pii as _detect
findings = _detect(text, kinds=kinds)
return {"findings": [{"kind": f.kind, "value": f.value,
"start": f.start, "end": f.end} for f in findings]}


def redact_pii(text, kinds=None, mode="label", mask_char="*"):
from je_auto_control.utils.pii_text import redact_pii_text
return {"text": redact_pii_text(text, kinds=kinds, mode=mode,
mask_char=mask_char)}


def vlm_locate(description: str,
screen_region: Optional[List[int]] = None,
model: Optional[str] = None) -> Optional[List[int]]:
Expand Down
6 changes: 6 additions & 0 deletions je_auto_control/utils/pii_text/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
"""Detect and redact PII in free text (emails, phones, SSNs, cards, IPs, IBANs)."""
from je_auto_control.utils.pii_text.pii_text import (
PII_KINDS, PIIFinding, detect_pii, redact_pii_text,
)

__all__ = ["PII_KINDS", "PIIFinding", "detect_pii", "redact_pii_text"]
Loading
Loading