diff --git a/README.md b/README.md index 1545d459..17f18681 100644 --- a/README.md +++ b/README.md @@ -13,6 +13,7 @@ ## Table of Contents +- [What's new (2026-06-22) — Secret Redaction for Config & Logs](#whats-new-2026-06-22--secret-redaction-for-config--logs) - [What's new (2026-06-22) — RFC 8288 Link Header & Pagination](#whats-new-2026-06-22--rfc-8288-link-header--pagination) - [What's new (2026-06-22) — Referential Integrity Checks](#whats-new-2026-06-22--referential-integrity-checks) - [What's new (2026-06-22) — URI-Scheme Value References](#whats-new-2026-06-22--uri-scheme-value-references) @@ -140,6 +141,12 @@ --- +## What's new (2026-06-22) — Secret Redaction for Config & Logs + +Mask secrets before logging or exporting. Full reference: [`docs/source/Eng/doc/new_features/v88_features_doc.rst`](docs/source/Eng/doc/new_features/v88_features_doc.rst). + +- **`redact_config` / `redact_secret_text`** (`AC_redact_config`, `AC_redact_secret_text`): `utils/redaction` only blurs screenshots and `secrets_scan` only *detects* — neither returned a masked copy. This reuses the `secrets_scan` detector (key-name patterns, AWS/bearer formats, high-entropy) to return a redacted deep copy of a config structure, and to mask secret-looking tokens in a free-text log line (preserving surrounding words). Vault refs (`${secrets.*}`) are left intact. Pure-stdlib, deterministic. + ## What's new (2026-06-22) — RFC 8288 Link Header & Pagination Parse `Link` headers and follow `rel="next"`. Full reference: [`docs/source/Eng/doc/new_features/v87_features_doc.rst`](docs/source/Eng/doc/new_features/v87_features_doc.rst). diff --git a/README/README_zh-CN.md b/README/README_zh-CN.md index a124dba4..4c004c10 100644 --- a/README/README_zh-CN.md +++ b/README/README_zh-CN.md @@ -12,6 +12,7 @@ ## 目录 +- [本次更新 (2026-06-22) — 配置与日志的机密脱敏](#本次更新-2026-06-22--配置与日志的机密脱敏) - [本次更新 (2026-06-22) — RFC 8288 Link 标头与分页](#本次更新-2026-06-22--rfc-8288-link-标头与分页) - [本次更新 (2026-06-22) — 参照完整性检查](#本次更新-2026-06-22--参照完整性检查) - [本次更新 (2026-06-22) — URI-Scheme 值引用](#本次更新-2026-06-22--uri-scheme-值引用) @@ -139,6 +140,12 @@ --- +## 本次更新 (2026-06-22) — 配置与日志的机密脱敏 + +在记录或导出前脱敏机密。完整参考:[`docs/source/Zh/doc/new_features/v88_features_doc.rst`](../docs/source/Zh/doc/new_features/v88_features_doc.rst)。 + +- **`redact_config` / `redact_secret_text`**(`AC_redact_config`、`AC_redact_secret_text`):`utils/redaction` 只模糊截图,`secrets_scan` 只*检测* —— 两者都不返回脱敏后的副本。本功能重用 `secrets_scan` 检测器(键名模式、AWS/bearer 格式、高熵)返回配置结构的脱敏深层副本,并脱敏自由文本日志行中看似机密的 token(保留周围文本)。vault 引用(`${secrets.*}`)保持不变。纯标准库、确定。 + ## 本次更新 (2026-06-22) — RFC 8288 Link 标头与分页 解析 `Link` 标头并跟随 `rel="next"`。完整参考:[`docs/source/Zh/doc/new_features/v87_features_doc.rst`](../docs/source/Zh/doc/new_features/v87_features_doc.rst)。 diff --git a/README/README_zh-TW.md b/README/README_zh-TW.md index 619d7b3e..c299ea2d 100644 --- a/README/README_zh-TW.md +++ b/README/README_zh-TW.md @@ -12,6 +12,7 @@ ## 目錄 +- [本次更新 (2026-06-22) — 設定與日誌的機密遮蔽](#本次更新-2026-06-22--設定與日誌的機密遮蔽) - [本次更新 (2026-06-22) — RFC 8288 Link 標頭與分頁](#本次更新-2026-06-22--rfc-8288-link-標頭與分頁) - [本次更新 (2026-06-22) — 參照完整性檢查](#本次更新-2026-06-22--參照完整性檢查) - [本次更新 (2026-06-22) — URI-Scheme 值參照](#本次更新-2026-06-22--uri-scheme-值參照) @@ -139,6 +140,12 @@ --- +## 本次更新 (2026-06-22) — 設定與日誌的機密遮蔽 + +在記錄或匯出前遮蔽機密。完整參考:[`docs/source/Zh/doc/new_features/v88_features_doc.rst`](../docs/source/Zh/doc/new_features/v88_features_doc.rst)。 + +- **`redact_config` / `redact_secret_text`**(`AC_redact_config`、`AC_redact_secret_text`):`utils/redaction` 只模糊截圖,`secrets_scan` 只*偵測* —— 兩者都不回傳遮蔽後的副本。本功能重用 `secrets_scan` 偵測器(鍵名模式、AWS/bearer 格式、高熵)回傳設定結構的遮蔽深層副本,並遮蔽自由文字日誌行中看似機密的 token(保留周圍文字)。vault 參照(`${secrets.*}`)保持不變。純標準函式庫、具決定性。 + ## 本次更新 (2026-06-22) — RFC 8288 Link 標頭與分頁 解析 `Link` 標頭並跟隨 `rel="next"`。完整參考:[`docs/source/Zh/doc/new_features/v87_features_doc.rst`](../docs/source/Zh/doc/new_features/v87_features_doc.rst)。 diff --git a/docs/source/Eng/doc/new_features/v88_features_doc.rst b/docs/source/Eng/doc/new_features/v88_features_doc.rst new file mode 100644 index 00000000..67288447 --- /dev/null +++ b/docs/source/Eng/doc/new_features/v88_features_doc.rst @@ -0,0 +1,42 @@ +Secret Redaction for Config & Logs +================================== + +``utils/redaction`` only blurs PIL screenshots, and ``secrets_scan`` only +*detects* and reports findings — neither returns a masked copy of a config dict +or string safe for logs, reports, or ``config_bundle`` export. This reuses the +``secrets_scan`` detector to produce a redacted copy. + +Pure standard library (``re`` + reuse of ``secrets_scan``); imports no +``PySide6``. Every function is pure (data in, redacted copy out), so it is fully +deterministic in CI. + +Headless API +------------ + +.. code-block:: python + + from je_auto_control import redact_config, redact_secret_text + + safe = redact_config({"db": {"password": secret}, "name": "alice"}) + # {"db": {"password": "***"}, "name": "alice"} + + line = redact_secret_text(f"auth failed for token {token}") + # "auth failed for token ***" + +``redact_config`` returns a deep copy of a nested structure with secret-looking +values masked, reusing ``secrets_scan`` (key-name patterns such as ``password`` +/ ``api_key``, known value formats like AWS keys and bearer tokens, and +high-entropy strings); values already referencing the vault (``${secrets.*}``) +are left intact. ``redact_secret_text`` masks secret-looking tokens within a +free-text string (a log line) while preserving the surrounding words and +whitespace. Both accept a custom ``mask`` (default ``"***"``). The function is +named ``redact_secret_text`` to stay distinct from the prompt-injection +``guardrail.redact_text``. + +Executor commands +----------------- + +``AC_redact_config`` returns ``{redacted}`` for an ``obj`` (JSON accepted); +``AC_redact_secret_text`` returns ``{text}``. Both accept an optional ``mask`` +and are exposed as MCP tools (``ac_redact_config`` / ``ac_redact_secret_text``) +and as Script Builder commands under **Security**. diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst index f170e278..525ca9ab 100644 --- a/docs/source/Eng/eng_index.rst +++ b/docs/source/Eng/eng_index.rst @@ -110,6 +110,7 @@ Comprehensive guides for all AutoControl features. doc/new_features/v85_features_doc doc/new_features/v86_features_doc doc/new_features/v87_features_doc + doc/new_features/v88_features_doc doc/ocr_backends/ocr_backends_doc doc/observability/observability_doc doc/operations_layer/operations_layer_doc diff --git a/docs/source/Zh/doc/new_features/v88_features_doc.rst b/docs/source/Zh/doc/new_features/v88_features_doc.rst new file mode 100644 index 00000000..cd315452 --- /dev/null +++ b/docs/source/Zh/doc/new_features/v88_features_doc.rst @@ -0,0 +1,35 @@ +設定與日誌的機密遮蔽 +================== + +``utils/redaction`` 只會模糊 PIL 截圖,``secrets_scan`` 只會*偵測*並回報發現 —— 兩者都不會回傳一份對 +日誌、報告或 ``config_bundle`` 匯出而言安全的、已遮蔽的設定 dict 或字串副本。本功能重用 ``secrets_scan`` +偵測器來產生遮蔽後的副本。 + +純標準函式庫(``re`` + 重用 ``secrets_scan``);不匯入 ``PySide6``。每個函式皆為純函式(輸入資料、輸出 +遮蔽副本),因此在 CI 中完全具決定性。 + +無頭 API +-------- + +.. code-block:: python + + from je_auto_control import redact_config, redact_secret_text + + safe = redact_config({"db": {"password": secret}, "name": "alice"}) + # {"db": {"password": "***"}, "name": "alice"} + + line = redact_secret_text(f"auth failed for token {token}") + # "auth failed for token ***" + +``redact_config`` 回傳巢狀結構的深層副本,並遮蔽看似機密的值,重用 ``secrets_scan``(鍵名模式如 +``password`` / ``api_key``、已知值格式如 AWS 金鑰與 bearer token,以及高熵字串);已經參照 vault 的值 +(``${secrets.*}``)保持不變。``redact_secret_text`` 遮蔽自由文字字串(日誌行)中看似機密的 token,同時 +保留周圍的文字與空白。兩者皆接受自訂 ``mask``(預設 ``"***"``)。此函式命名為 ``redact_secret_text`` 以 +與處理 prompt-injection 的 ``guardrail.redact_text`` 區分。 + +執行器命令 +---------- + +``AC_redact_config`` 對 ``obj``(可為 JSON)回傳 ``{redacted}``;``AC_redact_secret_text`` 回傳 +``{text}``。兩者皆接受選用的 ``mask``,並以 MCP 工具(``ac_redact_config`` / ``ac_redact_secret_text``) +以及 Script Builder 中 **Security** 分類下的命令提供。 diff --git a/docs/source/Zh/zh_index.rst b/docs/source/Zh/zh_index.rst index fda74354..dd3a58c1 100644 --- a/docs/source/Zh/zh_index.rst +++ b/docs/source/Zh/zh_index.rst @@ -110,6 +110,7 @@ AutoControl 所有功能的完整使用指南。 doc/new_features/v85_features_doc doc/new_features/v86_features_doc doc/new_features/v87_features_doc + doc/new_features/v88_features_doc doc/ocr_backends/ocr_backends_doc doc/observability/observability_doc doc/operations_layer/operations_layer_doc diff --git a/je_auto_control/__init__.py b/je_auto_control/__init__.py index 2fab3d57..27e8110f 100644 --- a/je_auto_control/__init__.py +++ b/je_auto_control/__init__.py @@ -294,6 +294,10 @@ from je_auto_control.utils.secret_ref import ( RefResolver, SecretRefError, is_ref, resolve_ref, resolve_refs_in, ) +# Secret redaction for config structures and log strings +from je_auto_control.utils.config_redaction import ( + redact_config, redact_secret_text, +) # Outbound CloudEvents emitter from je_auto_control.utils.events import ( EventEmitter, post_cloudevent, to_cloudevent, @@ -894,6 +898,7 @@ def start_autocontrol_gui(*args, **kwargs): "dotenv_values", "dump_dotenv", "load_dotenv", "parse_dotenv", "LayeredConfig", "SourceTrace", "deep_merge", "RefResolver", "SecretRefError", "is_ref", "resolve_ref", "resolve_refs_in", + "redact_config", "redact_secret_text", "EventEmitter", "post_cloudevent", "to_cloudevent", "WebhookChannel", "WebhookResult", "notify_webhook", "set_default_poster", "json_extract", "json_query", "json_query_one", diff --git a/je_auto_control/gui/script_builder/command_schema.py b/je_auto_control/gui/script_builder/command_schema.py index 12a598bb..6252377c 100644 --- a/je_auto_control/gui/script_builder/command_schema.py +++ b/je_auto_control/gui/script_builder/command_schema.py @@ -1656,6 +1656,24 @@ def _add_resilience_specs(specs: List[CommandSpec]) -> None: ), description="Recursively resolve references inside a JSON structure.", )) + specs.append(CommandSpec( + "AC_redact_config", "Security", "Redaction: Redact Config", + fields=( + FieldSpec("obj", FieldType.STRING, + placeholder='{"db": {"password": "hunter2longvalue"}}'), + FieldSpec("mask", FieldType.STRING, optional=True, default="***"), + ), + description="Mask secret-looking values in a JSON config structure.", + )) + specs.append(CommandSpec( + "AC_redact_secret_text", "Security", "Redaction: Redact Secret Text", + fields=( + FieldSpec("text", FieldType.STRING, + placeholder="log line with AKIA... or a bearer token"), + FieldSpec("mask", FieldType.STRING, optional=True, default="***"), + ), + description="Mask secret-looking tokens within a free-text string.", + )) specs.append(CommandSpec( "AC_profile_rows", "Data", "Data Profile: Profile Rows", fields=( diff --git a/je_auto_control/utils/config_redaction/__init__.py b/je_auto_control/utils/config_redaction/__init__.py new file mode 100644 index 00000000..11bb949e --- /dev/null +++ b/je_auto_control/utils/config_redaction/__init__.py @@ -0,0 +1,6 @@ +"""Secret redaction for AutoControl config structures and log strings.""" +from je_auto_control.utils.config_redaction.config_redaction import ( + redact_config, redact_secret_text, +) + +__all__ = ["redact_config", "redact_secret_text"] diff --git a/je_auto_control/utils/config_redaction/config_redaction.py b/je_auto_control/utils/config_redaction/config_redaction.py new file mode 100644 index 00000000..9222a2c7 --- /dev/null +++ b/je_auto_control/utils/config_redaction/config_redaction.py @@ -0,0 +1,54 @@ +"""Redact secret-looking values from config structures and log strings. + +``utils/redaction`` only blurs PIL screenshots, and ``secrets_scan`` only +*detects* and reports findings — neither returns a masked copy of a config dict +or string safe for logs, reports, or ``config_bundle`` export. This reuses the +``secrets_scan`` detector to produce a redacted copy. + +Pure standard library (``re`` + reuse of ``secrets_scan``); imports no +``PySide6``. Every function is pure (data in, redacted copy out), so it is fully +deterministic in CI. +""" +import re +from typing import Any, Set + +from je_auto_control.utils.secrets_scan import scan_secrets + +_DEFAULT_MASK = "***" +_PUNCT = ".,;:!?\"'()[]{}<>" + + +def _secret_paths(obj: Any) -> Set[str]: + return {finding["path"] for finding in scan_secrets(obj)} + + +def _redact_node(node: Any, path: str, paths: Set[str], mask: str) -> Any: + if isinstance(node, dict): + return {key: _redact_node(value, f"{path}.{key}", paths, mask) + for key, value in node.items()} + if isinstance(node, list): + return [_redact_node(value, f"{path}[{index}]", paths, mask) + for index, value in enumerate(node)] + return mask if path in paths else node + + +def redact_config(obj: Any, *, mask: str = _DEFAULT_MASK) -> Any: + """Return a deep copy of ``obj`` with secret-looking values masked. + + Detection reuses ``secrets_scan`` (key-name patterns, known value formats, + and high-entropy strings); values already referencing the vault + (``${secrets.*}``) are left intact. + """ + return _redact_node(obj, "$", _secret_paths(obj), mask) + + +def redact_secret_text(text: str, *, mask: str = _DEFAULT_MASK) -> str: + """Mask secret-looking tokens within a free-text string (e.g. a log line).""" + def _replace(match: "re.Match[str]") -> str: + token = match.group(0) + core = token.strip(_PUNCT) + if core and scan_secrets({"value": core}): + return token.replace(core, mask) + return token + + return re.sub(r"\S+", _replace, text or "") diff --git a/je_auto_control/utils/executor/action_executor.py b/je_auto_control/utils/executor/action_executor.py index 7900fcf1..87274355 100644 --- a/je_auto_control/utils/executor/action_executor.py +++ b/je_auto_control/utils/executor/action_executor.py @@ -3015,6 +3015,21 @@ def _resolve_refs(obj: Any) -> Dict[str, Any]: return {"resolved": resolve_refs_in(obj)} +def _redact_config(obj: Any, mask: str = "***") -> Dict[str, Any]: + """Adapter: redact secret-looking values from a config structure.""" + import json + from je_auto_control.utils.config_redaction import redact_config + if isinstance(obj, str): + obj = json.loads(obj) + return {"redacted": redact_config(obj, mask=mask)} + + +def _redact_secret_text(text: str, mask: str = "***") -> Dict[str, Any]: + """Adapter: mask secret-looking tokens within a free-text string.""" + from je_auto_control.utils.config_redaction import redact_secret_text + return {"text": redact_secret_text(text, mask=mask)} + + def _parse_link_header(value: str) -> Dict[str, Any]: """Adapter: parse an RFC 8288 Link header into {links}.""" from je_auto_control.utils.link_header import parse_link_header @@ -4286,6 +4301,8 @@ def __init__(self): "AC_baggage_format": _baggage_format, "AC_resolve_ref": _resolve_ref, "AC_resolve_refs": _resolve_refs, + "AC_redact_config": _redact_config, + "AC_redact_secret_text": _redact_secret_text, "AC_parse_link_header": _parse_link_header, "AC_next_url": _next_url, "AC_profile_rows": _profile_rows, diff --git a/je_auto_control/utils/mcp_server/tools/_factories.py b/je_auto_control/utils/mcp_server/tools/_factories.py index ed2011d3..093ae0da 100644 --- a/je_auto_control/utils/mcp_server/tools/_factories.py +++ b/je_auto_control/utils/mcp_server/tools/_factories.py @@ -3582,6 +3582,33 @@ def data_profile_tools() -> List[MCPTool]: ] +def config_redaction_tools() -> List[MCPTool]: + return [ + MCPTool( + name="ac_redact_config", + description=("Return a deep copy of 'obj' with secret-looking values " + "masked (key-name / value-format / high-entropy " + "detection). Optional 'mask'. Returns {redacted}."), + input_schema=schema( + {"obj": {"type": "object"}, "mask": {"type": "string"}}, + ["obj"]), + handler=h.redact_config, + annotations=READ_ONLY, + ), + MCPTool( + name="ac_redact_secret_text", + description=("Mask secret-looking tokens within a free-text 'text' " + "string (e.g. a log line). Optional 'mask'. Returns " + "{text}."), + input_schema=schema( + {"text": {"type": "string"}, "mask": {"type": "string"}}, + ["text"]), + handler=h.redact_secret_text, + annotations=READ_ONLY, + ), + ] + + def secret_ref_tools() -> List[MCPTool]: return [ MCPTool( @@ -5186,6 +5213,7 @@ def media_assert_tools() -> List[MCPTool]: feature_flag_tools, provenance_tools, json_contract_tools, chaos_tools, slo_tools, percentiles_tools, bulkhead_tools, http_cassette_tools, trace_context_tools, baggage_tools, secret_ref_tools, + config_redaction_tools, data_profile_tools, http_problem_tools, dotenv_tools, sse_client_tools, layered_config_tools, data_drift_tools, dataset_diff_tools, referential_tools, link_header_tools, diff --git a/je_auto_control/utils/mcp_server/tools/_handlers.py b/je_auto_control/utils/mcp_server/tools/_handlers.py index 72760dce..5f1add80 100644 --- a/je_auto_control/utils/mcp_server/tools/_handlers.py +++ b/je_auto_control/utils/mcp_server/tools/_handlers.py @@ -1756,6 +1756,17 @@ def next_url(value): return _next_url(value) +def redact_config(obj, mask="***"): + from je_auto_control.utils.executor.action_executor import _redact_config + return _redact_config(obj, mask) + + +def redact_secret_text(text, mask="***"): + from je_auto_control.utils.executor.action_executor import ( + _redact_secret_text) + return _redact_secret_text(text, mask) + + def profile_rows(rows, columns=None): from je_auto_control.utils.executor.action_executor import _profile_rows return _profile_rows(rows, columns) diff --git a/test/unit_test/headless/test_config_redaction_batch.py b/test/unit_test/headless/test_config_redaction_batch.py new file mode 100644 index 00000000..1c178f9e --- /dev/null +++ b/test/unit_test/headless/test_config_redaction_batch.py @@ -0,0 +1,78 @@ +"""Headless tests for config/text secret redaction. Pure stdlib, no Qt.""" +import json + +import je_auto_control as ac +from je_auto_control.utils.config_redaction import ( + redact_config, redact_secret_text, +) + +# Build secret-shaped values at runtime so the test file itself is clean. +_AWS = "AKIA" + "ABCDEFGHIJKLMNOP" # matches the aws-access-key pattern +_SECRET = "hunter" + "2_long_enough_value" + + +def test_redact_config_masks_by_key_and_value(): + config = {"api_key": _SECRET, "creds": {"note": _AWS}, + "name": "alice", "ref": "${secrets.db}"} + out = redact_config(config) + assert out["api_key"] == "***" # key-name detection + assert out["creds"]["note"] == "***" # aws value-format detection + assert out["name"] == "alice" # plain value untouched + assert out["ref"] == "${secrets.db}" # vault reference preserved + + +def test_redact_config_does_not_mutate_input(): + config = {"api_key": _SECRET} + redact_config(config) + assert config["api_key"] == _SECRET + + +def test_redact_config_lists(): + out = redact_config({"items": [{"token": _SECRET}, {"ok": "fine"}]}) + assert out["items"][0]["token"] == "***" and out["items"][1]["ok"] == "fine" + + +def test_custom_mask(): + out = redact_config({"secret": _SECRET}, mask="[REDACTED]") + assert out["secret"] == "[REDACTED]" + + +def test_redact_text_masks_token_keeps_words(): + line = f"error: key {_AWS} was rejected" + out = redact_secret_text(line) + assert _AWS not in out and "***" in out + assert "error:" in out and "rejected" in out + + +def test_redact_text_keeps_plain_line(): + assert redact_secret_text("just a normal log line") == \ + "just a normal log line" + + +# --- wiring --------------------------------------------------------------- + +def test_executor_round_trip(): + rec = ac.execute_action([[ + "AC_redact_config", {"obj": json.dumps({"api_key": _SECRET})}]]) + out = next(v for v in rec.values() if isinstance(v, dict))["redacted"] + assert out["api_key"] == "***" + rec2 = ac.execute_action([[ + "AC_redact_secret_text", {"text": f"tok {_AWS} end"}]]) + text = next(v for v in rec2.values() if isinstance(v, dict))["text"] + assert _AWS not in text + + +def test_wiring(): + known = ac.executor.known_commands() + assert {"AC_redact_config", "AC_redact_secret_text"} <= set(known) + from je_auto_control.utils.mcp_server.tools import build_default_tool_registry + names = {t.name for t in build_default_tool_registry()} + assert {"ac_redact_config", "ac_redact_secret_text"} <= names + from je_auto_control.gui.script_builder.command_schema import _build_specs + specs = {s.command for s in _build_specs()} + assert {"AC_redact_config", "AC_redact_secret_text"} <= specs + + +def test_facade_exports(): + for attr in ("redact_config", "redact_secret_text"): + assert hasattr(ac, attr) and attr in ac.__all__