Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions README/WHATS_NEW_zh-CN.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
# 本次更新 — AutoControl

## 本次更新 (2026-06-23) — 丰富剪贴板(HTML / CF_HTML)

把*格式化*的 HTML 复制粘贴到 Word / Outlook。完整参考:[`docs/source/Zh/doc/new_features/v144_features_doc.rst`](../docs/source/Zh/doc/new_features/v144_features_doc.rst)。

- **`build_cf_html` / `parse_cf_html` / `set_clipboard_html` / `get_clipboard_html`**(`AC_set_clipboard_html`、`AC_get_clipboard_html`):基础剪贴板只处理纯文字 + 图像——富文字粘贴需要 `CF_HTML`,其字节偏移标头(`StartHTML`/`EndHTML`/`StartFragment`/`EndFragment`)极易出错。`build_cf_html`/`parse_cf_html` 以纯 Python 计算与还原它(往返测试、多字节 UTF-8 正确);`set/get_clipboard_html` 将其包装于 Win32 剪贴板(含纯文字后备)。字节偏移运算可无头测试;只有 I/O 为 Windows。

## 本次更新 (2026-06-23) — 可串接 / 可过滤的候选定位器

用链式调用细化已定位的元素:`.within(panel).filter(has_text="Delete").nth(1)`。完整参考:[`docs/source/Zh/doc/new_features/v143_features_doc.rst`](../docs/source/Zh/doc/new_features/v143_features_doc.rst)。
Expand Down
6 changes: 6 additions & 0 deletions README/WHATS_NEW_zh-TW.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
# 本次更新 — AutoControl

## 本次更新 (2026-06-23) — 豐富剪貼簿(HTML / CF_HTML)

把*格式化*的 HTML 複製貼上到 Word / Outlook。完整參考:[`docs/source/Zh/doc/new_features/v144_features_doc.rst`](../docs/source/Zh/doc/new_features/v144_features_doc.rst)。

- **`build_cf_html` / `parse_cf_html` / `set_clipboard_html` / `get_clipboard_html`**(`AC_set_clipboard_html`、`AC_get_clipboard_html`):基礎剪貼簿只處理純文字 + 影像——富文字貼上需要 `CF_HTML`,其位元組偏移標頭(`StartHTML`/`EndHTML`/`StartFragment`/`EndFragment`)極易出錯。`build_cf_html`/`parse_cf_html` 以純 Python 計算與還原它(往返測試、多位元組 UTF-8 正確);`set/get_clipboard_html` 將其包裝於 Win32 剪貼簿(含純文字後備)。位元組偏移運算可無頭測試;只有 I/O 為 Windows。

## 本次更新 (2026-06-23) — 可串接 / 可過濾的候選定位器

用鏈式呼叫細化已定位的元素:`.within(panel).filter(has_text="Delete").nth(1)`。完整參考:[`docs/source/Zh/doc/new_features/v143_features_doc.rst`](../docs/source/Zh/doc/new_features/v143_features_doc.rst)。
Expand Down
6 changes: 6 additions & 0 deletions WHATS_NEW.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
# What's New — AutoControl

## What's new (2026-06-23) — Rich Clipboard (HTML / CF_HTML)

Copy and paste *formatted* HTML into Word / Outlook. Full reference: [`docs/source/Eng/doc/new_features/v144_features_doc.rst`](docs/source/Eng/doc/new_features/v144_features_doc.rst).

- **`build_cf_html` / `parse_cf_html` / `set_clipboard_html` / `get_clipboard_html`** (`AC_set_clipboard_html`, `AC_get_clipboard_html`): the base clipboard handles plain text + image only — rich paste needs `CF_HTML`, whose byte-offset header (`StartHTML`/`EndHTML`/`StartFragment`/`EndFragment`) is famously error-prone. `build_cf_html`/`parse_cf_html` compute and recover it in pure Python (round-trip tested, correct across multi-byte UTF-8); `set/get_clipboard_html` wrap them over the Win32 clipboard (with a plain-text fallback). Byte-offset math is headless-testable; only the I/O is Windows.

## What's new (2026-06-23) — Composable / Filtered Candidate Locators

Refine located elements with a chain: `.within(panel).filter(has_text="Delete").nth(1)`. Full reference: [`docs/source/Eng/doc/new_features/v143_features_doc.rst`](docs/source/Eng/doc/new_features/v143_features_doc.rst).
Expand Down
44 changes: 44 additions & 0 deletions docs/source/Eng/doc/new_features/v144_features_doc.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
Rich Clipboard — HTML (CF_HTML)
===============================

The base ``clipboard`` module handles plain text (``CF_UNICODETEXT``) and image
(``CF_DIB``) only. Pasting *formatted* content into Word / Outlook / a rich editor
needs the ``CF_HTML`` format, whose ``Version / StartHTML / EndHTML / StartFragment /
EndFragment`` **byte-offset** header is notoriously error-prone to build by hand.
``build_cf_html`` / ``parse_cf_html`` compute and recover that header in pure Python
(a fully unit-tested round-trip, correct across multi-byte UTF-8), and
``set_clipboard_html`` / ``get_clipboard_html`` wrap them over the Win32 clipboard.

The byte-offset math is platform-independent and headless-testable; only the actual
clipboard I/O is Windows (raising ``RuntimeError`` elsewhere, like the base module).
Imports no ``PySide6``.

Headless API
------------

.. code-block:: python

from je_auto_control import (build_cf_html, parse_cf_html,
set_clipboard_html, get_clipboard_html)

set_clipboard_html("<b>Bold</b> and <i>italic</i>",
fragment_plaintext="Bold and italic") # Windows
html = get_clipboard_html() # Windows

# The pure pieces work anywhere (e.g. to pre-build a payload):
payload = build_cf_html("<p>hello</p>") # bytes, valid CF_HTML
assert parse_cf_html(payload) == "<p>hello</p>"

``build_cf_html`` returns valid ``CF_HTML`` UTF-8 bytes whose offsets point exactly at
the fragment; ``parse_cf_html`` recovers the fragment from bytes or text (preferring
the comment markers, falling back to the byte offsets). ``set_clipboard_html`` also
seeds plain text via ``fragment_plaintext`` so apps that ignore HTML still paste
something.

Executor commands
-----------------

``AC_set_clipboard_html`` (``html`` / ``fragment_plaintext`` → ``{set, length}``) and
``AC_get_clipboard_html`` (→ ``{found, html}``). They are exposed as the MCP tools
``ac_set_clipboard_html`` / ``ac_get_clipboard_html`` and as Script Builder commands
under **Data**.
1 change: 1 addition & 0 deletions docs/source/Eng/eng_index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,7 @@ Comprehensive guides for all AutoControl features.
doc/new_features/v141_features_doc
doc/new_features/v142_features_doc
doc/new_features/v143_features_doc
doc/new_features/v144_features_doc
doc/ocr_backends/ocr_backends_doc
doc/observability/observability_doc
doc/operations_layer/operations_layer_doc
Expand Down
38 changes: 38 additions & 0 deletions docs/source/Zh/doc/new_features/v144_features_doc.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
豐富剪貼簿——HTML(CF_HTML)
=============================

基礎 ``clipboard`` 模組只處理純文字(``CF_UNICODETEXT``)與影像(``CF_DIB``)。將*格式化*內容貼進 Word /
Outlook / 富文字編輯器需要 ``CF_HTML`` 格式,其 ``Version / StartHTML / EndHTML / StartFragment /
EndFragment`` **位元組偏移**標頭以手寫極易出錯。``build_cf_html`` / ``parse_cf_html`` 以純 Python 計算與還原
該標頭(完整單元測試的往返,且在多位元組 UTF-8 下正確),而 ``set_clipboard_html`` / ``get_clipboard_html``
將其包裝於 Win32 剪貼簿之上。

位元組偏移運算與平台無關且可無頭測試;只有實際的剪貼簿 I/O 為 Windows(在其他平台丟出 ``RuntimeError``,
與基礎模組一致)。不匯入 ``PySide6``。

無頭 API
--------

.. code-block:: python

from je_auto_control import (build_cf_html, parse_cf_html,
set_clipboard_html, get_clipboard_html)

set_clipboard_html("<b>Bold</b> and <i>italic</i>",
fragment_plaintext="Bold and italic") # Windows
html = get_clipboard_html() # Windows

# 純函式在任何平台皆可用(例如預先建立 payload):
payload = build_cf_html("<p>hello</p>") # bytes,有效 CF_HTML
assert parse_cf_html(payload) == "<p>hello</p>"

``build_cf_html`` 回傳偏移精確指向片段的有效 ``CF_HTML`` UTF-8 位元組;``parse_cf_html`` 從 bytes 或文字還原片段
(優先用註解標記,退而用位元組偏移)。``set_clipboard_html`` 也會以 ``fragment_plaintext`` 設定純文字,讓忽略
HTML 的程式仍能貼上內容。

執行器命令
----------

``AC_set_clipboard_html``(``html`` / ``fragment_plaintext`` → ``{set, length}``)與 ``AC_get_clipboard_html``
(→ ``{found, html}``)。它們以 MCP 工具 ``ac_set_clipboard_html`` / ``ac_get_clipboard_html`` 以及 Script
Builder 中 **Data** 分類下的命令提供。
1 change: 1 addition & 0 deletions docs/source/Zh/zh_index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,7 @@ AutoControl 所有功能的完整使用指南。
doc/new_features/v141_features_doc
doc/new_features/v142_features_doc
doc/new_features/v143_features_doc
doc/new_features/v144_features_doc
doc/ocr_backends/ocr_backends_doc
doc/observability/observability_doc
doc/operations_layer/operations_layer_doc
Expand Down
8 changes: 8 additions & 0 deletions je_auto_control/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -335,6 +335,10 @@
)
# Composable / filtered candidate locators (chained-locator idiom)
from je_auto_control.utils.locator_chain import Candidates, from_boxes
# Rich clipboard formats — HTML (CF_HTML) build / parse / get / set
from je_auto_control.utils.rich_clipboard import (
build_cf_html, get_clipboard_html, parse_cf_html, set_clipboard_html,
)
# CI workflow annotations (GitHub Actions)
from je_auto_control.utils.ci_annotations import (
emit_annotations, format_annotation,
Expand Down Expand Up @@ -1187,6 +1191,10 @@ def start_autocontrol_gui(*args, **kwargs):
"to_be_stable",
"Candidates",
"from_boxes",
"build_cf_html",
"parse_cf_html",
"get_clipboard_html",
"set_clipboard_html",
"emit_annotations", "format_annotation",
"ClipboardHistory", "default_clipboard_history",
"analyze_heal_log", "heal_stats", "scan_secrets",
Expand Down
13 changes: 13 additions & 0 deletions je_auto_control/gui/script_builder/command_schema.py
Original file line number Diff line number Diff line change
Expand Up @@ -1044,6 +1044,19 @@ def _add_native_control_specs(specs: List[CommandSpec]) -> None:

def _add_misc_specs(specs: List[CommandSpec]) -> None:
_add_native_control_specs(specs)
specs.append(CommandSpec(
"AC_set_clipboard_html", "Data", "Set Clipboard HTML",
fields=(
FieldSpec("html", FieldType.STRING,
placeholder="<b>Bold</b> rich text"),
FieldSpec("fragment_plaintext", FieldType.STRING, optional=True),
),
description="Put rich HTML on the clipboard (CF_HTML, Windows).",
))
specs.append(CommandSpec(
"AC_get_clipboard_html", "Data", "Get Clipboard HTML",
description="Read the clipboard's HTML fragment (CF_HTML, Windows).",
))
specs.append(CommandSpec(
"AC_watchdog_add", "Flow", "Watchdog: Add Popup Rule",
fields=(
Expand Down
17 changes: 17 additions & 0 deletions je_auto_control/utils/executor/action_executor.py
Original file line number Diff line number Diff line change
Expand Up @@ -3661,6 +3661,21 @@ def _locate_chain(boxes: Any, ops: Any = None) -> Dict[str, Any]:
"center": candidates.center()}


def _set_clipboard_html(html: str, fragment_plaintext: Any = None
) -> Dict[str, Any]:
"""Adapter: put an HTML fragment on the clipboard as CF_HTML (Windows)."""
from je_auto_control.utils.rich_clipboard import set_clipboard_html
set_clipboard_html(str(html), fragment_plaintext=fragment_plaintext)
return {"set": True, "length": len(str(html))}


def _get_clipboard_html() -> Dict[str, Any]:
"""Adapter: read the clipboard's HTML fragment (Windows)."""
from je_auto_control.utils.rich_clipboard import get_clipboard_html
html = get_clipboard_html()
return {"found": html is not None, "html": html}


def _with_modifiers(modifiers: Any, actions: Any) -> Dict[str, Any]:
"""Adapter: run nested actions while modifier keys are held down."""
import json
Expand Down Expand Up @@ -5402,6 +5417,8 @@ def __init__(self):
"AC_find_separators": _find_separators,
"AC_expect_poll": _expect_poll,
"AC_locate_chain": _locate_chain,
"AC_set_clipboard_html": _set_clipboard_html,
"AC_get_clipboard_html": _get_clipboard_html,
"AC_tile_rect": _tile_rect,
"AC_grid_rects": _grid_rects,
"AC_cascade_rects": _cascade_rects,
Expand Down
28 changes: 27 additions & 1 deletion je_auto_control/utils/mcp_server/tools/_factories.py
Original file line number Diff line number Diff line change
Expand Up @@ -3060,6 +3060,32 @@ def locator_chain_tools() -> List[MCPTool]:
]


def rich_clipboard_tools() -> List[MCPTool]:
return [
MCPTool(
name="ac_set_clipboard_html",
description=("Put an HTML fragment on the clipboard as CF_HTML for rich "
"paste into Word / Outlook / rich editors (Windows). "
"'fragment_plaintext' is also set as plain text. Returns "
"{set, length}."),
input_schema=schema({
"html": {"type": "string"},
"fragment_plaintext": {"type": "string"}},
required=["html"]),
handler=h.set_clipboard_html,
annotations=SIDE_EFFECT_ONLY,
),
MCPTool(
name="ac_get_clipboard_html",
description=("Read the clipboard's HTML fragment (CF_HTML, Windows). "
"Returns {found, html}."),
input_schema=schema({}, required=[]),
handler=h.get_clipboard_html,
annotations=READ_ONLY,
),
]


def ssim_tools() -> List[MCPTool]:
return [
MCPTool(
Expand Down Expand Up @@ -6565,7 +6591,7 @@ def media_assert_tools() -> List[MCPTool]:
window_layout_tools, window_arrange_tools, preprocess_tools,
monitor_layout_tools, actionability_tools, element_parse_tools,
hsv_segment_tools, text_regions_tools, edge_lines_tools, expect_poll_tools,
locator_chain_tools, plugin_sdk_tools, governance_tools,
locator_chain_tools, rich_clipboard_tools, plugin_sdk_tools, governance_tools,
credential_lease_tools, egress_tools, approval_testing_tools,
trajectory_eval_tools, compliance_tools, agent_trace_tools,
video_report_tools, fuzzy_tools, artifact_store_tools, image_dedup_tools,
Expand Down
10 changes: 10 additions & 0 deletions je_auto_control/utils/mcp_server/tools/_handlers.py
Original file line number Diff line number Diff line change
Expand Up @@ -2237,6 +2237,16 @@ def locate_chain(boxes, ops=None):
return _locate_chain(boxes, ops)


def set_clipboard_html(html, fragment_plaintext=None):
from je_auto_control.utils.executor.action_executor import _set_clipboard_html
return _set_clipboard_html(html, fragment_plaintext)


def get_clipboard_html():
from je_auto_control.utils.executor.action_executor import _get_clipboard_html
return _get_clipboard_html()


def detect_drift(reference, current, threshold=0.25, bins=10):
from je_auto_control.utils.executor.action_executor import _detect_drift
return _detect_drift(reference, current, threshold, bins)
Expand Down
7 changes: 7 additions & 0 deletions je_auto_control/utils/rich_clipboard/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@
"""Rich clipboard formats — HTML (CF_HTML) build / parse / get / set."""
from je_auto_control.utils.rich_clipboard.rich_clipboard import (
build_cf_html, get_clipboard_html, parse_cf_html, set_clipboard_html,
)

__all__ = ["build_cf_html", "get_clipboard_html", "parse_cf_html",
"set_clipboard_html"]
Loading
Loading