Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions README/WHATS_NEW_zh-CN.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
# 本次更新 — AutoControl

## 本次更新 (2026-06-23) — 多显示器 / 虚拟桌面几何

在多台显示器间正确摆放窗口与坐标。完整参考:[`docs/source/Zh/doc/new_features/v136_features_doc.rst`](../docs/source/Zh/doc/new_features/v136_features_doc.rst)。

- **`enumerate_monitors` + `Monitor` / `virtual_bounds` / `monitor_at_point` / `monitor_for_window` / `to_local` / `to_virtual` / `remap_point`**(`AC_enumerate_monitors`、`AC_monitor_at_point`):`snap_window` / `arrange_grid` / 版面规划器都假设单一主屏 `(width, height)`——对多显示器无感,无法在第二台显示器铺排或处理负原点虚拟桌面。本功能补上实体层:并集虚拟边界、某点 / 某窗口属于哪台显示器、虚拟↔显示器区域坐标转换,以及跨分辨率 / DPI 的等效位置重映射。对 `Monitor` dataclass 的纯几何 → 完全无头可测;`enumerate_monitors` 具可注入 provider(默认 `mss`)。

## 本次更新 (2026-06-23) — 图像预处理(供 OCR / 模板匹配)

在识别或匹配前先清理画面。完整参考:[`docs/source/Zh/doc/new_features/v135_features_doc.rst`](../docs/source/Zh/doc/new_features/v135_features_doc.rst)。
Expand Down
6 changes: 6 additions & 0 deletions README/WHATS_NEW_zh-TW.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
# 本次更新 — AutoControl

## 本次更新 (2026-06-23) — 多螢幕 / 虛擬桌面幾何

在多台顯示器間正確擺放視窗與座標。完整參考:[`docs/source/Zh/doc/new_features/v136_features_doc.rst`](../docs/source/Zh/doc/new_features/v136_features_doc.rst)。

- **`enumerate_monitors` + `Monitor` / `virtual_bounds` / `monitor_at_point` / `monitor_for_window` / `to_local` / `to_virtual` / `remap_point`**(`AC_enumerate_monitors`、`AC_monitor_at_point`):`snap_window` / `arrange_grid` / 版面規劃器都假設單一主螢幕 `(width, height)`——對多螢幕無感,無法在第二台顯示器鋪排或處理負原點虛擬桌面。本功能補上實體層:聯集虛擬邊界、某點 / 某視窗屬於哪台螢幕、虛擬↔螢幕區域座標轉換,以及跨解析度 / DPI 的等效位置重映射。對 `Monitor` dataclass 的純幾何 → 完全無頭可測;`enumerate_monitors` 具可注入 provider(預設 `mss`)。

## 本次更新 (2026-06-23) — 影像前處理(供 OCR / 模板比對)

在辨識或比對前先清理畫面。完整參考:[`docs/source/Zh/doc/new_features/v135_features_doc.rst`](../docs/source/Zh/doc/new_features/v135_features_doc.rst)。
Expand Down
6 changes: 6 additions & 0 deletions WHATS_NEW.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
# What's New — AutoControl

## What's new (2026-06-23) — Multi-Monitor / Virtual-Desktop Geometry

Place windows and points correctly across several displays. Full reference: [`docs/source/Eng/doc/new_features/v136_features_doc.rst`](docs/source/Eng/doc/new_features/v136_features_doc.rst).

- **`enumerate_monitors` + `Monitor` / `virtual_bounds` / `monitor_at_point` / `monitor_for_window` / `to_local` / `to_virtual` / `remap_point`** (`AC_enumerate_monitors`, `AC_monitor_at_point`): `snap_window` / `arrange_grid` / the layout planner all assumed a single primary `(width, height)` — monitor-blind, unable to tile on a second display or handle a negative-origin virtual desktop. This adds the physical layer: union virtual bounds, which-monitor-owns-this-point/window, virtual↔monitor-local conversion, and equivalent-spot remapping across resolutions/DPI. Pure geometry over `Monitor` dataclasses → fully headless-testable; `enumerate_monitors` has an injectable provider (default `mss`).

## What's new (2026-06-23) — Image Pre-processing for OCR / Template Matching

Clean up the screen before reading or matching it. Full reference: [`docs/source/Eng/doc/new_features/v135_features_doc.rst`](docs/source/Eng/doc/new_features/v135_features_doc.rst).
Expand Down
46 changes: 46 additions & 0 deletions docs/source/Eng/doc/new_features/v136_features_doc.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
Multi-Monitor / Virtual-Desktop Geometry
========================================

``snap_window``, ``arrange_grid`` and the layout planner all take a single primary
``(width, height)`` — they are monitor-blind: they cannot tile on the second display
or cope with a negative-origin virtual desktop, and ``coordinate_space`` only rescales
a model grid. This adds the missing physical layer: enumerate the monitors, compute
the union virtual bounds, ask which monitor contains a point or a window, convert
between virtual and per-monitor-local coordinates, and remap a point to the equivalent
spot on another display.

The geometry is pure arithmetic over plain ``Monitor`` dataclasses, so it is fully
unit-testable; only ``enumerate_monitors``' default provider touches the OS (via
``mss``) and it is injectable. Imports no ``PySide6``.

Headless API
------------

.. code-block:: python

from je_auto_control import (enumerate_monitors, monitor_at_point,
virtual_bounds, to_local, remap_point)

monitors = enumerate_monitors()
print(virtual_bounds(monitors)) # (x, y, w, h) spanning all displays

here = monitor_at_point(monitors, x, y) # which monitor owns this point
idx, lx, ly = to_local(monitors, x, y) # virtual -> (monitor, local x, local y)

# Move a point to the equivalent relative spot on another monitor.
second = remap_point(monitors[0], monitors[1], 960, 540)

``Monitor`` carries ``index, x, y, width, height, scale, primary`` and a ``work``
area (``.bounds`` / ``.contains(x, y)`` / ``.to_dict()``). ``virtual_bounds`` returns
the union box (origin may be negative); ``primary_monitor`` picks the primary;
``monitor_for_window(rect, monitors)`` returns the display a window mostly occupies
(max overlap); ``to_virtual`` is the inverse of ``to_local``; ``remap_point``
preserves the fractional position so it works across differing resolutions and DPI.

Executor commands
-----------------

``AC_enumerate_monitors`` → ``{count, monitors, virtual_bounds}`` and
``AC_monitor_at_point`` (``x`` / ``y``) → ``{found, monitor}``. They are exposed as
the MCP tools ``ac_enumerate_monitors`` / ``ac_monitor_at_point`` and as Script
Builder commands under **Window**.
1 change: 1 addition & 0 deletions docs/source/Eng/eng_index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,7 @@ Comprehensive guides for all AutoControl features.
doc/new_features/v133_features_doc
doc/new_features/v134_features_doc
doc/new_features/v135_features_doc
doc/new_features/v136_features_doc
doc/ocr_backends/ocr_backends_doc
doc/observability/observability_doc
doc/operations_layer/operations_layer_doc
Expand Down
39 changes: 39 additions & 0 deletions docs/source/Zh/doc/new_features/v136_features_doc.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
多螢幕 / 虛擬桌面幾何
======================

``snap_window``、``arrange_grid`` 與版面規劃器都只取單一主螢幕 ``(width, height)``——它們對多螢幕無感:
無法在第二台顯示器上鋪排、也無法處理負原點的虛擬桌面,而 ``coordinate_space`` 只縮放模型網格。本功能補上缺少
的實體層:列舉各螢幕、計算聯集虛擬邊界、查詢某點或某視窗位於哪台螢幕、在虛擬座標與各螢幕區域座標間轉換,
並把某點重映射到另一台螢幕上的等效位置。

幾何運算皆是對純 ``Monitor`` dataclass 的算術,因此完全可單元測試;只有 ``enumerate_monitors`` 的預設 provider
會碰到 OS(透過 ``mss``),且可注入。不匯入 ``PySide6``。

無頭 API
--------

.. code-block:: python

from je_auto_control import (enumerate_monitors, monitor_at_point,
virtual_bounds, to_local, remap_point)

monitors = enumerate_monitors()
print(virtual_bounds(monitors)) # 涵蓋所有顯示器的 (x, y, w, h)

here = monitor_at_point(monitors, x, y) # 此點屬於哪台螢幕
idx, lx, ly = to_local(monitors, x, y) # 虛擬 -> (螢幕, 區域 x, 區域 y)

# 把某點移到另一台螢幕上的等效相對位置。
second = remap_point(monitors[0], monitors[1], 960, 540)

``Monitor`` 帶有 ``index, x, y, width, height, scale, primary`` 與 ``work`` 區域(``.bounds`` /
``.contains(x, y)`` / ``.to_dict()``)。``virtual_bounds`` 回傳聯集框(原點可能為負);``primary_monitor`` 取主螢幕;
``monitor_for_window(rect, monitors)`` 回傳視窗主要佔據的顯示器(最大重疊);``to_virtual`` 是 ``to_local`` 的反向;
``remap_point`` 保留分數位置,因此可跨不同解析度與 DPI 運作。

執行器命令
----------

``AC_enumerate_monitors`` → ``{count, monitors, virtual_bounds}`` 與 ``AC_monitor_at_point``(``x`` / ``y``)→
``{found, monitor}``。它們以 MCP 工具 ``ac_enumerate_monitors`` / ``ac_monitor_at_point`` 以及 Script Builder 中
**Window** 分類下的命令提供。
1 change: 1 addition & 0 deletions docs/source/Zh/zh_index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -158,6 +158,7 @@ AutoControl 所有功能的完整使用指南。
doc/new_features/v133_features_doc
doc/new_features/v134_features_doc
doc/new_features/v135_features_doc
doc/new_features/v136_features_doc
doc/ocr_backends/ocr_backends_doc
doc/observability/observability_doc
doc/operations_layer/operations_layer_doc
Expand Down
14 changes: 14 additions & 0 deletions je_auto_control/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -303,6 +303,11 @@
binarize, denoise, deskew, detect_skew_angle, enhance_contrast,
preprocess_image, to_grayscale, upscale,
)
# Multi-monitor / virtual-desktop geometry (which monitor, where, remapping)
from je_auto_control.utils.monitor_layout import (
Monitor, enumerate_monitors, monitor_at_point, monitor_for_window,
primary_monitor, remap_point, to_local, to_virtual, virtual_bounds,
)
# CI workflow annotations (GitHub Actions)
from je_auto_control.utils.ci_annotations import (
emit_annotations, format_annotation,
Expand Down Expand Up @@ -1119,6 +1124,15 @@ def start_autocontrol_gui(*args, **kwargs):
"deskew",
"detect_skew_angle",
"enhance_contrast",
"Monitor",
"enumerate_monitors",
"monitor_at_point",
"monitor_for_window",
"primary_monitor",
"remap_point",
"to_local",
"to_virtual",
"virtual_bounds",
"emit_annotations", "format_annotation",
"ClipboardHistory", "default_clipboard_history",
"analyze_heal_log", "heal_stats", "scan_secrets",
Expand Down
12 changes: 12 additions & 0 deletions je_auto_control/gui/script_builder/command_schema.py
Original file line number Diff line number Diff line change
Expand Up @@ -622,6 +622,18 @@ def _add_window_specs(specs: List[CommandSpec]) -> None:
),
description="Compute staggered, overlapping window rectangles.",
))
specs.append(CommandSpec(
"AC_enumerate_monitors", "Window", "Enumerate Monitors",
description="List monitors + virtual bounds (multi-display geometry).",
))
specs.append(CommandSpec(
"AC_monitor_at_point", "Window", "Monitor at Point",
fields=(
FieldSpec("x", FieldType.INT),
FieldSpec("y", FieldType.INT),
),
description="Report which monitor contains a virtual point.",
))


def _add_flow_specs(specs: List[CommandSpec]) -> None:
Expand Down
22 changes: 22 additions & 0 deletions je_auto_control/utils/executor/action_executor.py
Original file line number Diff line number Diff line change
Expand Up @@ -3431,6 +3431,26 @@ def _preprocess_image(output_path: str, source: Any = None, steps: Any = None,
"height": int(result.shape[0])}


def _enumerate_monitors() -> Dict[str, Any]:
"""Adapter: list connected monitors with virtual-desktop geometry."""
from je_auto_control.utils.monitor_layout import (
enumerate_monitors, virtual_bounds)
monitors = enumerate_monitors()
bounds = virtual_bounds(monitors) if monitors else (0, 0, 0, 0)
return {"count": len(monitors),
"monitors": [monitor.to_dict() for monitor in monitors],
"virtual_bounds": list(bounds)}


def _monitor_at_point(x: Any, y: Any) -> Dict[str, Any]:
"""Adapter: report which monitor contains a virtual point."""
from je_auto_control.utils.monitor_layout import (
enumerate_monitors, monitor_at_point)
monitor = monitor_at_point(enumerate_monitors(), int(x), int(y))
return {"found": monitor is not None,
"monitor": monitor.to_dict() if monitor else None}


def _with_modifiers(modifiers: Any, actions: Any) -> Dict[str, Any]:
"""Adapter: run nested actions while modifier keys are held down."""
import json
Expand Down Expand Up @@ -5158,6 +5178,8 @@ def __init__(self):
"AC_find_shapes": _find_shapes,
"AC_find_rectangles": _find_rectangles,
"AC_preprocess_image": _preprocess_image,
"AC_enumerate_monitors": _enumerate_monitors,
"AC_monitor_at_point": _monitor_at_point,
"AC_tile_rect": _tile_rect,
"AC_grid_rects": _grid_rects,
"AC_cascade_rects": _cascade_rects,
Expand Down
32 changes: 30 additions & 2 deletions je_auto_control/utils/mcp_server/tools/_factories.py
Original file line number Diff line number Diff line change
Expand Up @@ -2816,6 +2816,34 @@ def preprocess_tools() -> List[MCPTool]:
]


def monitor_layout_tools() -> List[MCPTool]:
return [
MCPTool(
name="ac_enumerate_monitors",
description=("List connected monitors with virtual-desktop geometry: "
"{count, monitors:[{index,x,y,width,height,scale,primary,"
"work}], virtual_bounds:[x,y,w,h]}. Unlike a single "
"screen_size, this exposes per-monitor origins (which may "
"be negative) for multi-display placement."),
input_schema=schema({}, required=[]),
handler=h.enumerate_monitors,
annotations=READ_ONLY,
),
MCPTool(
name="ac_monitor_at_point",
description=("Report which monitor contains virtual point (x, y): "
"{found, monitor}. Returns found=false when the point is "
"off every display."),
input_schema=schema({
"x": {"type": "integer"},
"y": {"type": "integer"}},
required=["x", "y"]),
handler=h.monitor_at_point,
annotations=READ_ONLY,
),
]


def ssim_tools() -> List[MCPTool]:
return [
MCPTool(
Expand Down Expand Up @@ -6318,8 +6346,8 @@ def media_assert_tools() -> List[MCPTool]:
key_hold_tools, mouse_relative_tools, text_unicode_tools,
modifier_state_tools, grid_locator_tools, visual_match_tools,
color_region_tools, ssim_tools, feature_match_tools, shape_locator_tools,
window_layout_tools, window_arrange_tools, preprocess_tools, plugin_sdk_tools,
governance_tools,
window_layout_tools, window_arrange_tools, preprocess_tools,
monitor_layout_tools, plugin_sdk_tools, governance_tools,
credential_lease_tools, egress_tools, approval_testing_tools,
trajectory_eval_tools, compliance_tools, agent_trace_tools,
video_report_tools, fuzzy_tools, artifact_store_tools, image_dedup_tools,
Expand Down
10 changes: 10 additions & 0 deletions je_auto_control/utils/mcp_server/tools/_handlers.py
Original file line number Diff line number Diff line change
Expand Up @@ -2162,6 +2162,16 @@ def preprocess_image(output_path, source=None, steps=None, scale=2.0, region=Non
block_size, c)


def enumerate_monitors():
from je_auto_control.utils.executor.action_executor import _enumerate_monitors
return _enumerate_monitors()


def monitor_at_point(x, y):
from je_auto_control.utils.executor.action_executor import _monitor_at_point
return _monitor_at_point(x, y)


def detect_drift(reference, current, threshold=0.25, bins=10):
from je_auto_control.utils.executor.action_executor import _detect_drift
return _detect_drift(reference, current, threshold, bins)
Expand Down
9 changes: 9 additions & 0 deletions je_auto_control/utils/monitor_layout/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
"""Multi-monitor / virtual-desktop geometry (which monitor, where, remapping)."""
from je_auto_control.utils.monitor_layout.monitor_layout import (
Monitor, enumerate_monitors, monitor_at_point, monitor_for_window,
primary_monitor, remap_point, to_local, to_virtual, virtual_bounds,
)

__all__ = ["Monitor", "enumerate_monitors", "monitor_at_point",
"monitor_for_window", "primary_monitor", "remap_point", "to_local",
"to_virtual", "virtual_bounds"]
Loading
Loading