Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions README/WHATS_NEW_zh-CN.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
# 本次更新 — AutoControl

## 本次更新 (2026-06-23) — 锚点序数与全部定位

挑选第 N 个锚点相对匹配,或列举全部。完整参考:[`docs/source/Zh/doc/new_features/v124_features_doc.rst`](../docs/source/Zh/doc/new_features/v124_features_doc.rst)。

- **`anchor_locate(..., ordinal=N)` / `anchor_locate_all`**(`AC_anchor_locate` ordinal、`AC_anchor_locate_all`):`anchor_locate` 总是返回单一最近的匹配——无法取「标题下方第 2 行」或列出每一行。本功能加入 1 起算的 `ordinal` 选择器(向后兼容;`ordinal=1` 即最近)与返回依距离排序所有匹配的 `anchor_locate_all`——表格/列表行选取的基础元件。纯排序核心、确定。

## 本次更新 (2026-06-23) — 在动作组中持续按住修饰键

在多个动作之间持续按住 ctrl/shift,即使出错也会放开。完整参考:[`docs/source/Zh/doc/new_features/v123_features_doc.rst`](../docs/source/Zh/doc/new_features/v123_features_doc.rst)。
Expand Down
6 changes: 6 additions & 0 deletions README/WHATS_NEW_zh-TW.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
# 本次更新 — AutoControl

## 本次更新 (2026-06-23) — 錨點序數與全部定位

挑選第 N 個錨點相對比對,或列舉全部。完整參考:[`docs/source/Zh/doc/new_features/v124_features_doc.rst`](../docs/source/Zh/doc/new_features/v124_features_doc.rst)。

- **`anchor_locate(..., ordinal=N)` / `anchor_locate_all`**(`AC_anchor_locate` ordinal、`AC_anchor_locate_all`):`anchor_locate` 總是回傳單一最近的比對——無法取「標題下方第 2 列」或列出每一列。本功能加入 1 起算的 `ordinal` 選擇器(向後相容;`ordinal=1` 即最近)與回傳依距離排序所有比對的 `anchor_locate_all`——表格/清單列選取的基礎元件。純排序核心、具決定性。

## 本次更新 (2026-06-23) — 在動作群組中持續按住修飾鍵

在多個動作之間持續按住 ctrl/shift,即使出錯也會放開。完整參考:[`docs/source/Zh/doc/new_features/v123_features_doc.rst`](../docs/source/Zh/doc/new_features/v123_features_doc.rst)。
Expand Down
6 changes: 6 additions & 0 deletions WHATS_NEW.md
Original file line number Diff line number Diff line change
@@ -1,5 +1,11 @@
# What's New — AutoControl

## What's new (2026-06-23) — Anchor Ordinal & Locate-All

Pick the Nth anchor-relative match, or enumerate them all. Full reference: [`docs/source/Eng/doc/new_features/v124_features_doc.rst`](docs/source/Eng/doc/new_features/v124_features_doc.rst).

- **`anchor_locate(..., ordinal=N)` / `anchor_locate_all`** (`AC_anchor_locate` ordinal, `AC_anchor_locate_all`): `anchor_locate` always returned the single nearest match — no way to grab "the 2nd row below the header" or list every row. Adds a 1-based `ordinal` selector (backward-compatible; `ordinal=1` = nearest) and `anchor_locate_all` returning every match sorted by distance — the building block for table/list-row selection. Pure ranking core, deterministic.

## What's new (2026-06-23) — Held Modifiers Across an Action Group

Hold ctrl/shift down across several actions, released even on error. Full reference: [`docs/source/Eng/doc/new_features/v123_features_doc.rst`](docs/source/Eng/doc/new_features/v123_features_doc.rst).
Expand Down
39 changes: 39 additions & 0 deletions docs/source/Eng/doc/new_features/v124_features_doc.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
Anchor Ordinal & Locate-All
===========================

``anchor_locate`` finds a target by spatial relation to an anchor but always
returned the single *nearest* match — there was no way to pick "the **2nd** row
below the header" or to enumerate every matching row. This adds an ``ordinal``
selector and a list-returning :func:`anchor_locate_all`.

Both build on a shared ranking helper (pure: filter by relation, sort by
distance) so the selection logic is unit-testable by injecting candidate boxes.
Headless and Qt-free.

Headless API
------------

.. code-block:: python

from je_auto_control import (
anchor_locate, anchor_locate_all, ocr_locator, image_locator,
)

header = ocr_locator("Name")
row = image_locator("row_handle.png")

anchor_locate(anchor=header, target=row, relation="below") # nearest
anchor_locate(anchor=header, target=row, relation="below", ordinal=2) # 2nd row
rows = anchor_locate_all(anchor=header, target=row, relation="below") # all rows

``ordinal`` is 1-based (``ordinal=1`` is the nearest, the previous behaviour, so
this is backward-compatible); an out-of-range ordinal returns a not-found outcome.
``anchor_locate_all`` returns a list of found ``AnchorOutcome`` ordered by
distance — the building block for table / list-row selection.

Executor commands
-----------------

``AC_anchor_locate`` gains an ``ordinal`` parameter; ``AC_anchor_locate_all``
returns ``{count, matches}``. Both are exposed as MCP tools (``ac_anchor_locate``
with ``ordinal`` / ``ac_anchor_locate_all``).
1 change: 1 addition & 0 deletions docs/source/Eng/eng_index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,7 @@ Comprehensive guides for all AutoControl features.
doc/new_features/v121_features_doc
doc/new_features/v122_features_doc
doc/new_features/v123_features_doc
doc/new_features/v124_features_doc
doc/ocr_backends/ocr_backends_doc
doc/observability/observability_doc
doc/operations_layer/operations_layer_doc
Expand Down
33 changes: 33 additions & 0 deletions docs/source/Zh/doc/new_features/v124_features_doc.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
錨點序數與全部定位
==================

``anchor_locate`` 依與錨點的空間關係尋找目標,但總是回傳單一*最近*的比對——先前無法選「標題下方的**第 2** 列」
或列舉每一個符合的列。本功能加入 ``ordinal`` 選擇器與回傳清單的 :func:`anchor_locate_all`。

兩者皆建立在共用的排序輔助函式上(純函式:依關係過濾、依距離排序),因此選擇邏輯可藉由注入候選框做單元測試。
無頭且不依賴 ``PySide6``。

無頭 API
--------

.. code-block:: python

from je_auto_control import (
anchor_locate, anchor_locate_all, ocr_locator, image_locator,
)

header = ocr_locator("Name")
row = image_locator("row_handle.png")

anchor_locate(anchor=header, target=row, relation="below") # 最近
anchor_locate(anchor=header, target=row, relation="below", ordinal=2) # 第 2 列
rows = anchor_locate_all(anchor=header, target=row, relation="below") # 所有列

``ordinal`` 為 1 起算(``ordinal=1`` 即最近,與先前行為相同,故向後相容);超出範圍的序數回傳未找到的結果。
``anchor_locate_all`` 回傳依距離排序的 found ``AnchorOutcome`` 清單——表格 / 清單列選取的基礎元件。

執行器命令
----------

``AC_anchor_locate`` 新增 ``ordinal`` 參數;``AC_anchor_locate_all`` 回傳 ``{count, matches}``。兩者皆以 MCP
工具(``ac_anchor_locate`` 含 ``ordinal`` / ``ac_anchor_locate_all``)提供。
1 change: 1 addition & 0 deletions docs/source/Zh/zh_index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -146,6 +146,7 @@ AutoControl 所有功能的完整使用指南。
doc/new_features/v121_features_doc
doc/new_features/v122_features_doc
doc/new_features/v123_features_doc
doc/new_features/v124_features_doc
doc/ocr_backends/ocr_backends_doc
doc/observability/observability_doc
doc/operations_layer/operations_layer_doc
Expand Down
7 changes: 4 additions & 3 deletions je_auto_control/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -607,8 +607,8 @@
# Anchor-based locators (spatial composition of locator backends)
from je_auto_control.utils.anchor_locator import (
AnchorLocatorError, AnchorOutcome, Locator as AnchorLocator,
a11y_locator, anchor_locate, image_locator, ocr_locator,
vlm_locator,
a11y_locator, anchor_locate, anchor_locate_all, image_locator,
ocr_locator, vlm_locator,
)
# Structured OCR (rows / tables / form fields)
from je_auto_control.utils.ocr.structure import (
Expand Down Expand Up @@ -1245,7 +1245,8 @@ def start_autocontrol_gui(*args, **kwargs):
"register_chatops_default_commands",
# Anchor-based locator
"AnchorLocator", "AnchorLocatorError", "AnchorOutcome",
"a11y_locator", "anchor_locate", "image_locator", "ocr_locator",
"a11y_locator", "anchor_locate", "anchor_locate_all", "image_locator",
"ocr_locator",
"vlm_locator",
# Structured OCR
"OCRField", "OCRRow", "OCRTable", "StructuredOCR",
Expand Down
8 changes: 4 additions & 4 deletions je_auto_control/utils/anchor_locator/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,15 +22,15 @@
AnchorLocatorError, AnchorOutcome, KIND_A11Y, KIND_IMAGE,
KIND_OCR, KIND_VLM, Locator, REL_ABOVE, REL_BELOW,
REL_LEFT_OF, REL_NEAR, REL_RIGHT_OF,
a11y_locator, anchor_locate, image_locator, ocr_locator,
vlm_locator,
a11y_locator, anchor_locate, anchor_locate_all, image_locator,
ocr_locator, vlm_locator,
)


__all__ = [
"AnchorLocatorError", "AnchorOutcome", "KIND_A11Y", "KIND_IMAGE",
"KIND_OCR", "KIND_VLM", "Locator", "REL_ABOVE", "REL_BELOW",
"REL_LEFT_OF", "REL_NEAR", "REL_RIGHT_OF",
"a11y_locator", "anchor_locate", "image_locator", "ocr_locator",
"vlm_locator",
"a11y_locator", "anchor_locate", "anchor_locate_all", "image_locator",
"ocr_locator", "vlm_locator",
]
76 changes: 54 additions & 22 deletions je_auto_control/utils/anchor_locator/locator.py
Original file line number Diff line number Diff line change
Expand Up @@ -137,7 +137,8 @@ def to_dict(self) -> Dict[str, Any]:

def anchor_locate(*, anchor: Locator, target: Locator,
relation: str = REL_NEAR,
max_distance_px: float = 200.0) -> AnchorOutcome:
max_distance_px: float = 200.0,
ordinal: int = 1) -> AnchorOutcome:
"""Find ``target`` near / above / below / beside ``anchor``.

Strategy:
Expand All @@ -147,8 +148,10 @@ def anchor_locate(*, anchor: Locator, target: Locator,
2. Resolve the target to a list of candidate bboxes. Image and
OCR can enumerate; VLM / a11y always return one point so
only that one candidate is considered.
3. Filter candidates by the spatial relation; tie-break by the
smallest centre-to-centre distance.
3. Filter candidates by the spatial relation and sort by the
centre-to-centre distance; return the ``ordinal``-th (1-based,
so ``ordinal=1`` is the nearest — "the 2nd row below the header"
is ``ordinal=2``).
"""
relation_norm = _normalise_relation(relation)
anchor_point = _resolve_single(anchor)
Expand All @@ -160,34 +163,61 @@ def anchor_locate(*, anchor: Locator, target: Locator,
error="anchor not found",
)
candidates = _resolve_candidates(target)
candidates_considered = len(candidates)
if not candidates:
return AnchorOutcome(
found=False, target_coords=None, anchor_coords=anchor_point,
distance_px=None, relation=relation_norm,
target_kind=target.kind, anchor_kind=anchor.kind,
candidates_considered=0, error="target not found",
)
chosen = _pick_best(
anchor_point, candidates, relation_norm, float(max_distance_px),
)
if chosen is None:
ranked = _ranked(anchor_point, candidates, relation_norm,
float(max_distance_px))
index = int(ordinal) - 1
if index < 0 or index >= len(ranked):
return AnchorOutcome(
found=False, target_coords=None, anchor_coords=anchor_point,
distance_px=None, relation=relation_norm,
target_kind=target.kind, anchor_kind=anchor.kind,
candidates_considered=candidates_considered,
error=f"no candidate satisfies relation {relation_norm!r}",
candidates_considered=len(candidates),
error=f"no candidate at ordinal {ordinal} for relation "
f"{relation_norm!r}",
)
coords, distance = chosen
coords, distance = ranked[index]
return AnchorOutcome(
found=True, target_coords=coords, anchor_coords=anchor_point,
distance_px=round(distance, 2), relation=relation_norm,
target_kind=target.kind, anchor_kind=anchor.kind,
candidates_considered=candidates_considered,
candidates_considered=len(candidates),
)


def anchor_locate_all(*, anchor: Locator, target: Locator,
relation: str = REL_NEAR,
max_distance_px: float = 200.0) -> List[AnchorOutcome]:
"""Return every target matching the relation, nearest-first.

The building block for table / list-row selection: each result is a found
:class:`AnchorOutcome` ordered by distance from the anchor. Returns an empty
list when the anchor or all targets are missing.
"""
relation_norm = _normalise_relation(relation)
anchor_point = _resolve_single(anchor)
if anchor_point is None:
return []
candidates = _resolve_candidates(target)
ranked = _ranked(anchor_point, candidates, relation_norm,
float(max_distance_px))
return [
AnchorOutcome(
found=True, target_coords=coords, anchor_coords=anchor_point,
distance_px=round(distance, 2), relation=relation_norm,
target_kind=target.kind, anchor_kind=anchor.kind,
candidates_considered=len(candidates),
)
for coords, distance in ranked
]


def _normalise_relation(relation: str) -> str:
normalised = (relation or "").strip().lower()
if normalised not in _VALID_RELATIONS:
Expand Down Expand Up @@ -221,21 +251,22 @@ def _resolve_candidates(locator: Locator) -> List[_Bbox]:
return []


def _pick_best(anchor_point: Tuple[int, int],
candidates: List[_Bbox],
relation: str,
max_distance: float,
) -> Optional[Tuple[Tuple[int, int], float]]:
best: Optional[Tuple[Tuple[int, int], float]] = None
def _ranked(anchor_point: Tuple[int, int],
candidates: List[_Bbox],
relation: str,
max_distance: float,
) -> List[Tuple[Tuple[int, int], float]]:
"""Return (centre, distance) of matching candidates, nearest first."""
matches: List[Tuple[Tuple[int, int], float]] = []
for bbox in candidates:
if not _matches_relation(anchor_point, bbox, relation):
continue
distance = _euclid(anchor_point, bbox.center)
if relation == REL_NEAR and distance > max_distance:
continue
if best is None or distance < best[1]:
best = (bbox.center, distance)
return best
matches.append((bbox.center, distance))
matches.sort(key=lambda item: item[1])
return matches


def _matches_relation(anchor_point: Tuple[int, int],
Expand Down Expand Up @@ -352,5 +383,6 @@ def _a11y_point(locator: Locator) -> Optional[Tuple[int, int]]:
"AnchorLocatorError", "AnchorOutcome", "KIND_A11Y", "KIND_IMAGE",
"KIND_OCR", "KIND_VLM", "Locator", "REL_ABOVE", "REL_BELOW",
"REL_LEFT_OF", "REL_NEAR", "REL_RIGHT_OF", "a11y_locator",
"anchor_locate", "image_locator", "ocr_locator", "vlm_locator",
"anchor_locate", "anchor_locate_all", "image_locator", "ocr_locator",
"vlm_locator",
]
19 changes: 17 additions & 2 deletions je_auto_control/utils/executor/action_executor.py
Original file line number Diff line number Diff line change
Expand Up @@ -478,8 +478,9 @@ def _ocr_read_structure(region: Optional[List[int]] = None,

def _anchor_locate(anchor: Dict[str, Any], target: Dict[str, Any],
relation: str = "near",
max_distance_px: float = 200.0) -> Dict[str, Any]:
"""Executor adapter: anchor-based spatial locator."""
max_distance_px: float = 200.0,
ordinal: Any = 1) -> Dict[str, Any]:
"""Executor adapter: anchor-based spatial locator (Nth match via ordinal)."""
from je_auto_control.utils.anchor_locator import (
Locator, anchor_locate,
)
Expand All @@ -488,10 +489,23 @@ def _anchor_locate(anchor: Dict[str, Any], target: Dict[str, Any],
outcome = anchor_locate(
anchor=anchor_loc, target=target_loc,
relation=relation, max_distance_px=float(max_distance_px),
ordinal=int(ordinal),
)
return outcome.to_dict()


def _anchor_locate_all(anchor: Dict[str, Any], target: Dict[str, Any],
relation: str = "near",
max_distance_px: float = 200.0) -> Dict[str, Any]:
"""Executor adapter: every anchor-relative match, nearest-first."""
from je_auto_control.utils.anchor_locator import Locator, anchor_locate_all
outcomes = anchor_locate_all(
anchor=Locator(**anchor), target=Locator(**target),
relation=relation, max_distance_px=float(max_distance_px),
)
return {"count": len(outcomes), "matches": [o.to_dict() for o in outcomes]}


def _anchor_click(anchor: Dict[str, Any], target: Dict[str, Any],
mouse_keycode: str = "mouse_left",
relation: str = "near",
Expand Down Expand Up @@ -5037,6 +5051,7 @@ def __init__(self):

# Anchor-based locator (spatial composition of locator backends)
"AC_anchor_locate": _anchor_locate,
"AC_anchor_locate_all": _anchor_locate_all,
"AC_anchor_click": _anchor_click,

# Structured OCR (rows / tables / form fields)
Expand Down
17 changes: 17 additions & 0 deletions je_auto_control/utils/mcp_server/tools/_factories.py
Original file line number Diff line number Diff line change
Expand Up @@ -1472,10 +1472,27 @@ def anchor_locator_tools() -> List[MCPTool]:
"enum": ["above", "below", "left_of",
"right_of", "near"]},
"max_distance_px": {"type": "number"},
"ordinal": {"type": "integer"},
}, required=["anchor", "target"]),
handler=h.anchor_locate,
annotations=READ_ONLY,
),
MCPTool(
name="ac_anchor_locate_all",
description=("Every target matching the spatial relation to the "
"anchor, nearest-first (for table / list-row "
"selection). Returns {count, matches}."),
input_schema=schema({
"anchor": locator_schema,
"target": locator_schema,
"relation": {"type": "string",
"enum": ["above", "below", "left_of",
"right_of", "near"]},
"max_distance_px": {"type": "number"},
}, required=["anchor", "target"]),
handler=h.anchor_locate_all,
annotations=READ_ONLY,
),
MCPTool(
name="ac_anchor_click",
description="Anchor-locate then click the resolved target point.",
Expand Down
19 changes: 11 additions & 8 deletions je_auto_control/utils/mcp_server/tools/_handlers.py
Original file line number Diff line number Diff line change
Expand Up @@ -2532,14 +2532,17 @@ def ocr_read_structure(region: Optional[List[int]] = None,

def anchor_locate(anchor: Dict[str, Any], target: Dict[str, Any],
relation: str = "near",
max_distance_px: float = 200.0) -> Dict[str, Any]:
from je_auto_control.utils.anchor_locator import (
Locator, anchor_locate as _impl,
)
return _impl(
anchor=Locator(**anchor), target=Locator(**target),
relation=relation, max_distance_px=float(max_distance_px),
).to_dict()
max_distance_px: float = 200.0,
ordinal: int = 1) -> Dict[str, Any]:
from je_auto_control.utils.executor.action_executor import _anchor_locate
return _anchor_locate(anchor, target, relation, max_distance_px, ordinal)


def anchor_locate_all(anchor: Dict[str, Any], target: Dict[str, Any],
relation: str = "near",
max_distance_px: float = 200.0) -> Dict[str, Any]:
from je_auto_control.utils.executor.action_executor import _anchor_locate_all
return _anchor_locate_all(anchor, target, relation, max_distance_px)


def anchor_click(anchor: Dict[str, Any], target: Dict[str, Any],
Expand Down
Loading
Loading