diff --git a/README/WHATS_NEW_zh-CN.md b/README/WHATS_NEW_zh-CN.md index d5fabbb6..51f42407 100644 --- a/README/WHATS_NEW_zh-CN.md +++ b/README/WHATS_NEW_zh-CN.md @@ -1,5 +1,29 @@ # 本次更新 — AutoControl +## 本次更新 (2026-06-24) — OCR 行的段落与列表分组 + +把 OCR 行分组成段落,并检测项目符号 / 编号列表。完整参考:[`docs/source/Zh/doc/new_features/v174_features_doc.rst`](../docs/source/Zh/doc/new_features/v174_features_doc.rst)。 + +- **`group_paragraphs` / `detect_lists`**(`AC_group_paragraphs`、`AC_detect_lists`):`text_regions` 把字形并成行,但没有功能把那些行分组成段落或检测列表;`ocr/structure` 止于平面行。`group_paragraphs` 在垂直间距超过 `line_gap_factor` × 中位行高处开始新段落;`detect_lists` 识别项目符号(`•`/`-`/`*`)或序号(`1.`/`2)`/`a.`)项目,返回 `{text, marker, indent, box}`。纯标准库,作用于行字典;重用 `table_grid_fill` 的框读取器;不导入 `PySide6`。 + +## 本次更新 (2026-06-24) — 列感知阅读顺序(XY-Cut) + +读多列布局时*完整读完*每一列,而非交错。完整参考:[`docs/source/Zh/doc/new_features/v173_features_doc.rst`](../docs/source/Zh/doc/new_features/v173_features_doc.rst)。 + +- **`flow_order` / `xy_cut` / `to_blocks`**(`AC_flow_order`、`AC_xy_cut`):`element_parse.reading_order` 是平面上到下排序,会交错列(读作 A1, B1, A2, B2…)。本功能以递归 XY-cut 还原正确顺序——在最宽留白谷切分(垂直 → 列、水平 → 行),故两列页面读作 A1, A2, B1, B2。`flow_order` 返回与 `reading_order` 相同的 `index` 标记契约(列感知的直接升级,且命名不遮蔽它);`xy_cut` 暴露区域树;`to_blocks` 列出叶区块。纯标准库;不导入 `PySide6`。 + +## 本次更新 (2026-06-24) — Grounding 自我一致性(提案共识) + +把多个 grounding 提案融合成单一一致目标,并附 agreement 分数。完整参考:[`docs/source/Zh/doc/new_features/v172_features_doc.rst`](../docs/source/Zh/doc/new_features/v172_features_doc.rst)。 + +- **`consensus_point` / `consensus_element` / `is_confident`**(`AC_consensus_point`、`AC_consensus_element`):一个目标可同时以多种方式 grounding(set-of-marks / OCR / 模板 / a11y / 模型 N 次采样)而未必一致。`ab_locator`/`element_scoring` 依历史排序*策略*;`snap_to_element` 只贴*单一*坐标——两者都不融合*同时*的提案。本功能将候选点聚类(或对候选元素投票),返回一致的 `point` + `agreement` 比例 + `spread`,而 `is_confident` 标记低一致度目标,让代理改为放大 / 询问而非盲目点击。纯标准库;不导入 `PySide6`。 + +## 本次更新 (2026-06-24) — 亚像素模板匹配精修 + +把匹配中心精修到像素的分数位,供拖曳 / 滑块 / 高 DPI 精度。完整参考:[`docs/source/Zh/doc/new_features/v171_features_doc.rst`](../docs/source/Zh/doc/new_features/v171_features_doc.rst)。 + +- **`match_subpixel` / `refine_peak`**(`AC_match_subpixel`):每个匹配器都从 `cv2.minMaxLoc` 返回*整数*坐标——对拖曳把手、细滑块或高 DPI 显示器,这种舍入是主要的点击落点误差。本功能以抛物线拟合峰值周围的 3×3 分数邻域(x/y 各自独立,标准 NCC 亚像素法),返回带浮点 `cx`/`cy` 与套用的 `offset_x`/`offset_y` 的 `SubPixelMatch`。重用 `visual_match._score_map`;`haystack` 可注入;不导入 `PySide6`。 + ## 本次更新 (2026-06-24) — 失败 / 无效果动作的修复策略引擎 当动作没效果时选择下一个修复战术——并驱动重试循环。完整参考:[`docs/source/Zh/doc/new_features/v170_features_doc.rst`](../docs/source/Zh/doc/new_features/v170_features_doc.rst)。 diff --git a/README/WHATS_NEW_zh-TW.md b/README/WHATS_NEW_zh-TW.md index 4fe1e509..65ca68c9 100644 --- a/README/WHATS_NEW_zh-TW.md +++ b/README/WHATS_NEW_zh-TW.md @@ -1,5 +1,29 @@ # 本次更新 — AutoControl +## 本次更新 (2026-06-24) — OCR 行的段落與清單分組 + +把 OCR 行分組成段落,並偵測項目符號 / 編號清單。完整參考:[`docs/source/Zh/doc/new_features/v174_features_doc.rst`](../docs/source/Zh/doc/new_features/v174_features_doc.rst)。 + +- **`group_paragraphs` / `detect_lists`**(`AC_group_paragraphs`、`AC_detect_lists`):`text_regions` 把字形併成行,但沒有功能把那些行分組成段落或偵測清單;`ocr/structure` 止於平面列。`group_paragraphs` 在垂直間距超過 `line_gap_factor` × 中位行高處開始新段落;`detect_lists` 辨識項目符號(`•`/`-`/`*`)或序號(`1.`/`2)`/`a.`)項目,回傳 `{text, marker, indent, box}`。純標準函式庫,作用於行字典;重用 `table_grid_fill` 的框讀取器;不匯入 `PySide6`。 + +## 本次更新 (2026-06-24) — 欄位感知閱讀順序(XY-Cut) + +讀多欄版面時*完整讀完*每一欄,而非交錯。完整參考:[`docs/source/Zh/doc/new_features/v173_features_doc.rst`](../docs/source/Zh/doc/new_features/v173_features_doc.rst)。 + +- **`flow_order` / `xy_cut` / `to_blocks`**(`AC_flow_order`、`AC_xy_cut`):`element_parse.reading_order` 是平面上到下排序,會交錯欄位(讀作 A1, B1, A2, B2…)。本功能以遞迴 XY-cut 還原正確順序——在最寬留白谷切分(垂直 → 欄、水平 → 列),故兩欄頁面讀作 A1, A2, B1, B2。`flow_order` 回傳與 `reading_order` 相同的 `index` 標記契約(欄位感知的直接升級,且命名不遮蔽它);`xy_cut` 暴露區域樹;`to_blocks` 列出葉區塊。純標準函式庫;不匯入 `PySide6`。 + +## 本次更新 (2026-06-24) — Grounding 自我一致性(提案共識) + +把多個 grounding 提案融合成單一一致目標,並附 agreement 分數。完整參考:[`docs/source/Zh/doc/new_features/v172_features_doc.rst`](../docs/source/Zh/doc/new_features/v172_features_doc.rst)。 + +- **`consensus_point` / `consensus_element` / `is_confident`**(`AC_consensus_point`、`AC_consensus_element`):一個目標可同時以多種方式 grounding(set-of-marks / OCR / 樣板 / a11y / 模型 N 次抽樣)而未必一致。`ab_locator`/`element_scoring` 依歷史排序*策略*;`snap_to_element` 只貼*單一*座標——兩者都不融合*同時*的提案。本功能將候選點聚類(或對候選元素投票),回傳一致的 `point` + `agreement` 比例 + `spread`,而 `is_confident` 標記低一致度目標,讓代理改為放大 / 詢問而非盲目點擊。純標準函式庫;不匯入 `PySide6`。 + +## 本次更新 (2026-06-24) — 次像素樣板比對精修 + +把比對中心精修到像素的分數位,供拖曳 / 滑桿 / 高 DPI 精度。完整參考:[`docs/source/Zh/doc/new_features/v171_features_doc.rst`](../docs/source/Zh/doc/new_features/v171_features_doc.rst)。 + +- **`match_subpixel` / `refine_peak`**(`AC_match_subpixel`):每個比對器都從 `cv2.minMaxLoc` 回傳*整數*座標——對拖曳把手、細滑桿或高 DPI 顯示器,這種捨入是主要的點擊落點誤差。本功能以拋物線擬合峰值周圍的 3×3 分數鄰域(x/y 各自獨立,標準 NCC 次像素法),回傳帶浮點 `cx`/`cy` 與套用的 `offset_x`/`offset_y` 的 `SubPixelMatch`。重用 `visual_match._score_map`;`haystack` 可注入;不匯入 `PySide6`。 + ## 本次更新 (2026-06-24) — 失敗 / 無效果動作的修復策略引擎 當動作沒效果時選擇下一個修復戰術——並驅動重試迴圈。完整參考:[`docs/source/Zh/doc/new_features/v170_features_doc.rst`](../docs/source/Zh/doc/new_features/v170_features_doc.rst)。 diff --git a/WHATS_NEW.md b/WHATS_NEW.md index d1b4ebd6..144438db 100644 --- a/WHATS_NEW.md +++ b/WHATS_NEW.md @@ -1,5 +1,29 @@ # What's New — AutoControl +## What's new (2026-06-24) — Paragraph & List Grouping of OCR Lines + +Group OCR lines into paragraphs and detect bulleted / numbered lists. Full reference: [`docs/source/Eng/doc/new_features/v174_features_doc.rst`](docs/source/Eng/doc/new_features/v174_features_doc.rst). + +- **`group_paragraphs` / `detect_lists`** (`AC_group_paragraphs`, `AC_detect_lists`): `text_regions` merges glyphs into lines but nothing grouped those lines into paragraphs or detected lists; `ocr/structure` stops at flat rows. `group_paragraphs` starts a new paragraph wherever the vertical gap exceeds `line_gap_factor` × the median line height; `detect_lists` recognises bullet (`•`/`-`/`*`) or ordinal (`1.`/`2)`/`a.`) items, returning `{text, marker, indent, box}`. Pure-stdlib over line dicts; reuses `table_grid_fill`'s box reader; no `PySide6`. + +## What's new (2026-06-24) — Column-Aware Reading Order (XY-Cut) + +Read multi-column layouts down each column instead of interleaving them. Full reference: [`docs/source/Eng/doc/new_features/v173_features_doc.rst`](docs/source/Eng/doc/new_features/v173_features_doc.rst). + +- **`flow_order` / `xy_cut` / `to_blocks`** (`AC_flow_order`, `AC_xy_cut`): `element_parse.reading_order` is a flat top-to-bottom sort that interleaves columns (reads A1, B1, A2, B2…). This recovers the correct order with recursive XY-cut — split at the widest whitespace valley (vertical → columns, horizontal → rows), so a two-column page reads A1, A2, B1, B2. `flow_order` returns the same `index`-tagged contract as `reading_order` (a drop-in column-aware upgrade, named to not shadow it); `xy_cut` exposes the region tree; `to_blocks` lists the leaf blocks. Pure-stdlib; no `PySide6`. + +## What's new (2026-06-24) — Grounding Self-Consistency (Consensus Over Proposals) + +Fuse several grounding proposals into one agreed target with an agreement score. Full reference: [`docs/source/Eng/doc/new_features/v172_features_doc.rst`](docs/source/Eng/doc/new_features/v172_features_doc.rst). + +- **`consensus_point` / `consensus_element` / `is_confident`** (`AC_consensus_point`, `AC_consensus_element`): a target can be grounded several ways at once (set-of-marks / OCR / template / a11y / N model samples) and they don't always agree. `ab_locator`/`element_scoring` rank *strategies* by history; `snap_to_element` snaps a *single* coordinate — neither fuses *simultaneous* proposals. This clusters candidate points (or votes candidate elements), returns the agreed `point` + an `agreement` fraction + `spread`, and `is_confident` flags low-agreement targets so the agent zooms / asks instead of clicking blind. Pure-stdlib; no `PySide6`. + +## What's new (2026-06-24) — Sub-Pixel Template-Match Refinement + +Refine a match's centre to a fraction of a pixel for drag / slider / high-DPI precision. Full reference: [`docs/source/Eng/doc/new_features/v171_features_doc.rst`](docs/source/Eng/doc/new_features/v171_features_doc.rst). + +- **`match_subpixel` / `refine_peak`** (`AC_match_subpixel`): every matcher returns *integer* coordinates from `cv2.minMaxLoc` — for a drag handle, fine slider or high-DPI display that rounding is the dominant click-placement error. This fits a parabola to the 3×3 score neighbourhood around the peak (independently on x/y, the standard NCC sub-pixel method) and returns a `SubPixelMatch` with float `cx`/`cy` + the applied `offset_x`/`offset_y`. Reuses `visual_match._score_map`; injectable `haystack`; no `PySide6`. + ## What's new (2026-06-24) — Repair-Tactic Policy for Failed / No-Effect Actions Pick the next repair tactic when an action does nothing — and drive the retry loop. Full reference: [`docs/source/Eng/doc/new_features/v170_features_doc.rst`](docs/source/Eng/doc/new_features/v170_features_doc.rst). diff --git a/docs/source/Eng/doc/new_features/v171_features_doc.rst b/docs/source/Eng/doc/new_features/v171_features_doc.rst new file mode 100644 index 00000000..f22e58bd --- /dev/null +++ b/docs/source/Eng/doc/new_features/v171_features_doc.rst @@ -0,0 +1,40 @@ +Sub-Pixel Template-Match Refinement +=================================== + +Every matcher (``match_template`` / ``match_rotated`` / ``match_masked``) returns *integer* +pixel coordinates straight from ``cv2.minMaxLoc``. For a drag handle, a fine slider, a +sub-pixel-rendered target, or a high-DPI display, that integer rounding is the dominant +click-placement error. ``subpixel_match`` refines the peak to a fraction of a pixel by fitting +a parabola to the 3x3 score neighbourhood around the integer maximum — independently on x and y +(the standard NCC sub-pixel method). + +It reuses ``visual_match._score_map`` (the full ``matchTemplate`` surface the public matchers +discard); no matching code is duplicated. The ``haystack`` is injectable; the analysis is +unit-testable on synthetic arrays. Imports no ``PySide6``. + +Headless API +------------ + +.. code-block:: python + + from je_auto_control import match_subpixel, refine_peak + + hit = match_subpixel("slider_thumb.png", min_score=0.8) + if hit: + # cx / cy are floats — pass to a sub-pixel-aware drag / move + move_to(hit.cx, hit.cy) + + # the fitting primitive over any score surface + offset_x, offset_y = refine_peak(score_map, (peak_x, peak_y)) + +``match_subpixel`` returns a ``SubPixelMatch`` (integer ``x`` / ``y`` / ``width`` / ``height`` / +``score`` plus float ``cx`` / ``cy`` and the ``offset_x`` / ``offset_y`` applied), or ``None`` +below ``min_score``. ``refine_peak`` returns the ``[-0.5, 0.5]`` quadratic-fit offset of a peak +from its neighbours — usable on any correlation surface. + +Executor command +---------------- + +``AC_match_subpixel`` (``template`` / ``min_score`` / ``region`` / ``method`` → +``{found, match}``) is exposed as the MCP tool ``ac_match_subpixel`` (read-only) and as the +Script Builder command **Match Template (sub-pixel)** under **Image**. diff --git a/docs/source/Eng/doc/new_features/v172_features_doc.rst b/docs/source/Eng/doc/new_features/v172_features_doc.rst new file mode 100644 index 00000000..c975a1ef --- /dev/null +++ b/docs/source/Eng/doc/new_features/v172_features_doc.rst @@ -0,0 +1,44 @@ +Grounding Self-Consistency (Consensus Over Proposals) +===================================================== + +A target can be grounded several ways at once — set-of-marks, OCR, template match, the a11y +tree, or N samples from a model — and they don't always agree. ``ab_locator`` / +``element_scoring`` rank locator *strategies* by historical reliability but never fuse +*simultaneous* proposals into one consensus point with a dispersion metric, and +``action_grounding.snap_to_element`` snaps a *single* coordinate to the nearest element with no +notion of disagreement. ``grounding_consensus`` clusters the candidate points (or votes +candidate elements), returns the agreed target plus an *agreement* fraction and *spread*, and +flags low-agreement targets so the caller can zoom / ask a human instead of clicking blind. + +Pure-stdlib geometry; deterministic and unit-testable with no device. Imports no ``PySide6``. + +Headless API +------------ + +.. code-block:: python + + from je_auto_control import consensus_point, consensus_element, is_confident + + proposals = [[480, 260], [484, 258], [477, 263], [900, 100]] # one outlier + result = consensus_point(proposals, cluster_radius=24) + if is_confident(result, min_agreement=0.6): + click(*result.point) + else: + print("proposals disagree:", result.agreement, result.spread) + + # or vote proposals onto a known element list + element, agreement = consensus_element(proposals, elements) + +``consensus_point`` clusters ``[x, y]`` or ``[x, y, weight]`` candidates and returns a +``ConsensusResult`` (``point`` / ``agreement`` — winning cluster weight over total / ``spread`` +/ ``n_clusters``), or ``None`` when empty. ``consensus_element`` votes each proposal to its +nearest element and returns ``(element, agreement)``. ``is_confident`` thresholds the agreement. + +Executor commands +----------------- + +``AC_consensus_point`` (``candidates`` / ``cluster_radius`` → ``{found, result}``) and +``AC_consensus_element`` (``candidates`` / ``elements`` → ``{found, element, agreement}``). They +are exposed as the MCP tools ``ac_consensus_point`` / ``ac_consensus_element`` (read-only) and +as the Script Builder commands **Grounding Consensus Point** / **Grounding Consensus Element** +under **Native UI**. diff --git a/docs/source/Eng/doc/new_features/v173_features_doc.rst b/docs/source/Eng/doc/new_features/v173_features_doc.rst new file mode 100644 index 00000000..c0b02063 --- /dev/null +++ b/docs/source/Eng/doc/new_features/v173_features_doc.rst @@ -0,0 +1,41 @@ +Column-Aware Reading Order (XY-Cut) +=================================== + +``element_parse.reading_order`` is a flat top-to-bottom / left-to-right sort — it interleaves +columns on any multi-column page (the well-documented naive-sort failure: it reads row 1 of +column A, then row 1 of column B, then row 2 of A …). ``reading_flow`` recovers the correct +order with recursive **XY-cut**: it repeatedly splits the boxes at the widest whitespace valley +(a vertical gutter → columns, a horizontal gutter → rows / blocks), so a two-column layout is +read *down* column A fully, then column B. + +The public flattener is named ``flow_order`` to sit *beside* — not shadow — +``element_parse.reading_order``; it returns the same ``index``-tagged element contract, so it is +a drop-in column-aware upgrade. Pure-stdlib geometry over plain box dicts (no image, no OCR +engine); reuses ``table_grid_fill``'s box-bounds reader. Imports no ``PySide6``. + +Headless API +------------ + +.. code-block:: python + + from je_auto_control import flow_order, xy_cut, to_blocks + + # two columns, two rows each: reads A1, A2, B1, B2 — not A1, B1, A2, B2 + for element in flow_order(ocr_boxes, min_gap=12): + print(element["index"], element["text"]) + + tree = xy_cut(ocr_boxes, min_gap=12) # {type, axis, children|boxes} + blocks = to_blocks(tree) # leaf blocks in reading order + +``flow_order`` returns the boxes in column-aware reading order, each tagged with an ``index``. +``xy_cut`` returns the recursive region tree (each node is a ``split`` on ``axis`` ``"x"`` / +``"y"`` or a ``leaf`` of boxes). ``to_blocks`` flattens the tree to its leaf blocks in order. +``min_gap`` is the smallest whitespace valley treated as a column / row break. + +Executor commands +----------------- + +``AC_flow_order`` (``boxes`` / ``min_gap`` → ``{count, elements}``) and ``AC_xy_cut`` +(``boxes`` / ``min_gap`` → ``{tree}``). They are exposed as the MCP tools ``ac_flow_order`` / +``ac_xy_cut`` (read-only) and as the Script Builder commands **Reading Order (column-aware)** / +**XY-Cut Region Tree** under **OCR**. diff --git a/docs/source/Eng/doc/new_features/v174_features_doc.rst b/docs/source/Eng/doc/new_features/v174_features_doc.rst new file mode 100644 index 00000000..9fc976ea --- /dev/null +++ b/docs/source/Eng/doc/new_features/v174_features_doc.rst @@ -0,0 +1,38 @@ +Paragraph & List Grouping of OCR Lines +====================================== + +``text_regions.find_text_lines`` merges glyphs into lines, but nothing groups those *lines* +into paragraphs or detects lists — ``ocr/structure`` stops at flat rows. ``text_blocks`` adds +that: ``group_paragraphs`` splits lines into paragraphs wherever the vertical gap exceeds a +multiple of the median line height (the standard whitespace-grouping heuristic), and +``detect_lists`` recognises bulleted / numbered items by their leading marker and left indent. + +Pure-stdlib over plain line dicts (text + bbox); fully unit-testable with no image and no OCR +engine. Reuses ``table_grid_fill``'s box-bounds reader. Imports no ``PySide6``. + +Headless API +------------ + +.. code-block:: python + + from je_auto_control import group_paragraphs, detect_lists + + for para in group_paragraphs(ocr_lines, line_gap_factor=1.6): + print(para["n_lines"], para["text"]) + + for item in detect_lists(ocr_lines): + print(item["marker"], item["indent"], item["text"]) + +``group_paragraphs`` returns paragraph dicts (``left`` / ``top`` / ``right`` / ``bottom`` / +``text`` / ``n_lines``) — a new paragraph begins when the gap from the previous line's bottom +exceeds ``line_gap_factor`` times the median line height. ``detect_lists`` returns the lines +whose text starts with a bullet (``•`` / ``-`` / ``*``) or an ordinal (``1.`` / ``2)`` / +``a.``), each as ``{text, marker, indent, box}`` (``indent`` is the left x, for nesting). + +Executor commands +----------------- + +``AC_group_paragraphs`` (``lines`` / ``line_gap_factor`` → ``{count, paragraphs}``) and +``AC_detect_lists`` (``lines`` → ``{count, items}``). They are exposed as the MCP tools +``ac_group_paragraphs`` / ``ac_detect_lists`` (read-only) and as the Script Builder commands +**Group Paragraphs** / **Detect Lists** under **OCR**. diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst index e46aa880..7d1ed13f 100644 --- a/docs/source/Eng/eng_index.rst +++ b/docs/source/Eng/eng_index.rst @@ -193,6 +193,10 @@ Comprehensive guides for all AutoControl features. doc/new_features/v168_features_doc doc/new_features/v169_features_doc doc/new_features/v170_features_doc + doc/new_features/v171_features_doc + doc/new_features/v172_features_doc + doc/new_features/v173_features_doc + doc/new_features/v174_features_doc doc/ocr_backends/ocr_backends_doc doc/observability/observability_doc doc/operations_layer/operations_layer_doc diff --git a/docs/source/Zh/doc/new_features/v171_features_doc.rst b/docs/source/Zh/doc/new_features/v171_features_doc.rst new file mode 100644 index 00000000..3eb34d6d --- /dev/null +++ b/docs/source/Zh/doc/new_features/v171_features_doc.rst @@ -0,0 +1,37 @@ +次像素樣板比對精修 +================== + +每個比對器(``match_template`` / ``match_rotated`` / ``match_masked``)都直接從 +``cv2.minMaxLoc`` 回傳*整數*像素座標。對拖曳把手、細滑桿、次像素渲染的目標或高 DPI 顯示器 +而言,這種整數捨入是主要的點擊落點誤差來源。``subpixel_match`` 以拋物線擬合整數極大值周圍的 +3x3 分數鄰域(x 與 y 各自獨立,即標準 NCC 次像素法),把峰值精修到像素的分數位。 + +本功能重用 ``visual_match._score_map``(公開比對器丟棄的完整 ``matchTemplate`` 曲面); +不重複任何比對程式。``haystack`` 可注入;分析可在合成陣列上單元測試。不匯入 ``PySide6``。 + +無頭 API +-------- + +.. code-block:: python + + from je_auto_control import match_subpixel, refine_peak + + hit = match_subpixel("slider_thumb.png", min_score=0.8) + if hit: + # cx / cy 為浮點數——傳給支援次像素的拖曳 / 移動 + move_to(hit.cx, hit.cy) + + # 可用於任何分數曲面的擬合原語 + offset_x, offset_y = refine_peak(score_map, (peak_x, peak_y)) + +``match_subpixel`` 回傳 ``SubPixelMatch``(整數 ``x`` / ``y`` / ``width`` / ``height`` / +``score`` 加上浮點 ``cx`` / ``cy`` 與套用的 ``offset_x`` / ``offset_y``),或在低於 +``min_score`` 時回傳 ``None``。``refine_peak`` 回傳峰值相對鄰點的 ``[-0.5, 0.5]`` 拋物線擬合 +偏移——可用於任何相關性曲面。 + +執行器指令 +---------- + +``AC_match_subpixel``(``template`` / ``min_score`` / ``region`` / ``method`` → +``{found, match}``)以 MCP 工具 ``ac_match_subpixel``(唯讀)及 Script Builder 指令 +**Match Template (sub-pixel)**(位於 **Image** 分類下)形式提供。 diff --git a/docs/source/Zh/doc/new_features/v172_features_doc.rst b/docs/source/Zh/doc/new_features/v172_features_doc.rst new file mode 100644 index 00000000..eb4c5389 --- /dev/null +++ b/docs/source/Zh/doc/new_features/v172_features_doc.rst @@ -0,0 +1,42 @@ +Grounding 自我一致性(提案共識) +================================ + +一個目標可同時以多種方式 grounding——set-of-marks、OCR、樣板比對、a11y 樹,或模型的 N 次 +抽樣——而它們未必一致。``ab_locator`` / ``element_scoring`` 依歷史可靠度排序定位器*策略*, +卻從不把*同時*的提案融合成單一共識點並附離散度指標;``action_grounding.snap_to_element`` +把*單一*座標貼到最近元素,沒有「不一致」的概念。``grounding_consensus`` 將候選點聚類(或對 +候選元素投票),回傳一致同意的目標加上 *agreement* 比例與 *spread*,並標記低一致度目標, +讓呼叫端可改為放大 / 詢問人工,而非盲目點擊。 + +純標準函式庫幾何;確定性、可在無裝置下單元測試。不匯入 ``PySide6``。 + +無頭 API +-------- + +.. code-block:: python + + from je_auto_control import consensus_point, consensus_element, is_confident + + proposals = [[480, 260], [484, 258], [477, 263], [900, 100]] # 一個離群 + result = consensus_point(proposals, cluster_radius=24) + if is_confident(result, min_agreement=0.6): + click(*result.point) + else: + print("提案不一致:", result.agreement, result.spread) + + # 或把提案投票到已知元素清單 + element, agreement = consensus_element(proposals, elements) + +``consensus_point`` 將 ``[x, y]`` 或 ``[x, y, weight]`` 候選聚類,回傳 ``ConsensusResult`` +(``point`` / ``agreement``——勝出群權重佔總權重 / ``spread`` / ``n_clusters``),空輸入回傳 +``None``。``consensus_element`` 把每個提案投到最近元素,回傳 ``(element, agreement)``。 +``is_confident`` 對 agreement 設門檻。 + +執行器指令 +---------- + +``AC_consensus_point``(``candidates`` / ``cluster_radius`` → ``{found, result}``)與 +``AC_consensus_element``(``candidates`` / ``elements`` → ``{found, element, agreement}``)。 +兩者以 MCP 工具 ``ac_consensus_point`` / ``ac_consensus_element``(唯讀)及 Script Builder 指令 +**Grounding Consensus Point** / **Grounding Consensus Element**(位於 **Native UI** 分類下) +形式提供。 diff --git a/docs/source/Zh/doc/new_features/v173_features_doc.rst b/docs/source/Zh/doc/new_features/v173_features_doc.rst new file mode 100644 index 00000000..2e31f7c8 --- /dev/null +++ b/docs/source/Zh/doc/new_features/v173_features_doc.rst @@ -0,0 +1,37 @@ +欄位感知閱讀順序(XY-Cut) +========================== + +``element_parse.reading_order`` 是平面的上到下 / 左到右排序——在任何多欄頁面都會交錯欄位 +(眾所周知的 naive 排序失敗:它先讀 A 欄第 1 列、再讀 B 欄第 1 列、再讀 A 欄第 2 列…)。 +``reading_flow`` 以遞迴 **XY-cut** 還原正確順序:在最寬的留白谷反覆切分(垂直 gutter → 欄、 +水平 gutter → 列 / 區塊),因此兩欄版面會*完整讀完*A 欄,再讀 B 欄。 + +公開的展平函式命名為 ``flow_order``,以*並列*而非遮蔽 ``element_parse.reading_order``;它回傳 +相同的 ``index`` 標記元素契約,因此是欄位感知的直接升級。純標準函式庫幾何,作用於純框字典 +(不需影像、不需 OCR 引擎);重用 ``table_grid_fill`` 的框邊界讀取器。不匯入 ``PySide6``。 + +無頭 API +-------- + +.. code-block:: python + + from je_auto_control import flow_order, xy_cut, to_blocks + + # 兩欄各兩列:讀作 A1, A2, B1, B2——而非 A1, B1, A2, B2 + for element in flow_order(ocr_boxes, min_gap=12): + print(element["index"], element["text"]) + + tree = xy_cut(ocr_boxes, min_gap=12) # {type, axis, children|boxes} + blocks = to_blocks(tree) # 依閱讀順序的葉區塊 + +``flow_order`` 以欄位感知閱讀順序回傳各框,每個帶 ``index``。``xy_cut`` 回傳遞迴區域樹 +(每個節點為對 ``axis`` ``"x"`` / ``"y"`` 的 ``split`` 或框的 ``leaf``)。``to_blocks`` 把樹 +展平為依序的葉區塊。``min_gap`` 為被視為欄 / 列分隔的最小留白谷。 + +執行器指令 +---------- + +``AC_flow_order``(``boxes`` / ``min_gap`` → ``{count, elements}``)與 ``AC_xy_cut`` +(``boxes`` / ``min_gap`` → ``{tree}``)。兩者以 MCP 工具 ``ac_flow_order`` / ``ac_xy_cut`` +(唯讀)及 Script Builder 指令 **Reading Order (column-aware)** / **XY-Cut Region Tree** +(位於 **OCR** 分類下)形式提供。 diff --git a/docs/source/Zh/doc/new_features/v174_features_doc.rst b/docs/source/Zh/doc/new_features/v174_features_doc.rst new file mode 100644 index 00000000..b2827984 --- /dev/null +++ b/docs/source/Zh/doc/new_features/v174_features_doc.rst @@ -0,0 +1,36 @@ +OCR 行的段落與清單分組 +====================== + +``text_regions.find_text_lines`` 把字形併成行,但沒有任何功能把那些*行*分組成段落或偵測 +清單——``ocr/structure`` 止於平面列。``text_blocks`` 補上這點:``group_paragraphs`` 在垂直 +間距超過中位行高的某倍數處把行切成段落(標準的留白分組啟發法),而 ``detect_lists`` 以前導 +標記與左縮排辨識項目符號 / 編號項目。 + +純標準函式庫,作用於純行字典(text + bbox);可在無影像、無 OCR 引擎下完整單元測試。重用 +``table_grid_fill`` 的框邊界讀取器。不匯入 ``PySide6``。 + +無頭 API +-------- + +.. code-block:: python + + from je_auto_control import group_paragraphs, detect_lists + + for para in group_paragraphs(ocr_lines, line_gap_factor=1.6): + print(para["n_lines"], para["text"]) + + for item in detect_lists(ocr_lines): + print(item["marker"], item["indent"], item["text"]) + +``group_paragraphs`` 回傳段落字典(``left`` / ``top`` / ``right`` / ``bottom`` / ``text`` / +``n_lines``)——當與前一行底部的間距超過 ``line_gap_factor`` 乘以中位行高時,開始新段落。 +``detect_lists`` 回傳文字以項目符號(``•`` / ``-`` / ``*``)或序號(``1.`` / ``2)`` / +``a.``)開頭的行,每個為 ``{text, marker, indent, box}``(``indent`` 為左 x,供巢狀用)。 + +執行器指令 +---------- + +``AC_group_paragraphs``(``lines`` / ``line_gap_factor`` → ``{count, paragraphs}``)與 +``AC_detect_lists``(``lines`` → ``{count, items}``)。兩者以 MCP 工具 ``ac_group_paragraphs`` / +``ac_detect_lists``(唯讀)及 Script Builder 指令 **Group Paragraphs** / **Detect Lists** +(位於 **OCR** 分類下)形式提供。 diff --git a/docs/source/Zh/zh_index.rst b/docs/source/Zh/zh_index.rst index a8364cd9..198b3e6a 100644 --- a/docs/source/Zh/zh_index.rst +++ b/docs/source/Zh/zh_index.rst @@ -193,6 +193,10 @@ AutoControl 所有功能的完整使用指南。 doc/new_features/v168_features_doc doc/new_features/v169_features_doc doc/new_features/v170_features_doc + doc/new_features/v171_features_doc + doc/new_features/v172_features_doc + doc/new_features/v173_features_doc + doc/new_features/v174_features_doc doc/ocr_backends/ocr_backends_doc doc/observability/observability_doc doc/operations_layer/operations_layer_doc diff --git a/je_auto_control/__init__.py b/je_auto_control/__init__.py index c01e1065..0be599b3 100644 --- a/je_auto_control/__init__.py +++ b/je_auto_control/__init__.py @@ -295,6 +295,10 @@ from je_auto_control.utils.match_autothresh import ( auto_threshold, match_auto, ) +# Sub-pixel refinement of a template match (quadratic peak fit) +from je_auto_control.utils.subpixel_match import ( + SubPixelMatch, match_subpixel, refine_peak, +) # Coarse labelled cell grid for VLM grounding (point <-> cell mapping) from je_auto_control.utils.screen_grid import ( GridCell, cell_for_point, grid_cells, point_for_cell, @@ -311,6 +315,14 @@ from je_auto_control.utils.column_layout import ( assign_columns, column_gutters, detect_borderless_table, vertical_projection, ) +# Column-aware reading order via recursive XY-cut +from je_auto_control.utils.reading_flow import ( + flow_order, to_blocks, xy_cut, +) +# Group OCR lines into paragraphs and bulleted / numbered lists +from je_auto_control.utils.text_blocks import ( + detect_lists, group_paragraphs, +) # Associate form labels with values (multi-direction) + checkbox state from je_auto_control.utils.form_fields import ( associate_fields, checkbox_state, match_labels_to_widgets, @@ -327,6 +339,10 @@ from je_auto_control.utils.step_repair import ( RepairOutcome, RepairPolicy, next_tactic, plan_repair, run_with_repair, ) +# Self-consistency over multiple grounding proposals for one target +from je_auto_control.utils.grounding_consensus import ( + ConsensusResult, consensus_element, consensus_point, is_confident, +) # Locate on-screen regions by colour (mask + connected components) from je_auto_control.utils.color_region import ( find_color_region, find_color_regions, @@ -1246,6 +1262,9 @@ def start_autocontrol_gui(*args, **kwargs): "edge_match", "edge_match_all", "chamfer_distance", + "SubPixelMatch", + "match_subpixel", + "refine_peak", "GridCell", "grid_cells", "cell_for_point", @@ -1261,6 +1280,11 @@ def start_autocontrol_gui(*args, **kwargs): "column_gutters", "assign_columns", "detect_borderless_table", + "xy_cut", + "flow_order", + "to_blocks", + "group_paragraphs", + "detect_lists", "associate_fields", "match_labels_to_widgets", "checkbox_state", @@ -1276,6 +1300,10 @@ def start_autocontrol_gui(*args, **kwargs): "plan_repair", "next_tactic", "run_with_repair", + "ConsensusResult", + "consensus_point", + "consensus_element", + "is_confident", "find_color_region", "find_color_regions", "ssim_compare", diff --git a/je_auto_control/gui/script_builder/command_schema.py b/je_auto_control/gui/script_builder/command_schema.py index ba6bb37a..a022c6d5 100644 --- a/je_auto_control/gui/script_builder/command_schema.py +++ b/je_auto_control/gui/script_builder/command_schema.py @@ -396,6 +396,17 @@ def _add_image_specs(specs: List[CommandSpec]) -> None: ), description="Find every edge-shape (Chamfer) match (NMS-deduped).", )) + specs.append(CommandSpec( + "AC_match_subpixel", "Image", "Match Template (sub-pixel)", + fields=( + FieldSpec("template", FieldType.FILE_PATH), + FieldSpec("min_score", FieldType.FLOAT, optional=True, default=0.0, + min_value=0.0, max_value=1.0), + FieldSpec("region", FieldType.STRING, optional=True, + placeholder=_REGION_PLACEHOLDER), + ), + description="Match with a sub-pixel-refined centre (drag / slider precision).", + )) specs.append(CommandSpec( "AC_grid_cells", "Image", "Grid Cells (coarse grounding)", fields=( @@ -777,6 +788,45 @@ def _add_ocr_specs(specs: List[CommandSpec]) -> None: ), description="Match each checkbox/radio/input to its nearest label by centre.", )) + specs.append(CommandSpec( + "AC_flow_order", "OCR", "Reading Order (column-aware)", + fields=( + FieldSpec("boxes", FieldType.STRING, + placeholder='[{"x":0,"y":0,"width":40,"height":20,' + '"text":"A1"}]'), + FieldSpec("min_gap", FieldType.INT, optional=True, default=12), + ), + description="Column-aware reading order via XY-cut (reads down columns).", + )) + specs.append(CommandSpec( + "AC_xy_cut", "OCR", "XY-Cut Region Tree", + fields=( + FieldSpec("boxes", FieldType.STRING, + placeholder='[{"x":0,"y":0,"width":40,"height":20}]'), + FieldSpec("min_gap", FieldType.INT, optional=True, default=12), + ), + description="Recursive XY-cut region tree (split at widest whitespace).", + )) + specs.append(CommandSpec( + "AC_group_paragraphs", "OCR", "Group Paragraphs", + fields=( + FieldSpec("lines", FieldType.STRING, + placeholder='[{"x":0,"y":0,"width":200,"height":20,' + '"text":"line"}]'), + FieldSpec("line_gap_factor", FieldType.FLOAT, optional=True, + default=1.6), + ), + description="Group OCR lines into paragraphs by vertical spacing.", + )) + specs.append(CommandSpec( + "AC_detect_lists", "OCR", "Detect Lists", + fields=( + FieldSpec("lines", FieldType.STRING, + placeholder='[{"x":0,"y":0,"width":200,"height":20,' + '"text":"* item"}]'), + ), + description="Detect bulleted / numbered list items among OCR lines.", + )) specs.append(CommandSpec( "AC_scroll_to_find", "OCR", "Scroll Until Visible", fields=( @@ -3214,6 +3264,25 @@ def _add_set_of_marks_specs(specs: List[CommandSpec]) -> None: ), description="Ordered repair tactics for a failed/no-effect action verdict.", )) + specs.append(CommandSpec( + "AC_consensus_point", "Native UI", "Grounding Consensus Point", + fields=( + FieldSpec("candidates", FieldType.STRING, + placeholder="[[100, 100], [104, 98], [97, 103]]"), + FieldSpec("cluster_radius", FieldType.INT, optional=True, default=24), + ), + description="Agreed target point from clustered grounding proposals.", + )) + specs.append(CommandSpec( + "AC_consensus_element", "Native UI", "Grounding Consensus Element", + fields=( + FieldSpec("candidates", FieldType.STRING, + placeholder="[[8, 8], [12, 10]]"), + FieldSpec("elements", FieldType.STRING, + placeholder='[{"role":"button","x":0,"y":0}]'), + ), + description="Vote grounding proposals to the nearest element.", + )) specs.append(CommandSpec( "AC_validate_action", "Native UI", "Validate / Snap Action", fields=( diff --git a/je_auto_control/utils/executor/action_executor.py b/je_auto_control/utils/executor/action_executor.py index b49081c6..cc4bd1b7 100644 --- a/je_auto_control/utils/executor/action_executor.py +++ b/je_auto_control/utils/executor/action_executor.py @@ -3387,6 +3387,19 @@ def _edge_match_all(template: str, min_score: Any = 0.7, max_results: Any = 20, return {"count": len(matches), "matches": [m.to_dict() for m in matches]} +def _match_subpixel(template: str, min_score: Any = 0.0, region: Any = None, + method: str = "ccoeff_normed") -> Dict[str, Any]: + """Adapter: best template match with a sub-pixel-refined centre.""" + import json + from je_auto_control.utils.subpixel_match import match_subpixel + if isinstance(region, str): + region = json.loads(region) if region.strip() else None + match = match_subpixel(template, region=region, method=method, + min_score=float(min_score)) + return {"found": match is not None, + "match": match.to_dict() if match else None} + + def _region_arg(value: Any) -> Optional[List[int]]: """Coerce a JSON-string / list region arg into a list of ints, or None.""" import json @@ -3485,6 +3498,45 @@ def _match_labels_to_widgets(labels: Any, widgets: Any) -> Dict[str, Any]: return {"count": len(pairs), "pairs": pairs} +def _flow_order(boxes: Any, min_gap: Any = 12) -> Dict[str, Any]: + """Adapter: column-aware reading order of OCR boxes (XY-cut).""" + import json + from je_auto_control.utils.reading_flow import flow_order + if isinstance(boxes, str): + boxes = json.loads(boxes) + ordered = flow_order(boxes, min_gap=int(min_gap)) + return {"count": len(ordered), "elements": ordered} + + +def _xy_cut(boxes: Any, min_gap: Any = 12) -> Dict[str, Any]: + """Adapter: recursive XY-cut region tree of OCR boxes.""" + import json + from je_auto_control.utils.reading_flow import xy_cut + if isinstance(boxes, str): + boxes = json.loads(boxes) + return {"tree": xy_cut(boxes, min_gap=int(min_gap))} + + +def _group_paragraphs(lines: Any, line_gap_factor: Any = 1.6) -> Dict[str, Any]: + """Adapter: group OCR lines into paragraphs by vertical spacing.""" + import json + from je_auto_control.utils.text_blocks import group_paragraphs + if isinstance(lines, str): + lines = json.loads(lines) + paragraphs = group_paragraphs(lines, line_gap_factor=float(line_gap_factor)) + return {"count": len(paragraphs), "paragraphs": paragraphs} + + +def _detect_lists(lines: Any) -> Dict[str, Any]: + """Adapter: detect bulleted / numbered list items among OCR lines.""" + import json + from je_auto_control.utils.text_blocks import detect_lists + if isinstance(lines, str): + lines = json.loads(lines) + items = detect_lists(lines) + return {"count": len(items), "items": items} + + def _find_color_region(rgb: Any, tolerance: Any = 20, min_area: Any = 50, region: Any = None) -> Dict[str, Any]: """Adapter: locate coloured regions on the screen, largest first.""" @@ -4155,6 +4207,31 @@ def _plan_repair(verdict: Any, max_attempts: Any = 3) -> Dict[str, Any]: return {"count": len(tactics), "tactics": tactics} +def _consensus_point(candidates: Any, cluster_radius: Any = 24) -> Dict[str, Any]: + """Adapter: agreed target point from clustered grounding proposals.""" + import json + from je_auto_control.utils.grounding_consensus import consensus_point + if isinstance(candidates, str): + candidates = json.loads(candidates) + result = consensus_point(candidates, cluster_radius=float(cluster_radius)) + return {"found": result is not None, + "result": result.to_dict() if result else None} + + +def _consensus_element(candidates: Any, elements: Any) -> Dict[str, Any]: + """Adapter: vote grounding proposals to the nearest element.""" + import json + from je_auto_control.utils.grounding_consensus import consensus_element + if isinstance(candidates, str): + candidates = json.loads(candidates) + if isinstance(elements, str): + elements = json.loads(elements) + winner = consensus_element(candidates, elements) + return {"found": winner is not None, + "element": winner[0] if winner else None, + "agreement": winner[1] if winner else 0.0} + + def _validate_action(action: Any, screen: Any = None, targets: Any = None) -> Dict[str, Any]: """Adapter: validate a coordinate action (bounds + optional snap-to-target).""" @@ -5986,6 +6063,7 @@ def __init__(self): "AC_match_auto": _match_auto, "AC_edge_match": _edge_match, "AC_edge_match_all": _edge_match_all, + "AC_match_subpixel": _match_subpixel, "AC_grid_cells": _grid_cells, "AC_cell_for_point": _cell_for_point, "AC_point_for_cell": _point_for_cell, @@ -5994,6 +6072,10 @@ def __init__(self): "AC_detect_borderless_table": _detect_borderless_table, "AC_associate_fields": _associate_fields, "AC_match_labels_to_widgets": _match_labels_to_widgets, + "AC_flow_order": _flow_order, + "AC_xy_cut": _xy_cut, + "AC_group_paragraphs": _group_paragraphs, + "AC_detect_lists": _detect_lists, "AC_ssim_compare": _ssim_compare, "AC_ssim_changed_regions": _ssim_changed_regions, "AC_feature_match": _feature_match, @@ -6037,6 +6119,8 @@ def __init__(self): "AC_effect_near_point": _effect_near_point, "AC_check_postcondition": _check_postcondition, "AC_plan_repair": _plan_repair, + "AC_consensus_point": _consensus_point, + "AC_consensus_element": _consensus_element, "AC_validate_action": _validate_action, "AC_replay_trace": _replay_trace, "AC_match_elements": _match_elements, diff --git a/je_auto_control/utils/grounding_consensus/__init__.py b/je_auto_control/utils/grounding_consensus/__init__.py new file mode 100644 index 00000000..a6297444 --- /dev/null +++ b/je_auto_control/utils/grounding_consensus/__init__.py @@ -0,0 +1,8 @@ +"""Self-consistency over multiple grounding proposals for one target.""" +from je_auto_control.utils.grounding_consensus.grounding_consensus import ( + ConsensusResult, consensus_element, consensus_point, is_confident, +) + +__all__ = [ + "ConsensusResult", "consensus_point", "consensus_element", "is_confident", +] diff --git a/je_auto_control/utils/grounding_consensus/grounding_consensus.py b/je_auto_control/utils/grounding_consensus/grounding_consensus.py new file mode 100644 index 00000000..096aa686 --- /dev/null +++ b/je_auto_control/utils/grounding_consensus/grounding_consensus.py @@ -0,0 +1,119 @@ +"""Self-consistency over multiple grounding proposals for one target. + +A target can be grounded several ways at once — set-of-marks, OCR, template match, the a11y +tree, or N samples from a model — and they don't always agree. ``ab_locator`` / +``element_scoring`` rank locator *strategies* by historical reliability but never fuse +*simultaneous* proposals into one consensus point with a dispersion metric, and +``action_grounding.snap_to_element`` snaps a *single* coordinate to the nearest element with no +notion of disagreement. ``grounding_consensus`` clusters the candidate points (or votes +candidate elements), returns the agreed target plus an *agreement* fraction and *spread*, and +flags low-agreement targets so the caller can zoom / ask a human instead of clicking blind. + +Pure-stdlib geometry; deterministic and unit-testable with no device. Imports no ``PySide6``. +""" +from dataclasses import asdict, dataclass +from typing import Any, Dict, List, Optional, Sequence, Tuple + +Candidate = Any +Element = Dict[str, Any] + + +@dataclass(frozen=True) +class ConsensusResult: + """The agreed target point plus its agreement fraction, spread and cluster count.""" + + point: List[int] + agreement: float + spread: float + n_clusters: int + + def to_dict(self) -> Dict[str, Any]: + """Return the result as a plain dict.""" + return asdict(self) + + +def _xyw(candidate: Candidate) -> Tuple[float, float, float]: + """Normalise a candidate to ``(x, y, weight)`` from a dict or ``[x, y[, w]]``.""" + if isinstance(candidate, dict): + return (float(candidate.get("x", 0)), float(candidate.get("y", 0)), + float(candidate.get("weight", 1.0))) + seq = list(candidate) + return float(seq[0]), float(seq[1]), float(seq[2]) if len(seq) > 2 else 1.0 + + +def _assign(point: Tuple[float, float, float], clusters: List[Dict[str, Any]], + radius: float) -> bool: + """Add ``point`` to the first cluster whose centroid is within ``radius``.""" + x, y, weight = point + for cluster in clusters: + cx, cy = cluster["sx"] / cluster["w"], cluster["sy"] / cluster["w"] + if abs(x - cx) <= radius and abs(y - cy) <= radius: + cluster["sx"] += x * weight + cluster["sy"] += y * weight + cluster["w"] += weight + cluster["members"].append((x, y)) + return True + return False + + +def consensus_point(candidates: Sequence[Candidate], *, + cluster_radius: float = 24) -> Optional[ConsensusResult]: + """Cluster candidate points and return the agreed target, or ``None`` if empty. + + ``agreement`` is the winning cluster's weight over the total; ``spread`` is the + largest member distance from its centroid; ``n_clusters`` is how many groups formed. + """ + points = [_xyw(c) for c in candidates] + if not points: + return None + total = sum(weight for _, _, weight in points) + clusters: List[Dict[str, Any]] = [] + for point in points: + if not _assign(point, clusters, float(cluster_radius)): + x, y, weight = point + clusters.append({"sx": x * weight, "sy": y * weight, "w": weight, + "members": [(x, y)]}) + best = max(clusters, key=lambda c: c["w"]) + cx, cy = best["sx"] / best["w"], best["sy"] / best["w"] + spread = max(abs(mx - cx) + abs(my - cy) for mx, my in best["members"]) + return ConsensusResult([int(round(cx)), int(round(cy))], + round(best["w"] / total, 4), round(spread, 2), + len(clusters)) + + +def _center(element: Element) -> Tuple[float, float]: + return (float(element.get("x", 0)) + float(element.get("width", 0)) / 2.0, + float(element.get("y", 0)) + float(element.get("height", 0)) / 2.0) + + +def _nearest_index(x: float, y: float, elements: Sequence[Element]) -> int: + """Index of the element whose centre is closest (Manhattan) to ``(x, y)``.""" + best_index, best_distance = 0, None + for index, element in enumerate(elements): + cx, cy = _center(element) + distance = abs(cx - x) + abs(cy - y) + if best_distance is None or distance < best_distance: + best_index, best_distance = index, distance + return best_index + + +def consensus_element(candidates: Sequence[Candidate], + elements: Sequence[Element] + ) -> Optional[Tuple[Element, float]]: + """Vote each candidate point to its nearest element; return ``(winner, agreement)``.""" + if not elements or not candidates: + return None + votes = [0.0] * len(elements) + total = 0.0 + for candidate in candidates: + x, y, weight = _xyw(candidate) + votes[_nearest_index(x, y, elements)] += weight + total += weight + best = max(range(len(votes)), key=votes.__getitem__) + return elements[best], round(votes[best] / total, 4) + + +def is_confident(result: Optional[ConsensusResult], *, + min_agreement: float = 0.6) -> bool: + """Return whether a consensus result clears the ``min_agreement`` threshold.""" + return result is not None and result.agreement >= float(min_agreement) diff --git a/je_auto_control/utils/mcp_server/tools/_factories.py b/je_auto_control/utils/mcp_server/tools/_factories.py index e76c5451..2793876e 100644 --- a/je_auto_control/utils/mcp_server/tools/_factories.py +++ b/je_auto_control/utils/mcp_server/tools/_factories.py @@ -3409,6 +3409,36 @@ def observation_tools() -> List[MCPTool]: handler=h.plan_repair, annotations=READ_ONLY, ), + MCPTool( + name="ac_consensus_point", + description=("Fuse several grounding proposals into one agreed point: " + "cluster 'candidates' ([[x,y] or [x,y,weight]]) and return " + "{found, result:{point,agreement,spread,n_clusters}}. Low " + "'agreement' = the proposals disagree — don't click blind. " + "'cluster_radius'."), + input_schema=schema({ + "candidates": {"type": "array", + "items": {"type": "array", + "items": {"type": "number"}}}, + "cluster_radius": {"type": "number"}}, + required=["candidates"]), + handler=h.consensus_point, + annotations=READ_ONLY, + ), + MCPTool( + name="ac_consensus_element", + description=("Vote grounding 'candidates' ([[x,y]…]) to the nearest of " + "'elements' (role/name/x/y/width/height dicts) and return " + "{found, element, agreement}."), + input_schema=schema({ + "candidates": {"type": "array", + "items": {"type": "array", + "items": {"type": "number"}}}, + "elements": {"type": "array", "items": {"type": "object"}}}, + required=["candidates", "elements"]), + handler=h.consensus_element, + annotations=READ_ONLY, + ), ] @@ -3759,6 +3789,22 @@ def rotated_match_tools() -> List[MCPTool]: handler=h.edge_match_all, annotations=READ_ONLY, ), + MCPTool( + name="ac_match_subpixel", + description=("Find 'template' and refine the centre to SUB-PIXEL precision " + "(quadratic fit on the score peak). Returns {found, match:" + "{x,y,width,height,score,cx,cy,offset_x,offset_y,center}} — " + "cx/cy are float for drag / slider / high-DPI accuracy. " + "'min_score', 'region', 'method'."), + input_schema=schema({ + "template": {"type": "string"}, + "min_score": {"type": "number"}, + "region": {"type": "array", "items": {"type": "integer"}}, + "method": {"type": "string"}}, + required=["template"]), + handler=h.match_subpixel, + annotations=READ_ONLY, + ), ] @@ -3878,6 +3924,57 @@ def screen_grid_tools() -> List[MCPTool]: handler=h.match_labels_to_widgets, annotations=READ_ONLY, ), + MCPTool( + name="ac_flow_order", + description=("Column-aware reading order of OCR 'boxes' via XY-cut — reads " + "DOWN each column before the next (vs the naive sort that " + "interleaves columns). Returns {count, elements} each tagged " + "with an 'index'. 'min_gap' is the smallest whitespace valley " + "treated as a column / row break."), + input_schema=schema({ + "boxes": {"type": "array", "items": {"type": "object"}}, + "min_gap": {"type": "integer"}}, + required=["boxes"]), + handler=h.flow_order, + annotations=READ_ONLY, + ), + MCPTool( + name="ac_xy_cut", + description=("Recursive XY-cut region tree of OCR 'boxes' (splits at the " + "widest whitespace valley). Returns {tree:{type,axis," + "children|boxes}}."), + input_schema=schema({ + "boxes": {"type": "array", "items": {"type": "object"}}, + "min_gap": {"type": "integer"}}, + required=["boxes"]), + handler=h.xy_cut, + annotations=READ_ONLY, + ), + MCPTool( + name="ac_group_paragraphs", + description=("Group OCR 'lines' (text + bbox) into paragraphs: a new " + "paragraph starts where the vertical gap exceeds " + "'line_gap_factor' x the median line height. Returns {count, " + "paragraphs:[{left,top,right,bottom,text,n_lines}]}."), + input_schema=schema({ + "lines": {"type": "array", "items": {"type": "object"}}, + "line_gap_factor": {"type": "number"}}, + required=["lines"]), + handler=h.group_paragraphs, + annotations=READ_ONLY, + ), + MCPTool( + name="ac_detect_lists", + description=("Detect bulleted / numbered list items among OCR 'lines' by " + "their leading marker (•/-/* or 1./2)/a.). Returns {count, " + "items:[{text,marker,indent,box}]}; 'indent' is the left x " + "for nesting."), + input_schema=schema({ + "lines": {"type": "array", "items": {"type": "object"}}}, + required=["lines"]), + handler=h.detect_lists, + annotations=READ_ONLY, + ), ] diff --git a/je_auto_control/utils/mcp_server/tools/_handlers.py b/je_auto_control/utils/mcp_server/tools/_handlers.py index bec7c7fd..8a6fc2ef 100644 --- a/je_auto_control/utils/mcp_server/tools/_handlers.py +++ b/je_auto_control/utils/mcp_server/tools/_handlers.py @@ -2137,6 +2137,11 @@ def edge_match_all(template, min_score=0.7, max_results=20, nms_iou=0.3, return _edge_match_all(template, min_score, max_results, nms_iou, region) +def match_subpixel(template, min_score=0.0, region=None, method="ccoeff_normed"): + from je_auto_control.utils.executor.action_executor import _match_subpixel + return _match_subpixel(template, min_score, region, method) + + def grid_cells(rows, cols, region=None): from je_auto_control.utils.executor.action_executor import _grid_cells return _grid_cells(rows, cols, region) @@ -2180,6 +2185,26 @@ def match_labels_to_widgets(labels, widgets): return _match_labels_to_widgets(labels, widgets) +def flow_order(boxes, min_gap=12): + from je_auto_control.utils.executor.action_executor import _flow_order + return _flow_order(boxes, min_gap) + + +def xy_cut(boxes, min_gap=12): + from je_auto_control.utils.executor.action_executor import _xy_cut + return _xy_cut(boxes, min_gap) + + +def group_paragraphs(lines, line_gap_factor=1.6): + from je_auto_control.utils.executor.action_executor import _group_paragraphs + return _group_paragraphs(lines, line_gap_factor) + + +def detect_lists(lines): + from je_auto_control.utils.executor.action_executor import _detect_lists + return _detect_lists(lines) + + def find_color_region(rgb, tolerance=20, min_area=50, region=None): from je_auto_control.utils.executor.action_executor import ( _find_color_region) @@ -2443,6 +2468,16 @@ def plan_repair(verdict, max_attempts=3): return _plan_repair(verdict, max_attempts) +def consensus_point(candidates, cluster_radius=24): + from je_auto_control.utils.executor.action_executor import _consensus_point + return _consensus_point(candidates, cluster_radius) + + +def consensus_element(candidates, elements): + from je_auto_control.utils.executor.action_executor import _consensus_element + return _consensus_element(candidates, elements) + + def validate_action(action, screen=None, targets=None): from je_auto_control.utils.executor.action_executor import _validate_action return _validate_action(action, screen, targets) diff --git a/je_auto_control/utils/reading_flow/__init__.py b/je_auto_control/utils/reading_flow/__init__.py new file mode 100644 index 00000000..8bc59a99 --- /dev/null +++ b/je_auto_control/utils/reading_flow/__init__.py @@ -0,0 +1,6 @@ +"""Column-aware reading order via recursive XY-cut.""" +from je_auto_control.utils.reading_flow.reading_flow import ( + flow_order, to_blocks, xy_cut, +) + +__all__ = ["xy_cut", "flow_order", "to_blocks"] diff --git a/je_auto_control/utils/reading_flow/reading_flow.py b/je_auto_control/utils/reading_flow/reading_flow.py new file mode 100644 index 00000000..e07f829a --- /dev/null +++ b/je_auto_control/utils/reading_flow/reading_flow.py @@ -0,0 +1,113 @@ +"""Column-aware reading order via recursive XY-cut. + +``element_parse.reading_order`` is a flat top-to-bottom / left-to-right sort — it interleaves +columns on any multi-column page (the well-documented naive-sort failure: it reads row 1 of +column A, then row 1 of column B, then row 2 of A …). ``reading_flow`` recovers the correct +order with recursive XY-cut: it repeatedly splits the boxes at the widest whitespace valley +(vertical gutter → columns, horizontal gutter → rows/blocks), so a two-column layout is read +*down* column A fully, then column B. It returns the region tree and a column-aware flattened +order; the public flattener is named ``flow_order`` to sit beside (not shadow) +``reading_order``. + +Pure-stdlib geometry over plain box dicts (no image, no OCR engine); reuses ``table_grid_fill``'s +box-bounds reader. Imports no ``PySide6``. +""" +from typing import Any, Dict, List, Optional, Sequence, Tuple + +from je_auto_control.utils.table_grid_fill.table_grid_fill import _box_bounds + +Box = Dict[str, Any] + + +def _axis_bounds(box: Box, axis: str) -> Tuple[int, int]: + left, top, right, bottom = _box_bounds(box) + return (left, right) if axis == "x" else (top, bottom) + + +def _center_on(box: Box, axis: str) -> float: + low, high = _axis_bounds(box, axis) + return (low + high) / 2.0 + + +def _widest_gap(boxes: Sequence[Box], axis: str, min_gap: int): + """Return ``(position, width)`` of the widest whitespace gap on ``axis``, or ``None``.""" + spans = sorted(_axis_bounds(box, axis) for box in boxes) + best_pos: Optional[float] = None + best_gap = 0.0 + cur_end = spans[0][1] + for low, high in spans[1:]: + gap = low - cur_end + if gap > best_gap: + best_gap, best_pos = gap, cur_end + gap / 2.0 + cur_end = max(cur_end, high) + if best_pos is not None and best_gap >= min_gap: + return best_pos, best_gap + return None + + +def _choose_axis(boxes: Sequence[Box], min_gap: int): + """Return ``(axis, position)`` of the widest gap across both axes, or ``None``.""" + candidates = [] + for axis in ("x", "y"): + found = _widest_gap(boxes, axis, min_gap) + if found is not None: + candidates.append((found[1], axis, found[0])) + if not candidates: + return None + _, axis, position = max(candidates) + return axis, position + + +def _cut(boxes: List[Box], min_gap: int, depth: int) -> Dict[str, Any]: + """Recursively XY-cut ``boxes`` into a region tree.""" + if len(boxes) <= 1 or depth <= 0: + return {"type": "leaf", "boxes": list(boxes)} + chosen = _choose_axis(boxes, min_gap) + if chosen is None: + return {"type": "leaf", "boxes": list(boxes)} + axis, position = chosen + first = [box for box in boxes if _center_on(box, axis) < position] + second = [box for box in boxes if _center_on(box, axis) >= position] + if not first or not second: + return {"type": "leaf", "boxes": list(boxes)} + return {"type": "split", "axis": axis, + "children": [_cut(first, min_gap, depth - 1), + _cut(second, min_gap, depth - 1)]} + + +def xy_cut(boxes: Sequence[Box], *, min_gap: int = 12, + max_depth: int = 8) -> Dict[str, Any]: + """Return the recursive XY-cut region tree of ``boxes``.""" + if not boxes: + return {"type": "leaf", "boxes": []} + return _cut(list(boxes), int(min_gap), int(max_depth)) + + +def _flatten(tree: Dict[str, Any]) -> List[Box]: + if tree["type"] == "leaf": + return sorted(tree["boxes"], + key=lambda b: (_box_bounds(b)[1], _box_bounds(b)[0])) + flat: List[Box] = [] + for child in tree["children"]: + flat.extend(_flatten(child)) + return flat + + +def flow_order(boxes: Sequence[Box], *, min_gap: int = 12, + max_depth: int = 8) -> List[Box]: + """Return ``boxes`` in column-aware reading order, each tagged with an ``index``.""" + flat = _flatten(xy_cut(boxes, min_gap=min_gap, max_depth=max_depth)) + return [dict(box, index=i) for i, box in enumerate(flat)] + + +def to_blocks(tree: Dict[str, Any]) -> List[List[Box]]: + """Return the leaf blocks of an XY-cut ``tree`` in reading order.""" + if tree["type"] == "leaf": + if not tree["boxes"]: + return [] + return [sorted(tree["boxes"], + key=lambda b: (_box_bounds(b)[1], _box_bounds(b)[0]))] + blocks: List[List[Box]] = [] + for child in tree["children"]: + blocks.extend(to_blocks(child)) + return blocks diff --git a/je_auto_control/utils/subpixel_match/__init__.py b/je_auto_control/utils/subpixel_match/__init__.py new file mode 100644 index 00000000..8cec6e09 --- /dev/null +++ b/je_auto_control/utils/subpixel_match/__init__.py @@ -0,0 +1,6 @@ +"""Sub-pixel refinement of a template match by quadratic peak fitting.""" +from je_auto_control.utils.subpixel_match.subpixel_match import ( + SubPixelMatch, match_subpixel, refine_peak, +) + +__all__ = ["SubPixelMatch", "match_subpixel", "refine_peak"] diff --git a/je_auto_control/utils/subpixel_match/subpixel_match.py b/je_auto_control/utils/subpixel_match/subpixel_match.py new file mode 100644 index 00000000..25299bef --- /dev/null +++ b/je_auto_control/utils/subpixel_match/subpixel_match.py @@ -0,0 +1,95 @@ +"""Sub-pixel refinement of a template match by quadratic peak fitting. + +Every matcher (``match_template`` / ``match_rotated`` / ``match_masked``) returns *integer* +pixel coordinates straight from ``cv2.minMaxLoc``. For a drag handle, a fine slider, a +sub-pixel-rendered target, or a high-DPI display, that integer rounding is the dominant +click-placement error. ``subpixel_match`` refines the peak to a fraction of a pixel by fitting +a parabola to the 3x3 score neighbourhood around the integer maximum (the standard NCC +sub-pixel method) — independently on x and y. + +It reuses ``visual_match._score_map`` (the full ``matchTemplate`` surface the public matchers +discard); no matching code is duplicated. The ``haystack`` is injectable; the analysis is +unit-testable on synthetic arrays. Imports no ``PySide6``. +""" +from dataclasses import asdict, dataclass +from typing import Any, Dict, List, Optional, Sequence, Tuple + +from je_auto_control.utils.visual_match.visual_match import _score_map + +ImageSource = Any + + +@dataclass(frozen=True) +class SubPixelMatch: + """A match whose centre is refined to sub-pixel precision.""" + + x: int + y: int + width: int + height: int + score: float + cx: float + cy: float + offset_x: float + offset_y: float + + @property + def center(self) -> List[int]: + """The integer centre point ``[x, y]`` (top-left + half size).""" + return [self.x + self.width // 2, self.y + self.height // 2] + + def to_dict(self) -> Dict[str, Any]: + """Return the match as a plain dict including the integer centre point.""" + data = asdict(self) + data["center"] = self.center + return data + + +def _parabola_offset(low: float, mid: float, high: float) -> float: + """Sub-sample offset in [-0.5, 0.5] of a peak from its two neighbours' scores.""" + denominator = low - 2.0 * mid + high + if abs(denominator) < 1e-12: + return 0.0 + offset = 0.5 * (low - high) / denominator + return max(-0.5, min(0.5, offset)) + + +def refine_peak(score_map, peak_xy: Tuple[int, int]) -> Tuple[float, float]: + """Return the sub-pixel ``(offset_x, offset_y)`` of a peak via 3x3 quadratic fit.""" + peak_x, peak_y = int(peak_xy[0]), int(peak_xy[1]) + height, width = score_map.shape + offset_x = offset_y = 0.0 + if 0 < peak_x < width - 1: + offset_x = _parabola_offset(float(score_map[peak_y, peak_x - 1]), + float(score_map[peak_y, peak_x]), + float(score_map[peak_y, peak_x + 1])) + if 0 < peak_y < height - 1: + offset_y = _parabola_offset(float(score_map[peak_y - 1, peak_x]), + float(score_map[peak_y, peak_x]), + float(score_map[peak_y + 1, peak_x])) + return offset_x, offset_y + + +def match_subpixel(template: ImageSource, *, haystack: Optional[ImageSource] = None, + region: Optional[Sequence[int]] = None, + method: str = "ccoeff_normed", + min_score: float = 0.0) -> Optional[SubPixelMatch]: + """Return the best match with a sub-pixel-refined centre, or ``None``. + + The integer top-left and score come from the correlation peak; ``cx`` / ``cy`` add the + quadratic-fit offset to the integer centre for fractional-pixel click placement. + """ + import cv2 + score_map, tmpl = _score_map(template, haystack, region=region, method=method) + if score_map is None: + return None + _, max_val, _, max_loc = cv2.minMaxLoc(score_map) + if max_val < min_score: + return None + peak_x, peak_y = int(max_loc[0]), int(max_loc[1]) + offset_x, offset_y = refine_peak(score_map, (peak_x, peak_y)) + height, width = tmpl.shape[:2] + return SubPixelMatch(peak_x, peak_y, width, height, round(float(max_val), 4), + round(peak_x + width / 2.0 + offset_x, 3), + round(peak_y + height / 2.0 + offset_y, 3), + round(offset_x, 4), round(offset_y, 4)) diff --git a/je_auto_control/utils/text_blocks/__init__.py b/je_auto_control/utils/text_blocks/__init__.py new file mode 100644 index 00000000..3ca86a09 --- /dev/null +++ b/je_auto_control/utils/text_blocks/__init__.py @@ -0,0 +1,6 @@ +"""Group OCR lines into paragraphs and bulleted / numbered lists.""" +from je_auto_control.utils.text_blocks.text_blocks import ( + detect_lists, group_paragraphs, +) + +__all__ = ["group_paragraphs", "detect_lists"] diff --git a/je_auto_control/utils/text_blocks/text_blocks.py b/je_auto_control/utils/text_blocks/text_blocks.py new file mode 100644 index 00000000..36ba4e10 --- /dev/null +++ b/je_auto_control/utils/text_blocks/text_blocks.py @@ -0,0 +1,82 @@ +"""Group OCR lines into paragraphs and bulleted / numbered lists. + +``text_regions.find_text_lines`` merges glyphs into lines, but nothing groups those *lines* +into paragraphs or detects lists — ``ocr/structure`` stops at flat rows. ``text_blocks`` adds +that: ``group_paragraphs`` splits lines into paragraphs wherever the vertical gap exceeds a +multiple of the median line height (the standard whitespace-grouping heuristic), and +``detect_lists`` recognises bulleted / numbered items by their leading marker and left indent. + +Pure-stdlib over plain line dicts (text + bbox); fully unit-testable with no image and no OCR +engine. Reuses ``table_grid_fill``'s box-bounds reader. Imports no ``PySide6``. +""" +import re +from typing import Any, Dict, List, Sequence + +from je_auto_control.utils.table_grid_fill.table_grid_fill import _box_bounds + +Line = Dict[str, Any] + +_MARKER = re.compile(r"^\s*([•‣◦\-\*]|\d+[.)]|[A-Za-z][.)])\s+") + + +def _height(line: Line) -> int: + _, top, _, bottom = _box_bounds(line) + return bottom - top + + +def _make_paragraph(lines: Sequence[Line]) -> Dict[str, Any]: + """Build a paragraph dict (union bbox + joined text) from its lines.""" + bounds = [_box_bounds(line) for line in lines] + left = min(b[0] for b in bounds) + top = min(b[1] for b in bounds) + right = max(b[2] for b in bounds) + bottom = max(b[3] for b in bounds) + text = " ".join(str(line.get("text", "")).strip() for line in lines).strip() + return {"left": left, "top": top, "right": right, "bottom": bottom, + "text": text, "n_lines": len(lines)} + + +def group_paragraphs(lines: Sequence[Line], *, + line_gap_factor: float = 1.6) -> List[Dict[str, Any]]: + """Group lines into paragraphs, splitting where the vertical gap is large. + + A new paragraph starts when the gap from the previous line's bottom exceeds + ``line_gap_factor`` times the median line height. Returns paragraph dicts + (``left`` / ``top`` / ``right`` / ``bottom`` / ``text`` / ``n_lines``). + """ + ordered = sorted(lines, key=lambda line: _box_bounds(line)[1]) + if not ordered: + return [] + heights = sorted(_height(line) for line in ordered) + threshold = heights[len(heights) // 2] * float(line_gap_factor) + paragraphs: List[List[Line]] = [[ordered[0]]] + prev_bottom = _box_bounds(ordered[0])[3] + for line in ordered[1:]: + top, bottom = _box_bounds(line)[1], _box_bounds(line)[3] + if top - prev_bottom > threshold: + paragraphs.append([line]) + else: + paragraphs[-1].append(line) + prev_bottom = bottom + return [_make_paragraph(group) for group in paragraphs] + + +def detect_lists(lines: Sequence[Line]) -> List[Dict[str, Any]]: + """Return the lines that are list items: ``{text, marker, indent, box}``. + + A line is a list item when its text starts with a bullet (``•`` / ``-`` / ``*``) + or an ordinal (``1.`` / ``2)`` / ``a.``); ``indent`` is its left x (for nesting), + ``text`` is the content after the marker. + """ + items: List[Dict[str, Any]] = [] + for line in lines: + text = str(line.get("text", "")) + match = _MARKER.match(text) + if match is None: + continue + left, top, right, bottom = _box_bounds(line) + items.append({"text": text[match.end():].strip(), + "marker": match.group(1), "indent": left, + "box": {"left": left, "top": top, "right": right, + "bottom": bottom}}) + return items diff --git a/test/unit_test/headless/test_grounding_consensus_batch.py b/test/unit_test/headless/test_grounding_consensus_batch.py new file mode 100644 index 00000000..ee39b072 --- /dev/null +++ b/test/unit_test/headless/test_grounding_consensus_batch.py @@ -0,0 +1,63 @@ +"""Headless tests for grounding self-consistency (pure stdlib).""" +import je_auto_control as ac +from je_auto_control.utils.grounding_consensus import ( + consensus_element, consensus_point, is_confident, +) + + +def test_majority_cluster_wins(): + # three proposals agree near (100,100); one outlier far away + candidates = [[100, 100], [104, 98], [97, 103], [500, 400]] + result = consensus_point(candidates, cluster_radius=24) + assert result is not None + assert abs(result.point[0] - 100) <= 5 and abs(result.point[1] - 100) <= 5 + assert result.n_clusters == 2 + assert abs(result.agreement - 0.75) < 1e-9 + + +def test_weights_influence_consensus(): + # the lone heavy-weight proposal outvotes two light ones + candidates = [[10, 10, 1.0], [12, 9, 1.0], [200, 200, 5.0]] + result = consensus_point(candidates, cluster_radius=20) + assert abs(result.point[0] - 200) <= 2 + + +def test_is_confident_threshold(): + strong = consensus_point([[0, 0], [1, 1], [2, 2]]) + assert is_confident(strong, min_agreement=0.6) is True + split = consensus_point([[0, 0], [500, 500]], cluster_radius=10) + assert is_confident(split, min_agreement=0.6) is False + + +def test_consensus_element_votes_nearest(): + elements = [{"x": 0, "y": 0, "width": 20, "height": 20}, + {"x": 200, "y": 0, "width": 20, "height": 20}] + # two votes near element 0, one near element 1 + winner, agreement = consensus_element([[8, 8], [12, 10], [205, 9]], elements) + assert winner is elements[0] + assert abs(agreement - (2 / 3)) < 1e-3 # rounded to 4 dp + + +def test_empty_inputs(): + assert consensus_point([]) is None + assert consensus_element([[1, 1]], []) is None + assert is_confident(None) is False + + +# --- wiring --------------------------------------------------------------- + +def test_wiring(): + known = set(ac.executor.known_commands()) + assert {"AC_consensus_point", "AC_consensus_element"} <= known + from je_auto_control.utils.mcp_server.tools import build_default_tool_registry + names = {t.name for t in build_default_tool_registry()} + assert {"ac_consensus_point", "ac_consensus_element"} <= names + from je_auto_control.gui.script_builder.command_schema import _build_specs + specs = {s.command for s in _build_specs()} + assert {"AC_consensus_point", "AC_consensus_element"} <= specs + + +def test_facade_exports(): + for name in ("consensus_point", "consensus_element", "is_confident", + "ConsensusResult"): + assert hasattr(ac, name) and name in ac.__all__ diff --git a/test/unit_test/headless/test_reading_flow_batch.py b/test/unit_test/headless/test_reading_flow_batch.py new file mode 100644 index 00000000..8837497f --- /dev/null +++ b/test/unit_test/headless/test_reading_flow_batch.py @@ -0,0 +1,64 @@ +"""Headless tests for column-aware reading order (XY-cut, pure stdlib).""" +import je_auto_control as ac +from je_auto_control.utils.reading_flow import flow_order, to_blocks, xy_cut + + +def _box(x, y, name, w=40, h=20): + return {"x": x, "y": y, "width": w, "height": h, "text": name} + + +def _two_columns(): + # column A at x[0,40], column B at x[100,140]; rows at y 0 and 30 + return [_box(0, 0, "A1"), _box(100, 0, "B1"), + _box(0, 30, "A2"), _box(100, 30, "B2")] + + +def test_column_aware_order_reads_down_columns(): + order = [b["text"] for b in flow_order(_two_columns(), min_gap=12)] + # column A fully, then column B — NOT the naive A1,B1,A2,B2 interleave + assert order == ["A1", "A2", "B1", "B2"] + + +def test_top_level_split_is_vertical(): + tree = xy_cut(_two_columns(), min_gap=12) + assert tree["type"] == "split" and tree["axis"] == "x" + + +def test_single_column_is_top_to_bottom(): + boxes = [_box(0, 60, "c"), _box(0, 0, "a"), _box(0, 30, "b")] + order = [b["text"] for b in flow_order(boxes, min_gap=12)] + assert order == ["a", "b", "c"] + + +def test_to_blocks_counts_leaves(): + blocks = to_blocks(xy_cut(_two_columns(), min_gap=12)) + assert len(blocks) == 2 # one leaf per column + assert [b["text"] for b in blocks[0]] == ["A1", "A2"] + + +def test_index_is_assigned(): + ordered = flow_order(_two_columns(), min_gap=12) + assert [b["index"] for b in ordered] == [0, 1, 2, 3] + + +def test_empty(): + assert flow_order([]) == [] + assert xy_cut([])["boxes"] == [] + + +# --- wiring --------------------------------------------------------------- + +def test_wiring(): + known = set(ac.executor.known_commands()) + assert {"AC_flow_order", "AC_xy_cut"} <= known + from je_auto_control.utils.mcp_server.tools import build_default_tool_registry + names = {t.name for t in build_default_tool_registry()} + assert {"ac_flow_order", "ac_xy_cut"} <= names + from je_auto_control.gui.script_builder.command_schema import _build_specs + specs = {s.command for s in _build_specs()} + assert {"AC_flow_order", "AC_xy_cut"} <= specs + + +def test_facade_exports(): + for name in ("flow_order", "xy_cut", "to_blocks"): + assert hasattr(ac, name) and name in ac.__all__ diff --git a/test/unit_test/headless/test_subpixel_match_batch.py b/test/unit_test/headless/test_subpixel_match_batch.py new file mode 100644 index 00000000..9ce3da2e --- /dev/null +++ b/test/unit_test/headless/test_subpixel_match_batch.py @@ -0,0 +1,69 @@ +"""Headless tests for sub-pixel template-match refinement.""" +import pytest + +import je_auto_control as ac + +np = pytest.importorskip("numpy") +pytest.importorskip("cv2") + +from je_auto_control.utils.subpixel_match import ( # noqa: E402 + match_subpixel, refine_peak, +) + + +def _template(): + tmpl = np.zeros((24, 24), dtype=np.uint8) + tmpl[:, :12] = 200 + return tmpl + + +def _haystack(top, left): + hay = np.zeros((120, 160), dtype=np.uint8) + hay[top:top + 24, left:left + 24] = _template() + return hay + + +def test_refine_peak_symmetric_is_zero(): + smap = np.array([[0.0, 0.0, 0.0], + [0.0, 1.0, 0.0], + [0.0, 0.0, 0.0]], dtype=np.float32) + offset_x, offset_y = refine_peak(smap, (1, 1)) + assert abs(offset_x) < 1e-9 and abs(offset_y) < 1e-9 + + +def test_refine_peak_biased_offset_sign(): + # right neighbour higher than left → peak nudged toward +x + row = np.array([[0.2, 1.0, 0.6]], dtype=np.float32) + offset_x, offset_y = refine_peak(row, (1, 0)) + assert offset_x > 0.0 and abs(offset_y) < 1e-9 + + +def test_match_subpixel_locates_and_centres(): + match = match_subpixel(_template(), haystack=_haystack(20, 30), min_score=0.8) + assert match is not None + assert match.x == 30 and match.y == 20 + # exact alignment → offset ~0, cx ≈ integer centre + assert abs(match.cx - (30 + 12)) < 0.6 + assert abs(match.cy - (20 + 12)) < 0.6 + + +def test_match_subpixel_min_score_filters(): + blank = np.zeros((120, 160), dtype=np.uint8) + assert match_subpixel(_template(), haystack=blank, min_score=0.95) is None + + +# --- wiring --------------------------------------------------------------- + +def test_wiring(): + assert "AC_match_subpixel" in set(ac.executor.known_commands()) + from je_auto_control.utils.mcp_server.tools import build_default_tool_registry + names = {t.name for t in build_default_tool_registry()} + assert "ac_match_subpixel" in names + from je_auto_control.gui.script_builder.command_schema import _build_specs + specs = {s.command for s in _build_specs()} + assert "AC_match_subpixel" in specs + + +def test_facade_exports(): + for name in ("match_subpixel", "refine_peak", "SubPixelMatch"): + assert hasattr(ac, name) and name in ac.__all__ diff --git a/test/unit_test/headless/test_text_blocks_batch.py b/test/unit_test/headless/test_text_blocks_batch.py new file mode 100644 index 00000000..46d837b6 --- /dev/null +++ b/test/unit_test/headless/test_text_blocks_batch.py @@ -0,0 +1,59 @@ +"""Headless tests for paragraph / list grouping of OCR lines (pure stdlib).""" +import je_auto_control as ac +from je_auto_control.utils.text_blocks import detect_lists, group_paragraphs + + +def _line(y, text, x=0, w=200, h=20): + return {"x": x, "y": y, "width": w, "height": h, "text": text} + + +def test_group_paragraphs_splits_on_large_gap(): + # two tight lines, a big gap, then two more tight lines → 2 paragraphs + lines = [_line(0, "a1"), _line(25, "a2"), + _line(120, "b1"), _line(145, "b2")] + paras = group_paragraphs(lines, line_gap_factor=1.6) + assert len(paras) == 2 + assert paras[0]["text"] == "a1 a2" and paras[1]["text"] == "b1 b2" + assert paras[0]["n_lines"] == 2 + + +def test_group_paragraphs_single_block(): + lines = [_line(0, "x"), _line(25, "y"), _line(50, "z")] + paras = group_paragraphs(lines) + assert len(paras) == 1 and paras[0]["text"] == "x y z" + + +def test_detect_lists_bullets_and_ordinals(): + lines = [_line(0, "• first"), _line(30, "2) second"), + _line(60, "a. third"), _line(90, "not a list item")] + items = detect_lists(lines) + assert [i["text"] for i in items] == ["first", "second", "third"] + assert items[0]["marker"] == "•" and items[1]["marker"] == "2)" + + +def test_detect_lists_indent_recorded(): + items = detect_lists([_line(0, "- top", x=10), _line(30, "- nested", x=40)]) + assert items[0]["indent"] == 10 and items[1]["indent"] == 40 + + +def test_empty(): + assert group_paragraphs([]) == [] + assert detect_lists([]) == [] + + +# --- wiring --------------------------------------------------------------- + +def test_wiring(): + known = set(ac.executor.known_commands()) + assert {"AC_group_paragraphs", "AC_detect_lists"} <= known + from je_auto_control.utils.mcp_server.tools import build_default_tool_registry + names = {t.name for t in build_default_tool_registry()} + assert {"ac_group_paragraphs", "ac_detect_lists"} <= names + from je_auto_control.gui.script_builder.command_schema import _build_specs + specs = {s.command for s in _build_specs()} + assert {"AC_group_paragraphs", "AC_detect_lists"} <= specs + + +def test_facade_exports(): + for name in ("group_paragraphs", "detect_lists"): + assert hasattr(ac, name) and name in ac.__all__