Add reading_flow: column-aware reading order via XY-cut

JE-Chen · JE-Chen · commit dab3a01b95d8 · 2026-06-24T05:37:36.000+08:00
element_parse.reading_order is a flat top-to-bottom sort that interleaves
columns on multi-column pages (A1, B1, A2, B2). Recover the correct order with
recursive XY-cut: split at the widest whitespace valley (vertical -&gt; columns,
horizontal -&gt; rows), so a two-column page reads A1, A2, B1, B2. flow_order
returns the same index-tagged contract as reading_order (a drop-in column-aware
upgrade, named to not shadow it).
diff --git a/README/WHATS_NEW_zh-CN.md b/README/WHATS_NEW_zh-CN.md
@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-24) — 列感知阅读顺序(XY-Cut)
+
+读多列布局时*完整读完*每一列,而非交错。完整参考:[`docs/source/Zh/doc/new_features/v173_features_doc.rst`](../docs/source/Zh/doc/new_features/v173_features_doc.rst)。
+
+- **`flow_order` / `xy_cut` / `to_blocks`**(`AC_flow_order`、`AC_xy_cut`):`element_parse.reading_order` 是平面上到下排序,会交错列(读作 A1, B1, A2, B2…)。本功能以递归 XY-cut 还原正确顺序——在最宽留白谷切分(垂直 → 列、水平 → 行),故两列页面读作 A1, A2, B1, B2。`flow_order` 返回与 `reading_order` 相同的 `index` 标记契约(列感知的直接升级,且命名不遮蔽它);`xy_cut` 暴露区域树;`to_blocks` 列出叶区块。纯标准库;不导入 `PySide6`。
+
 ## 本次更新 (2026-06-24) — Grounding 自我一致性(提案共识)
 
 把多个 grounding 提案融合成单一一致目标,并附 agreement 分数。完整参考:[`docs/source/Zh/doc/new_features/v172_features_doc.rst`](../docs/source/Zh/doc/new_features/v172_features_doc.rst)。
diff --git a/README/WHATS_NEW_zh-TW.md b/README/WHATS_NEW_zh-TW.md
@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-24) — 欄位感知閱讀順序(XY-Cut)
+
+讀多欄版面時*完整讀完*每一欄,而非交錯。完整參考:[`docs/source/Zh/doc/new_features/v173_features_doc.rst`](../docs/source/Zh/doc/new_features/v173_features_doc.rst)。
+
+- **`flow_order` / `xy_cut` / `to_blocks`**(`AC_flow_order`、`AC_xy_cut`):`element_parse.reading_order` 是平面上到下排序,會交錯欄位(讀作 A1, B1, A2, B2…)。本功能以遞迴 XY-cut 還原正確順序——在最寬留白谷切分(垂直 → 欄、水平 → 列),故兩欄頁面讀作 A1, A2, B1, B2。`flow_order` 回傳與 `reading_order` 相同的 `index` 標記契約(欄位感知的直接升級,且命名不遮蔽它);`xy_cut` 暴露區域樹;`to_blocks` 列出葉區塊。純標準函式庫;不匯入 `PySide6`。
+
 ## 本次更新 (2026-06-24) — Grounding 自我一致性(提案共識)
 
 把多個 grounding 提案融合成單一一致目標,並附 agreement 分數。完整參考:[`docs/source/Zh/doc/new_features/v172_features_doc.rst`](../docs/source/Zh/doc/new_features/v172_features_doc.rst)。
diff --git a/WHATS_NEW.md b/WHATS_NEW.md
@@ -1,5 +1,11 @@
 # What's New — AutoControl
 
+## What's new (2026-06-24) — Column-Aware Reading Order (XY-Cut)
+
+Read multi-column layouts down each column instead of interleaving them. Full reference: [`docs/source/Eng/doc/new_features/v173_features_doc.rst`](docs/source/Eng/doc/new_features/v173_features_doc.rst).
+
+- **`flow_order` / `xy_cut` / `to_blocks`** (`AC_flow_order`, `AC_xy_cut`): `element_parse.reading_order` is a flat top-to-bottom sort that interleaves columns (reads A1, B1, A2, B2…). This recovers the correct order with recursive XY-cut — split at the widest whitespace valley (vertical → columns, horizontal → rows), so a two-column page reads A1, A2, B1, B2. `flow_order` returns the same `index`-tagged contract as `reading_order` (a drop-in column-aware upgrade, named to not shadow it); `xy_cut` exposes the region tree; `to_blocks` lists the leaf blocks. Pure-stdlib; no `PySide6`.
+
 ## What's new (2026-06-24) — Grounding Self-Consistency (Consensus Over Proposals)
 
 Fuse several grounding proposals into one agreed target with an agreement score. Full reference: [`docs/source/Eng/doc/new_features/v172_features_doc.rst`](docs/source/Eng/doc/new_features/v172_features_doc.rst).
diff --git a/docs/source/Eng/doc/new_features/v173_features_doc.rst b/docs/source/Eng/doc/new_features/v173_features_doc.rst
@@ -0,0 +1,41 @@
+Column-Aware Reading Order (XY-Cut)
+===================================
+
+``element_parse.reading_order`` is a flat top-to-bottom / left-to-right sort — it interleaves
+columns on any multi-column page (the well-documented naive-sort failure: it reads row 1 of
+column A, then row 1 of column B, then row 2 of A …). ``reading_flow`` recovers the correct
+order with recursive **XY-cut**: it repeatedly splits the boxes at the widest whitespace valley
+(a vertical gutter → columns, a horizontal gutter → rows / blocks), so a two-column layout is
+read *down* column A fully, then column B.
+
+The public flattener is named ``flow_order`` to sit *beside* — not shadow —
+``element_parse.reading_order``; it returns the same ``index``-tagged element contract, so it is
+a drop-in column-aware upgrade. Pure-stdlib geometry over plain box dicts (no image, no OCR
+engine); reuses ``table_grid_fill``'s box-bounds reader. Imports no ``PySide6``.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import flow_order, xy_cut, to_blocks
+
+    # two columns, two rows each: reads A1, A2, B1, B2 — not A1, B1, A2, B2
+    for element in flow_order(ocr_boxes, min_gap=12):
+        print(element["index"], element["text"])
+
+    tree = xy_cut(ocr_boxes, min_gap=12)     # {type, axis, children|boxes}
+    blocks = to_blocks(tree)                  # leaf blocks in reading order
+
+``flow_order`` returns the boxes in column-aware reading order, each tagged with an ``index``.
+``xy_cut`` returns the recursive region tree (each node is a ``split`` on ``axis`` ``"x"`` /
+``"y"`` or a ``leaf`` of boxes). ``to_blocks`` flattens the tree to its leaf blocks in order.
+``min_gap`` is the smallest whitespace valley treated as a column / row break.
+
+Executor commands
+-----------------
+
+``AC_flow_order`` (``boxes`` / ``min_gap`` → ``{count, elements}``) and ``AC_xy_cut``
+(``boxes`` / ``min_gap`` → ``{tree}``). They are exposed as the MCP tools ``ac_flow_order`` /
+``ac_xy_cut`` (read-only) and as the Script Builder commands **Reading Order (column-aware)** /
+**XY-Cut Region Tree** under **OCR**.
diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst
@@ -195,6 +195,7 @@ Comprehensive guides for all AutoControl features.
    doc/new_features/v170_features_doc
    doc/new_features/v171_features_doc
    doc/new_features/v172_features_doc
+   doc/new_features/v173_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc
diff --git a/docs/source/Zh/doc/new_features/v173_features_doc.rst b/docs/source/Zh/doc/new_features/v173_features_doc.rst
@@ -0,0 +1,37 @@
+欄位感知閱讀順序(XY-Cut)
+==========================
+
+``element_parse.reading_order`` 是平面的上到下 / 左到右排序——在任何多欄頁面都會交錯欄位
+(眾所周知的 naive 排序失敗:它先讀 A 欄第 1 列、再讀 B 欄第 1 列、再讀 A 欄第 2 列…)。
+``reading_flow`` 以遞迴 **XY-cut** 還原正確順序:在最寬的留白谷反覆切分(垂直 gutter → 欄、
+水平 gutter → 列 / 區塊),因此兩欄版面會*完整讀完*A 欄,再讀 B 欄。
+
+公開的展平函式命名為 ``flow_order``,以*並列*而非遮蔽 ``element_parse.reading_order``;它回傳
+相同的 ``index`` 標記元素契約,因此是欄位感知的直接升級。純標準函式庫幾何,作用於純框字典
+(不需影像、不需 OCR 引擎);重用 ``table_grid_fill`` 的框邊界讀取器。不匯入 ``PySide6``。
+
+無頭 API
+--------
+
+.. code-block:: python
+
+    from je_auto_control import flow_order, xy_cut, to_blocks
+
+    # 兩欄各兩列:讀作 A1, A2, B1, B2——而非 A1, B1, A2, B2
+    for element in flow_order(ocr_boxes, min_gap=12):
+        print(element["index"], element["text"])
+
+    tree = xy_cut(ocr_boxes, min_gap=12)     # {type, axis, children|boxes}
+    blocks = to_blocks(tree)                  # 依閱讀順序的葉區塊
+
+``flow_order`` 以欄位感知閱讀順序回傳各框,每個帶 ``index``。``xy_cut`` 回傳遞迴區域樹
+(每個節點為對 ``axis`` ``"x"`` / ``"y"`` 的 ``split`` 或框的 ``leaf``)。``to_blocks`` 把樹
+展平為依序的葉區塊。``min_gap`` 為被視為欄 / 列分隔的最小留白谷。
+
+執行器指令
+----------
+
+``AC_flow_order``(``boxes`` / ``min_gap`` → ``{count, elements}``)與 ``AC_xy_cut``
+(``boxes`` / ``min_gap`` → ``{tree}``)。兩者以 MCP 工具 ``ac_flow_order`` / ``ac_xy_cut``
+(唯讀)及 Script Builder 指令 **Reading Order (column-aware)** / **XY-Cut Region Tree**
+(位於 **OCR** 分類下)形式提供。
diff --git a/docs/source/Zh/zh_index.rst b/docs/source/Zh/zh_index.rst
@@ -195,6 +195,7 @@ AutoControl 所有功能的完整使用指南。
    doc/new_features/v170_features_doc
    doc/new_features/v171_features_doc
    doc/new_features/v172_features_doc
+   doc/new_features/v173_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc
diff --git a/je_auto_control/__init__.py b/je_auto_control/__init__.py
@@ -315,6 +315,10 @@
 from je_auto_control.utils.column_layout import (
     assign_columns, column_gutters, detect_borderless_table, vertical_projection,
 )
+# Column-aware reading order via recursive XY-cut
+from je_auto_control.utils.reading_flow import (
+    flow_order, to_blocks, xy_cut,
+)
 # Associate form labels with values (multi-direction) + checkbox state
 from je_auto_control.utils.form_fields import (
     associate_fields, checkbox_state, match_labels_to_widgets,
@@ -1272,6 +1276,9 @@ def start_autocontrol_gui(*args, **kwargs):
     "column_gutters",
     "assign_columns",
     "detect_borderless_table",
+    "xy_cut",
+    "flow_order",
+    "to_blocks",
     "associate_fields",
     "match_labels_to_widgets",
     "checkbox_state",
diff --git a/je_auto_control/gui/script_builder/command_schema.py b/je_auto_control/gui/script_builder/command_schema.py
@@ -788,6 +788,25 @@ def _add_ocr_specs(specs: List[CommandSpec]) -> None:
         ),
         description="Match each checkbox/radio/input to its nearest label by centre.",
     ))
+    specs.append(CommandSpec(
+        "AC_flow_order", "OCR", "Reading Order (column-aware)",
+        fields=(
+            FieldSpec("boxes", FieldType.STRING,
+                      placeholder='[{"x":0,"y":0,"width":40,"height":20,'
+                                  '"text":"A1"}]'),
+            FieldSpec("min_gap", FieldType.INT, optional=True, default=12),
+        ),
+        description="Column-aware reading order via XY-cut (reads down columns).",
+    ))
+    specs.append(CommandSpec(
+        "AC_xy_cut", "OCR", "XY-Cut Region Tree",
+        fields=(
+            FieldSpec("boxes", FieldType.STRING,
+                      placeholder='[{"x":0,"y":0,"width":40,"height":20}]'),
+            FieldSpec("min_gap", FieldType.INT, optional=True, default=12),
+        ),
+        description="Recursive XY-cut region tree (split at widest whitespace).",
+    ))
     specs.append(CommandSpec(
         "AC_scroll_to_find", "OCR", "Scroll Until Visible",
         fields=(
diff --git a/je_auto_control/utils/executor/action_executor.py b/je_auto_control/utils/executor/action_executor.py
@@ -3498,6 +3498,25 @@ def _match_labels_to_widgets(labels: Any, widgets: Any) -> Dict[str, Any]:
     return {"count": len(pairs), "pairs": pairs}
 
 
+def _flow_order(boxes: Any, min_gap: Any = 12) -> Dict[str, Any]:
+    """Adapter: column-aware reading order of OCR boxes (XY-cut)."""
+    import json
+    from je_auto_control.utils.reading_flow import flow_order
+    if isinstance(boxes, str):
+        boxes = json.loads(boxes)
+    ordered = flow_order(boxes, min_gap=int(min_gap))
+    return {"count": len(ordered), "elements": ordered}
+
+
+def _xy_cut(boxes: Any, min_gap: Any = 12) -> Dict[str, Any]:
+    """Adapter: recursive XY-cut region tree of OCR boxes."""
+    import json
+    from je_auto_control.utils.reading_flow import xy_cut
+    if isinstance(boxes, str):
+        boxes = json.loads(boxes)
+    return {"tree": xy_cut(boxes, min_gap=int(min_gap))}
+
+
 def _find_color_region(rgb: Any, tolerance: Any = 20, min_area: Any = 50,
                        region: Any = None) -> Dict[str, Any]:
     """Adapter: locate coloured regions on the screen, largest first."""
@@ -6033,6 +6052,8 @@ def __init__(self):
             "AC_detect_borderless_table": _detect_borderless_table,
             "AC_associate_fields": _associate_fields,
             "AC_match_labels_to_widgets": _match_labels_to_widgets,
+            "AC_flow_order": _flow_order,
+            "AC_xy_cut": _xy_cut,
             "AC_ssim_compare": _ssim_compare,
             "AC_ssim_changed_regions": _ssim_changed_regions,
             "AC_feature_match": _feature_match,
diff --git a/je_auto_control/utils/mcp_server/tools/_factories.py b/je_auto_control/utils/mcp_server/tools/_factories.py
@@ -3924,6 +3924,32 @@ def screen_grid_tools() -> List[MCPTool]:
             handler=h.match_labels_to_widgets,
             annotations=READ_ONLY,
         ),
+        MCPTool(
+            name="ac_flow_order",
+            description=("Column-aware reading order of OCR 'boxes' via XY-cut — reads "
+                         "DOWN each column before the next (vs the naive sort that "
+                         "interleaves columns). Returns {count, elements} each tagged "
+                         "with an 'index'. 'min_gap' is the smallest whitespace valley "
+                         "treated as a column / row break."),
+            input_schema=schema({
+                "boxes": {"type": "array", "items": {"type": "object"}},
+                "min_gap": {"type": "integer"}},
+                required=["boxes"]),
+            handler=h.flow_order,
+            annotations=READ_ONLY,
+        ),
+        MCPTool(
+            name="ac_xy_cut",
+            description=("Recursive XY-cut region tree of OCR 'boxes' (splits at the "
+                         "widest whitespace valley). Returns {tree:{type,axis,"
+                         "children|boxes}}."),
+            input_schema=schema({
+                "boxes": {"type": "array", "items": {"type": "object"}},
+                "min_gap": {"type": "integer"}},
+                required=["boxes"]),
+            handler=h.xy_cut,
+            annotations=READ_ONLY,
+        ),
     ]
 
 
diff --git a/je_auto_control/utils/mcp_server/tools/_handlers.py b/je_auto_control/utils/mcp_server/tools/_handlers.py
@@ -2185,6 +2185,16 @@ def match_labels_to_widgets(labels, widgets):
     return _match_labels_to_widgets(labels, widgets)
 
 
+def flow_order(boxes, min_gap=12):
+    from je_auto_control.utils.executor.action_executor import _flow_order
+    return _flow_order(boxes, min_gap)
+
+
+def xy_cut(boxes, min_gap=12):
+    from je_auto_control.utils.executor.action_executor import _xy_cut
+    return _xy_cut(boxes, min_gap)
+
+
 def find_color_region(rgb, tolerance=20, min_area=50, region=None):
     from je_auto_control.utils.executor.action_executor import (
         _find_color_region)
diff --git a/je_auto_control/utils/reading_flow/__init__.py b/je_auto_control/utils/reading_flow/__init__.py
@@ -0,0 +1,6 @@
+"""Column-aware reading order via recursive XY-cut."""
+from je_auto_control.utils.reading_flow.reading_flow import (
+    flow_order, to_blocks, xy_cut,
+)
+
+__all__ = ["xy_cut", "flow_order", "to_blocks"]
diff --git a/je_auto_control/utils/reading_flow/reading_flow.py b/je_auto_control/utils/reading_flow/reading_flow.py
@@ -0,0 +1,113 @@
+"""Column-aware reading order via recursive XY-cut.
+
+``element_parse.reading_order`` is a flat top-to-bottom / left-to-right sort — it interleaves
+columns on any multi-column page (the well-documented naive-sort failure: it reads row 1 of
+column A, then row 1 of column B, then row 2 of A …). ``reading_flow`` recovers the correct
+order with recursive XY-cut: it repeatedly splits the boxes at the widest whitespace valley
+(vertical gutter → columns, horizontal gutter → rows/blocks), so a two-column layout is read
+*down* column A fully, then column B. It returns the region tree and a column-aware flattened
+order; the public flattener is named ``flow_order`` to sit beside (not shadow)
+``reading_order``.
+
+Pure-stdlib geometry over plain box dicts (no image, no OCR engine); reuses ``table_grid_fill``'s
+box-bounds reader. Imports no ``PySide6``.
+"""
+from typing import Any, Dict, List, Optional, Sequence, Tuple
+
+from je_auto_control.utils.table_grid_fill.table_grid_fill import _box_bounds
+
+Box = Dict[str, Any]
+
+
+def _axis_bounds(box: Box, axis: str) -> Tuple[int, int]:
+    left, top, right, bottom = _box_bounds(box)
+    return (left, right) if axis == "x" else (top, bottom)
+
+
+def _center_on(box: Box, axis: str) -> float:
+    low, high = _axis_bounds(box, axis)
+    return (low + high) / 2.0
+
+
+def _widest_gap(boxes: Sequence[Box], axis: str, min_gap: int):
+    """Return ``(position, width)`` of the widest whitespace gap on ``axis``, or ``None``."""
+    spans = sorted(_axis_bounds(box, axis) for box in boxes)
+    best_pos: Optional[float] = None
+    best_gap = 0.0
+    cur_end = spans[0][1]
+    for low, high in spans[1:]:
+        gap = low - cur_end
+        if gap > best_gap:
+            best_gap, best_pos = gap, cur_end + gap / 2.0
+        cur_end = max(cur_end, high)
+    if best_pos is not None and best_gap >= min_gap:
+        return best_pos, best_gap
+    return None
+
+
+def _choose_axis(boxes: Sequence[Box], min_gap: int):
+    """Return ``(axis, position)`` of the widest gap across both axes, or ``None``."""
+    candidates = []
+    for axis in ("x", "y"):
+        found = _widest_gap(boxes, axis, min_gap)
+        if found is not None:
+            candidates.append((found[1], axis, found[0]))
+    if not candidates:
+        return None
+    _, axis, position = max(candidates)
+    return axis, position
+
+
+def _cut(boxes: List[Box], min_gap: int, depth: int) -> Dict[str, Any]:
+    """Recursively XY-cut ``boxes`` into a region tree."""
+    if len(boxes) <= 1 or depth <= 0:
+        return {"type": "leaf", "boxes": list(boxes)}
+    chosen = _choose_axis(boxes, min_gap)
+    if chosen is None:
+        return {"type": "leaf", "boxes": list(boxes)}
+    axis, position = chosen
+    first = [box for box in boxes if _center_on(box, axis) < position]
+    second = [box for box in boxes if _center_on(box, axis) >= position]
+    if not first or not second:
+        return {"type": "leaf", "boxes": list(boxes)}
+    return {"type": "split", "axis": axis,
+            "children": [_cut(first, min_gap, depth - 1),
+                         _cut(second, min_gap, depth - 1)]}
+
+
+def xy_cut(boxes: Sequence[Box], *, min_gap: int = 12,
+           max_depth: int = 8) -> Dict[str, Any]:
+    """Return the recursive XY-cut region tree of ``boxes``."""
+    if not boxes:
+        return {"type": "leaf", "boxes": []}
+    return _cut(list(boxes), int(min_gap), int(max_depth))
+
+
+def _flatten(tree: Dict[str, Any]) -> List[Box]:
+    if tree["type"] == "leaf":
+        return sorted(tree["boxes"],
+                      key=lambda b: (_box_bounds(b)[1], _box_bounds(b)[0]))
+    flat: List[Box] = []
+    for child in tree["children"]:
+        flat.extend(_flatten(child))
+    return flat
+
+
+def flow_order(boxes: Sequence[Box], *, min_gap: int = 12,
+               max_depth: int = 8) -> List[Box]:
+    """Return ``boxes`` in column-aware reading order, each tagged with an ``index``."""
+    flat = _flatten(xy_cut(boxes, min_gap=min_gap, max_depth=max_depth))
+    return [dict(box, index=i) for i, box in enumerate(flat)]
+
+
+def to_blocks(tree: Dict[str, Any]) -> List[List[Box]]:
+    """Return the leaf blocks of an XY-cut ``tree`` in reading order."""
+    if tree["type"] == "leaf":
+        if not tree["boxes"]:
+            return []
+        return [sorted(tree["boxes"],
+                       key=lambda b: (_box_bounds(b)[1], _box_bounds(b)[0]))]
+    blocks: List[List[Box]] = []
+    for child in tree["children"]:
+        blocks.extend(to_blocks(child))
+    return blocks
diff --git a/test/unit_test/headless/test_reading_flow_batch.py b/test/unit_test/headless/test_reading_flow_batch.py
