Merge pull request #368 from Integration-Automation/feat/barcode-batch

JE-Chen · web-flow · commit f453138b960c · 2026-06-23T22:08:43.000+08:00
Add 1-D barcode decoding (read_barcodes)
diff --git a/README/WHATS_NEW_zh-CN.md b/README/WHATS_NEW_zh-CN.md
@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-23) — 一维条码解码
+
+从屏幕或图像读取 EAN / UPC / Code-128 条码。完整参考:[`docs/source/Zh/doc/new_features/v157_features_doc.rst`](../docs/source/Zh/doc/new_features/v157_features_doc.rst)。
+
+- **`read_barcodes`**(`AC_read_barcodes`):框架已能解码 QR Code(`read_qr`),但缺少能读取*一维*条码(EAN-13/8、UPC-A、Code-128)的功能——这些正是商品、库存标签与物流面单上最常见的条码。本功能通过 OpenCV 的 `cv2.barcode.BarcodeDetector` 解码,每个条码返回 `{text, type, points}`。解码步骤为可注入接缝(默认调用 OpenCV;测试可传入自己的 `decoder`),因此可完整无头测试且能优雅降级——若 OpenCV 编译时未含 `barcode` 模块,返回 `[]` 而非抛出异常。重用共用的 `visual_match` haystack 加载器;不导入 `PySide6`。
+
 ## 本次更新 (2026-06-23) — 加权候选评分
 
 以信心分数排序模棱两可的元素候选。完整参考:[`docs/source/Zh/doc/new_features/v156_features_doc.rst`](../docs/source/Zh/doc/new_features/v156_features_doc.rst)。
diff --git a/README/WHATS_NEW_zh-TW.md b/README/WHATS_NEW_zh-TW.md
@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-23) — 一維條碼解碼
+
+從螢幕或影像讀取 EAN / UPC / Code-128 條碼。完整參考:[`docs/source/Zh/doc/new_features/v157_features_doc.rst`](../docs/source/Zh/doc/new_features/v157_features_doc.rst)。
+
+- **`read_barcodes`**(`AC_read_barcodes`):框架已能解碼 QR Code(`read_qr`),但缺少能讀取*一維*條碼(EAN-13/8、UPC-A、Code-128)的功能——這些正是商品、庫存標籤與物流面單上最常見的條碼。本功能透過 OpenCV 的 `cv2.barcode.BarcodeDetector` 解碼,每個條碼回傳 `{text, type, points}`。解碼步驟為可注入接縫(預設呼叫 OpenCV;測試可傳入自己的 `decoder`),因此可完整無頭測試且能優雅降級——若 OpenCV 編譯時未含 `barcode` 模組,回傳 `[]` 而非拋出例外。重用共用的 `visual_match` haystack 載入器;不匯入 `PySide6`。
+
 ## 本次更新 (2026-06-23) — 加權候選評分
 
 以信心分數排序模稜兩可的元素候選。完整參考:[`docs/source/Zh/doc/new_features/v156_features_doc.rst`](../docs/source/Zh/doc/new_features/v156_features_doc.rst)。
diff --git a/WHATS_NEW.md b/WHATS_NEW.md
@@ -1,5 +1,11 @@
 # What's New — AutoControl
 
+## What's new (2026-06-23) — Barcode Decoding (1-D)
+
+Read EAN / UPC / Code-128 barcodes off the screen or an image. Full reference: [`docs/source/Eng/doc/new_features/v157_features_doc.rst`](docs/source/Eng/doc/new_features/v157_features_doc.rst).
+
+- **`read_barcodes`** (`AC_read_barcodes`): the framework decoded QR codes (`read_qr`) but had no reader for the *1-D* barcodes (EAN-13/8, UPC-A, Code-128) that label physical goods, inventory tickets and shipping labels. This decodes them via OpenCV's `cv2.barcode.BarcodeDetector`, returning `{text, type, points}` per code. The decode step is an injectable seam (default calls OpenCV; tests pass their own `decoder`), so it's fully headless-testable and degrades gracefully — an OpenCV build without the `barcode` module returns `[]` instead of raising. Reuses the shared `visual_match` haystack loader; no `PySide6`.
+
 ## What's new (2026-06-23) — Weighted Candidate Scoring
 
 Rank ambiguous element candidates by a confidence score. Full reference: [`docs/source/Eng/doc/new_features/v156_features_doc.rst`](docs/source/Eng/doc/new_features/v156_features_doc.rst).
diff --git a/docs/source/Eng/doc/new_features/v157_features_doc.rst b/docs/source/Eng/doc/new_features/v157_features_doc.rst
@@ -0,0 +1,43 @@
+Barcode Decoding (1-D)
+======================
+
+The framework already decodes QR codes (``read_qr``), but had no reader for the
+*1-D* barcodes (EAN-13 / EAN-8 / UPC-A / Code-128) that label physical goods,
+inventory tickets and shipping labels — the most common thing a desktop or kiosk
+automation needs to read off a product screen. ``read_barcodes`` fills that gap
+using OpenCV's ``cv2.barcode.BarcodeDetector``.
+
+The decode step is an **injectable seam**: the default decoder calls OpenCV, but
+tests (and alternative engines) can pass their own ``decoder`` callable, so the
+feature is fully unit-testable headlessly and degrades gracefully — a build of
+OpenCV without the ``barcode`` module simply returns an empty list instead of
+raising. Imports no ``PySide6``.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import read_barcodes
+
+    # decode every 1-D barcode currently on screen
+    for code in read_barcodes():
+        print(code["type"], code["text"], code["points"])
+
+    # restrict to a region, or decode a saved image instead of the screen
+    read_barcodes(region=[0, 0, 400, 200])
+    read_barcodes("label.png")
+
+``read_barcodes(source=None, *, region=None, decoder=None)`` returns a list of
+``{"text", "type", "points"}`` dicts, one per detected barcode (``points`` is the
+four-corner polygon in image coordinates). ``source`` may be an image path or an
+array; when omitted the screen (optionally cropped to ``region``) is grabbed. The
+grayscale conversion reuses the shared ``visual_match`` haystack loader, so no new
+image-loading code is added.
+
+Executor command
+----------------
+
+``AC_read_barcodes`` (``source`` / ``region`` → ``{count, barcodes}``) is exposed
+as the MCP tool ``ac_read_barcodes`` (read-only) and as a Script Builder command
+**Read Barcodes (1-D)** under **OCR**.
diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst
@@ -179,6 +179,7 @@ Comprehensive guides for all AutoControl features.
    doc/new_features/v154_features_doc
    doc/new_features/v155_features_doc
    doc/new_features/v156_features_doc
+   doc/new_features/v157_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc
diff --git a/docs/source/Zh/doc/new_features/v157_features_doc.rst b/docs/source/Zh/doc/new_features/v157_features_doc.rst
@@ -0,0 +1,40 @@
+一維條碼解碼
+============
+
+框架已能解碼 QR Code（``read_qr``），但缺少能讀取 *一維* 條碼（EAN-13 / EAN-8 /
+UPC-A / Code-128）的功能——這些正是商品、庫存標籤與物流面單上最常見的條碼，也是
+桌面或自助機自動化最需要從商品畫面讀取的資訊。``read_barcodes`` 透過 OpenCV 的
+``cv2.barcode.BarcodeDetector`` 補上這一塊。
+
+解碼步驟是一個**可注入接縫**：預設解碼器呼叫 OpenCV，但測試（或其他引擎）可以傳入
+自己的 ``decoder`` 可呼叫物件，因此此功能可在無頭環境下完整單元測試，且能優雅降級
+——若 OpenCV 編譯時未含 ``barcode`` 模組，僅回傳空清單而非拋出例外。不匯入
+``PySide6``。
+
+無頭 API
+--------
+
+.. code-block:: python
+
+    from je_auto_control import read_barcodes
+
+    # 解碼螢幕上目前所有一維條碼
+    for code in read_barcodes():
+        print(code["type"], code["text"], code["points"])
+
+    # 限定區域，或改為解碼已存檔的影像
+    read_barcodes(region=[0, 0, 400, 200])
+    read_barcodes("label.png")
+
+``read_barcodes(source=None, *, region=None, decoder=None)`` 回傳
+``{"text", "type", "points"}`` 字典清單，每偵測到一個條碼一筆（``points`` 為影像
+座標中的四角多邊形）。``source`` 可為影像路徑或陣列；省略時擷取螢幕（可選擇以
+``region`` 裁切）。灰階轉換重用共用的 ``visual_match`` haystack 載入器，不新增
+影像載入程式碼。
+
+執行器指令
+----------
+
+``AC_read_barcodes``（``source`` / ``region`` → ``{count, barcodes}``）以 MCP 工具
+``ac_read_barcodes``（唯讀）及 Script Builder 指令 **Read Barcodes (1-D)**（位於
+**OCR** 分類下）形式提供。
diff --git a/docs/source/Zh/zh_index.rst b/docs/source/Zh/zh_index.rst
@@ -179,6 +179,7 @@ AutoControl 所有功能的完整使用指南。
    doc/new_features/v154_features_doc
    doc/new_features/v155_features_doc
    doc/new_features/v156_features_doc
+   doc/new_features/v157_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc
diff --git a/je_auto_control/__init__.py b/je_auto_control/__init__.py
@@ -385,6 +385,8 @@
 from je_auto_control.utils.element_scoring import (
     ScoredCandidate, best_candidate, score_candidates,
 )
+# 1-D barcode decoding (EAN / UPC) with an injectable decoder seam
+from je_auto_control.utils.barcode import read_barcodes
 # CI workflow annotations (GitHub Actions)
 from je_auto_control.utils.ci_annotations import (
     emit_annotations, format_annotation,
@@ -1278,6 +1280,7 @@ def start_autocontrol_gui(*args, **kwargs):
     "score_candidates",
     "best_candidate",
     "ScoredCandidate",
+    "read_barcodes",
     "emit_annotations", "format_annotation",
     "ClipboardHistory", "default_clipboard_history",
     "analyze_heal_log", "heal_stats", "scan_secrets",
diff --git a/je_auto_control/gui/script_builder/command_schema.py b/je_auto_control/gui/script_builder/command_schema.py
@@ -586,6 +586,15 @@ def _add_ocr_specs(specs: List[CommandSpec]) -> None:
         ),
         description="Decode QR codes in a screen region (OpenCV).",
     ))
+    specs.append(CommandSpec(
+        "AC_read_barcodes", "OCR", "Read Barcodes (1-D)",
+        fields=(
+            FieldSpec("source", FieldType.FILE_PATH, optional=True),
+            FieldSpec("region", FieldType.STRING, optional=True,
+                      placeholder=_REGION_PLACEHOLDER),
+        ),
+        description="Decode 1-D barcodes (EAN / UPC) in an image / screen region.",
+    ))
     specs.append(CommandSpec(
         "AC_scroll_to_find", "OCR", "Scroll Until Visible",
         fields=(
diff --git a/je_auto_control/utils/barcode/__init__.py b/je_auto_control/utils/barcode/__init__.py
@@ -0,0 +1,4 @@
+"""1-D barcode decoding (EAN / UPC) with an injectable decoder seam."""
+from je_auto_control.utils.barcode.barcode import read_barcodes
+
+__all__ = ["read_barcodes"]
diff --git a/je_auto_control/utils/barcode/barcode.py b/je_auto_control/utils/barcode/barcode.py
@@ -0,0 +1,49 @@
+"""1-D barcode decoding — EAN / UPC, with an injectable decoder seam.
+
+The ``qr`` module decodes QR codes only (``cv2.QRCodeDetector``); there is no 1-D /
+linear barcode (EAN-8/13, UPC-A/E, Code-128) decode. This mirrors ``qr``'s injectable-
+decoder pattern so it is testable without a real barcode and future-proof against
+backend availability: the default decoder uses ``cv2.barcode.BarcodeDetector`` (base
+OpenCV since 4.8) and degrades to an empty result when that module is absent.
+
+Runs on an injectable image (ndarray / path / PIL, default: grab the screen / region),
+so it is headless-testable on synthetic arrays with an injected decoder. OpenCV +
+NumPy come in via ``je_open_cv``. Imports no ``PySide6``.
+"""
+from typing import Any, Callable, Dict, List, Optional, Sequence
+
+from je_auto_control.utils.visual_match.visual_match import _haystack_gray
+
+ImageSource = Any
+Decoder = Callable[[Any], List[Dict[str, Any]]]
+
+
+def _default_decoder(image) -> List[Dict[str, Any]]:
+    """Decode 1-D barcodes with ``cv2.barcode`` (empty if the module is absent)."""
+    import cv2
+    if not hasattr(cv2, "barcode"):
+        return []
+    retval, infos, types, points = cv2.barcode.BarcodeDetector(
+    ).detectAndDecodeWithType(image)
+    if not retval:
+        return []
+    results: List[Dict[str, Any]] = []
+    for text, kind, corners in zip(infos, types, points):
+        if not text:
+            continue
+        results.append({"text": text, "type": str(kind),
+                        "points": [[int(x), int(y)] for x, y in corners]})
+    return results
+
+
+def read_barcodes(source: Optional[ImageSource] = None, *,
+                  region: Optional[Sequence[int]] = None,
+                  decoder: Optional[Decoder] = None) -> List[Dict[str, Any]]:
+    """Return the 1-D barcodes found in ``source`` (or the screen / ``region``).
+
+    Each result is ``{text, type, points}``. ``decoder`` is injectable (it receives the
+    loaded image and returns the result list); the default uses ``cv2.barcode`` and
+    returns ``[]`` when that backend is unavailable.
+    """
+    image = _haystack_gray(source, region)
+    return (decoder or _default_decoder)(image)
diff --git a/je_auto_control/utils/executor/action_executor.py b/je_auto_control/utils/executor/action_executor.py
@@ -3953,6 +3953,16 @@ def _best_candidate(candidates: Any, want_role: Any = None, want_name: Any = Non
             "best": best.to_dict() if best is not None else None}
 
 
+def _read_barcodes(source: Any = None, region: Any = None) -> Dict[str, Any]:
+    """Adapter: decode 1-D barcodes on screen / in an image."""
+    import json
+    from je_auto_control.utils.barcode import read_barcodes
+    if isinstance(region, str):
+        region = json.loads(region) if region.strip() else None
+    barcodes = read_barcodes(source, region=region)
+    return {"count": len(barcodes), "barcodes": barcodes}
+
+
 def _with_modifiers(modifiers: Any, actions: Any) -> Dict[str, Any]:
     """Adapter: run nested actions while modifier keys are held down."""
     import json
@@ -5716,6 +5726,7 @@ def __init__(self):
             "AC_assign_stable_ids": _assign_stable_ids,
             "AC_score_candidates": _score_candidates,
             "AC_best_candidate": _best_candidate,
+            "AC_read_barcodes": _read_barcodes,
             "AC_tile_rect": _tile_rect,
             "AC_grid_rects": _grid_rects,
             "AC_cascade_rects": _cascade_rects,
diff --git a/je_auto_control/utils/mcp_server/tools/_factories.py b/je_auto_control/utils/mcp_server/tools/_factories.py
@@ -3410,6 +3410,24 @@ def element_scoring_tools() -> List[MCPTool]:
     ]
 
 
+def barcode_tools() -> List[MCPTool]:
+    return [
+        MCPTool(
+            name="ac_read_barcodes",
+            description=("Decode 1-D barcodes (EAN / UPC / Code-128) in 'source' "
+                         "(image path; default: screen grab of 'region'). Returns "
+                         "{count, barcodes:[{text, type, points}]}. QR codes have "
+                         "their own tool."),
+            input_schema=schema({
+                "source": {"type": "string"},
+                "region": {"type": "array", "items": {"type": "integer"}}},
+                required=[]),
+            handler=h.read_barcodes,
+            annotations=READ_ONLY,
+        ),
+    ]
+
+
 def ssim_tools() -> List[MCPTool]:
     return [
         MCPTool(
@@ -6919,7 +6937,8 @@ def media_assert_tools() -> List[MCPTool]:
     motion_regions_tools, window_zorder_tools, soft_assert_tools,
     perceptual_diff_tools, window_geometry_tools, cua_action_tools,
     observation_tools, action_grounding_tools, agent_replay_tools,
-    element_diff_tools, element_scoring_tools, plugin_sdk_tools, governance_tools,
+    element_diff_tools, element_scoring_tools, barcode_tools, plugin_sdk_tools,
+    governance_tools,
     credential_lease_tools, egress_tools, approval_testing_tools,
     trajectory_eval_tools, compliance_tools, agent_trace_tools,
     video_report_tools, fuzzy_tools, artifact_store_tools, image_dedup_tools,
diff --git a/je_auto_control/utils/mcp_server/tools/_handlers.py b/je_auto_control/utils/mcp_server/tools/_handlers.py
@@ -2350,6 +2350,11 @@ def best_candidate(candidates, want_role=None, want_name=None, anchor=None):
     return _best_candidate(candidates, want_role, want_name, anchor)
 
 
+def read_barcodes(source=None, region=None):
+    from je_auto_control.utils.executor.action_executor import _read_barcodes
+    return _read_barcodes(source, region)
+
+
 def detect_drift(reference, current, threshold=0.25, bins=10):
     from je_auto_control.utils.executor.action_executor import _detect_drift
     return _detect_drift(reference, current, threshold, bins)
diff --git a/test/unit_test/headless/test_barcode_batch.py b/test/unit_test/headless/test_barcode_batch.py
@@ -0,0 +1,52 @@
+"""Headless tests for 1-D barcode decoding. No Qt; decoder is injected."""
+import pytest
+
+import je_auto_control as ac
+
+np = pytest.importorskip("numpy")
+pytest.importorskip("cv2")
+
+from je_auto_control.utils.barcode import read_barcodes   # noqa: E402
+
+
+def _image():
+    return np.full((40, 120), 255, dtype=np.uint8)
+
+
+def test_injected_decoder_is_used():
+    rows = [{"text": "012345678905", "type": "EAN_13",
+             "points": [[0, 0], [100, 0], [100, 30], [0, 30]]}]
+    result = read_barcodes(_image(), decoder=lambda image: rows)
+    assert result == rows
+
+
+def test_decoder_receives_the_image():
+    seen = {}
+
+    def decoder(image):
+        seen["shape"] = getattr(image, "shape", None)
+        return []
+
+    read_barcodes(_image(), decoder=decoder)
+    assert seen["shape"] == (40, 120)
+
+
+def test_default_decoder_blank_image_is_empty():
+    # a blank image has no barcodes (graceful, regardless of cv2.barcode presence)
+    assert read_barcodes(_image()) == []
+
+
+# --- wiring ---------------------------------------------------------------
+
+def test_wiring():
+    assert "AC_read_barcodes" in set(ac.executor.known_commands())
+    from je_auto_control.utils.mcp_server.tools import build_default_tool_registry
+    names = {t.name for t in build_default_tool_registry()}
+    assert "ac_read_barcodes" in names
+    from je_auto_control.gui.script_builder.command_schema import _build_specs
+    specs = {s.command for s in _build_specs()}
+    assert "AC_read_barcodes" in specs
+
+
+def test_facade_exports():
+    assert hasattr(ac, "read_barcodes") and "read_barcodes" in ac.__all__
