diff --git a/README/WHATS_NEW_zh-CN.md b/README/WHATS_NEW_zh-CN.md index 0de9b107..1bd6e974 100644 --- a/README/WHATS_NEW_zh-CN.md +++ b/README/WHATS_NEW_zh-CN.md @@ -1,5 +1,11 @@ # 本次更新 — AutoControl +## 本次更新 (2026-06-23) — 一维条码解码 + +从屏幕或图像读取 EAN / UPC / Code-128 条码。完整参考:[`docs/source/Zh/doc/new_features/v157_features_doc.rst`](../docs/source/Zh/doc/new_features/v157_features_doc.rst)。 + +- **`read_barcodes`**(`AC_read_barcodes`):框架已能解码 QR Code(`read_qr`),但缺少能读取*一维*条码(EAN-13/8、UPC-A、Code-128)的功能——这些正是商品、库存标签与物流面单上最常见的条码。本功能通过 OpenCV 的 `cv2.barcode.BarcodeDetector` 解码,每个条码返回 `{text, type, points}`。解码步骤为可注入接缝(默认调用 OpenCV;测试可传入自己的 `decoder`),因此可完整无头测试且能优雅降级——若 OpenCV 编译时未含 `barcode` 模块,返回 `[]` 而非抛出异常。重用共用的 `visual_match` haystack 加载器;不导入 `PySide6`。 + ## 本次更新 (2026-06-23) — 加权候选评分 以信心分数排序模棱两可的元素候选。完整参考:[`docs/source/Zh/doc/new_features/v156_features_doc.rst`](../docs/source/Zh/doc/new_features/v156_features_doc.rst)。 diff --git a/README/WHATS_NEW_zh-TW.md b/README/WHATS_NEW_zh-TW.md index 3e420563..d87e16a3 100644 --- a/README/WHATS_NEW_zh-TW.md +++ b/README/WHATS_NEW_zh-TW.md @@ -1,5 +1,11 @@ # 本次更新 — AutoControl +## 本次更新 (2026-06-23) — 一維條碼解碼 + +從螢幕或影像讀取 EAN / UPC / Code-128 條碼。完整參考:[`docs/source/Zh/doc/new_features/v157_features_doc.rst`](../docs/source/Zh/doc/new_features/v157_features_doc.rst)。 + +- **`read_barcodes`**(`AC_read_barcodes`):框架已能解碼 QR Code(`read_qr`),但缺少能讀取*一維*條碼(EAN-13/8、UPC-A、Code-128)的功能——這些正是商品、庫存標籤與物流面單上最常見的條碼。本功能透過 OpenCV 的 `cv2.barcode.BarcodeDetector` 解碼,每個條碼回傳 `{text, type, points}`。解碼步驟為可注入接縫(預設呼叫 OpenCV;測試可傳入自己的 `decoder`),因此可完整無頭測試且能優雅降級——若 OpenCV 編譯時未含 `barcode` 模組,回傳 `[]` 而非拋出例外。重用共用的 `visual_match` haystack 載入器;不匯入 `PySide6`。 + ## 本次更新 (2026-06-23) — 加權候選評分 以信心分數排序模稜兩可的元素候選。完整參考:[`docs/source/Zh/doc/new_features/v156_features_doc.rst`](../docs/source/Zh/doc/new_features/v156_features_doc.rst)。 diff --git a/WHATS_NEW.md b/WHATS_NEW.md index f2916e42..f59ec8c1 100644 --- a/WHATS_NEW.md +++ b/WHATS_NEW.md @@ -1,5 +1,11 @@ # What's New — AutoControl +## What's new (2026-06-23) — Barcode Decoding (1-D) + +Read EAN / UPC / Code-128 barcodes off the screen or an image. Full reference: [`docs/source/Eng/doc/new_features/v157_features_doc.rst`](docs/source/Eng/doc/new_features/v157_features_doc.rst). + +- **`read_barcodes`** (`AC_read_barcodes`): the framework decoded QR codes (`read_qr`) but had no reader for the *1-D* barcodes (EAN-13/8, UPC-A, Code-128) that label physical goods, inventory tickets and shipping labels. This decodes them via OpenCV's `cv2.barcode.BarcodeDetector`, returning `{text, type, points}` per code. The decode step is an injectable seam (default calls OpenCV; tests pass their own `decoder`), so it's fully headless-testable and degrades gracefully — an OpenCV build without the `barcode` module returns `[]` instead of raising. Reuses the shared `visual_match` haystack loader; no `PySide6`. + ## What's new (2026-06-23) — Weighted Candidate Scoring Rank ambiguous element candidates by a confidence score. Full reference: [`docs/source/Eng/doc/new_features/v156_features_doc.rst`](docs/source/Eng/doc/new_features/v156_features_doc.rst). diff --git a/docs/source/Eng/doc/new_features/v157_features_doc.rst b/docs/source/Eng/doc/new_features/v157_features_doc.rst new file mode 100644 index 00000000..13e25a7f --- /dev/null +++ b/docs/source/Eng/doc/new_features/v157_features_doc.rst @@ -0,0 +1,43 @@ +Barcode Decoding (1-D) +====================== + +The framework already decodes QR codes (``read_qr``), but had no reader for the +*1-D* barcodes (EAN-13 / EAN-8 / UPC-A / Code-128) that label physical goods, +inventory tickets and shipping labels — the most common thing a desktop or kiosk +automation needs to read off a product screen. ``read_barcodes`` fills that gap +using OpenCV's ``cv2.barcode.BarcodeDetector``. + +The decode step is an **injectable seam**: the default decoder calls OpenCV, but +tests (and alternative engines) can pass their own ``decoder`` callable, so the +feature is fully unit-testable headlessly and degrades gracefully — a build of +OpenCV without the ``barcode`` module simply returns an empty list instead of +raising. Imports no ``PySide6``. + +Headless API +------------ + +.. code-block:: python + + from je_auto_control import read_barcodes + + # decode every 1-D barcode currently on screen + for code in read_barcodes(): + print(code["type"], code["text"], code["points"]) + + # restrict to a region, or decode a saved image instead of the screen + read_barcodes(region=[0, 0, 400, 200]) + read_barcodes("label.png") + +``read_barcodes(source=None, *, region=None, decoder=None)`` returns a list of +``{"text", "type", "points"}`` dicts, one per detected barcode (``points`` is the +four-corner polygon in image coordinates). ``source`` may be an image path or an +array; when omitted the screen (optionally cropped to ``region``) is grabbed. The +grayscale conversion reuses the shared ``visual_match`` haystack loader, so no new +image-loading code is added. + +Executor command +---------------- + +``AC_read_barcodes`` (``source`` / ``region`` → ``{count, barcodes}``) is exposed +as the MCP tool ``ac_read_barcodes`` (read-only) and as a Script Builder command +**Read Barcodes (1-D)** under **OCR**. diff --git a/docs/source/Eng/eng_index.rst b/docs/source/Eng/eng_index.rst index 93c28570..c978ad5c 100644 --- a/docs/source/Eng/eng_index.rst +++ b/docs/source/Eng/eng_index.rst @@ -179,6 +179,7 @@ Comprehensive guides for all AutoControl features. doc/new_features/v154_features_doc doc/new_features/v155_features_doc doc/new_features/v156_features_doc + doc/new_features/v157_features_doc doc/ocr_backends/ocr_backends_doc doc/observability/observability_doc doc/operations_layer/operations_layer_doc diff --git a/docs/source/Zh/doc/new_features/v157_features_doc.rst b/docs/source/Zh/doc/new_features/v157_features_doc.rst new file mode 100644 index 00000000..be272927 --- /dev/null +++ b/docs/source/Zh/doc/new_features/v157_features_doc.rst @@ -0,0 +1,40 @@ +一維條碼解碼 +============ + +框架已能解碼 QR Code（``read_qr``），但缺少能讀取 *一維* 條碼（EAN-13 / EAN-8 / +UPC-A / Code-128）的功能——這些正是商品、庫存標籤與物流面單上最常見的條碼，也是 +桌面或自助機自動化最需要從商品畫面讀取的資訊。``read_barcodes`` 透過 OpenCV 的 +``cv2.barcode.BarcodeDetector`` 補上這一塊。 + +解碼步驟是一個**可注入接縫**：預設解碼器呼叫 OpenCV，但測試（或其他引擎）可以傳入 +自己的 ``decoder`` 可呼叫物件，因此此功能可在無頭環境下完整單元測試，且能優雅降級 +——若 OpenCV 編譯時未含 ``barcode`` 模組，僅回傳空清單而非拋出例外。不匯入 +``PySide6``。 + +無頭 API +-------- + +.. code-block:: python + + from je_auto_control import read_barcodes + + # 解碼螢幕上目前所有一維條碼 + for code in read_barcodes(): + print(code["type"], code["text"], code["points"]) + + # 限定區域，或改為解碼已存檔的影像 + read_barcodes(region=[0, 0, 400, 200]) + read_barcodes("label.png") + +``read_barcodes(source=None, *, region=None, decoder=None)`` 回傳 +``{"text", "type", "points"}`` 字典清單，每偵測到一個條碼一筆（``points`` 為影像 +座標中的四角多邊形）。``source`` 可為影像路徑或陣列；省略時擷取螢幕（可選擇以 +``region`` 裁切）。灰階轉換重用共用的 ``visual_match`` haystack 載入器，不新增 +影像載入程式碼。 + +執行器指令 +---------- + +``AC_read_barcodes``（``source`` / ``region`` → ``{count, barcodes}``）以 MCP 工具 +``ac_read_barcodes``（唯讀）及 Script Builder 指令 **Read Barcodes (1-D)**（位於 +**OCR** 分類下）形式提供。 diff --git a/docs/source/Zh/zh_index.rst b/docs/source/Zh/zh_index.rst index d0ebb495..870d606a 100644 --- a/docs/source/Zh/zh_index.rst +++ b/docs/source/Zh/zh_index.rst @@ -179,6 +179,7 @@ AutoControl 所有功能的完整使用指南。 doc/new_features/v154_features_doc doc/new_features/v155_features_doc doc/new_features/v156_features_doc + doc/new_features/v157_features_doc doc/ocr_backends/ocr_backends_doc doc/observability/observability_doc doc/operations_layer/operations_layer_doc diff --git a/je_auto_control/__init__.py b/je_auto_control/__init__.py index a0019dec..ba0b903c 100644 --- a/je_auto_control/__init__.py +++ b/je_auto_control/__init__.py @@ -385,6 +385,8 @@ from je_auto_control.utils.element_scoring import ( ScoredCandidate, best_candidate, score_candidates, ) +# 1-D barcode decoding (EAN / UPC) with an injectable decoder seam +from je_auto_control.utils.barcode import read_barcodes # CI workflow annotations (GitHub Actions) from je_auto_control.utils.ci_annotations import ( emit_annotations, format_annotation, @@ -1278,6 +1280,7 @@ def start_autocontrol_gui(*args, **kwargs): "score_candidates", "best_candidate", "ScoredCandidate", + "read_barcodes", "emit_annotations", "format_annotation", "ClipboardHistory", "default_clipboard_history", "analyze_heal_log", "heal_stats", "scan_secrets", diff --git a/je_auto_control/gui/script_builder/command_schema.py b/je_auto_control/gui/script_builder/command_schema.py index c09d0823..93ff8672 100644 --- a/je_auto_control/gui/script_builder/command_schema.py +++ b/je_auto_control/gui/script_builder/command_schema.py @@ -586,6 +586,15 @@ def _add_ocr_specs(specs: List[CommandSpec]) -> None: ), description="Decode QR codes in a screen region (OpenCV).", )) + specs.append(CommandSpec( + "AC_read_barcodes", "OCR", "Read Barcodes (1-D)", + fields=( + FieldSpec("source", FieldType.FILE_PATH, optional=True), + FieldSpec("region", FieldType.STRING, optional=True, + placeholder=_REGION_PLACEHOLDER), + ), + description="Decode 1-D barcodes (EAN / UPC) in an image / screen region.", + )) specs.append(CommandSpec( "AC_scroll_to_find", "OCR", "Scroll Until Visible", fields=( diff --git a/je_auto_control/utils/barcode/__init__.py b/je_auto_control/utils/barcode/__init__.py new file mode 100644 index 00000000..deceedb1 --- /dev/null +++ b/je_auto_control/utils/barcode/__init__.py @@ -0,0 +1,4 @@ +"""1-D barcode decoding (EAN / UPC) with an injectable decoder seam.""" +from je_auto_control.utils.barcode.barcode import read_barcodes + +__all__ = ["read_barcodes"] diff --git a/je_auto_control/utils/barcode/barcode.py b/je_auto_control/utils/barcode/barcode.py new file mode 100644 index 00000000..41c06792 --- /dev/null +++ b/je_auto_control/utils/barcode/barcode.py @@ -0,0 +1,49 @@ +"""1-D barcode decoding — EAN / UPC, with an injectable decoder seam. + +The ``qr`` module decodes QR codes only (``cv2.QRCodeDetector``); there is no 1-D / +linear barcode (EAN-8/13, UPC-A/E, Code-128) decode. This mirrors ``qr``'s injectable- +decoder pattern so it is testable without a real barcode and future-proof against +backend availability: the default decoder uses ``cv2.barcode.BarcodeDetector`` (base +OpenCV since 4.8) and degrades to an empty result when that module is absent. + +Runs on an injectable image (ndarray / path / PIL, default: grab the screen / region), +so it is headless-testable on synthetic arrays with an injected decoder. OpenCV + +NumPy come in via ``je_open_cv``. Imports no ``PySide6``. +""" +from typing import Any, Callable, Dict, List, Optional, Sequence + +from je_auto_control.utils.visual_match.visual_match import _haystack_gray + +ImageSource = Any +Decoder = Callable[[Any], List[Dict[str, Any]]] + + +def _default_decoder(image) -> List[Dict[str, Any]]: + """Decode 1-D barcodes with ``cv2.barcode`` (empty if the module is absent).""" + import cv2 + if not hasattr(cv2, "barcode"): + return [] + retval, infos, types, points = cv2.barcode.BarcodeDetector( + ).detectAndDecodeWithType(image) + if not retval: + return [] + results: List[Dict[str, Any]] = [] + for text, kind, corners in zip(infos, types, points): + if not text: + continue + results.append({"text": text, "type": str(kind), + "points": [[int(x), int(y)] for x, y in corners]}) + return results + + +def read_barcodes(source: Optional[ImageSource] = None, *, + region: Optional[Sequence[int]] = None, + decoder: Optional[Decoder] = None) -> List[Dict[str, Any]]: + """Return the 1-D barcodes found in ``source`` (or the screen / ``region``). + + Each result is ``{text, type, points}``. ``decoder`` is injectable (it receives the + loaded image and returns the result list); the default uses ``cv2.barcode`` and + returns ``[]`` when that backend is unavailable. + """ + image = _haystack_gray(source, region) + return (decoder or _default_decoder)(image) diff --git a/je_auto_control/utils/executor/action_executor.py b/je_auto_control/utils/executor/action_executor.py index fe457219..624c9ccf 100644 --- a/je_auto_control/utils/executor/action_executor.py +++ b/je_auto_control/utils/executor/action_executor.py @@ -3953,6 +3953,16 @@ def _best_candidate(candidates: Any, want_role: Any = None, want_name: Any = Non "best": best.to_dict() if best is not None else None} +def _read_barcodes(source: Any = None, region: Any = None) -> Dict[str, Any]: + """Adapter: decode 1-D barcodes on screen / in an image.""" + import json + from je_auto_control.utils.barcode import read_barcodes + if isinstance(region, str): + region = json.loads(region) if region.strip() else None + barcodes = read_barcodes(source, region=region) + return {"count": len(barcodes), "barcodes": barcodes} + + def _with_modifiers(modifiers: Any, actions: Any) -> Dict[str, Any]: """Adapter: run nested actions while modifier keys are held down.""" import json @@ -5716,6 +5726,7 @@ def __init__(self): "AC_assign_stable_ids": _assign_stable_ids, "AC_score_candidates": _score_candidates, "AC_best_candidate": _best_candidate, + "AC_read_barcodes": _read_barcodes, "AC_tile_rect": _tile_rect, "AC_grid_rects": _grid_rects, "AC_cascade_rects": _cascade_rects, diff --git a/je_auto_control/utils/mcp_server/tools/_factories.py b/je_auto_control/utils/mcp_server/tools/_factories.py index a961da75..a43eccab 100644 --- a/je_auto_control/utils/mcp_server/tools/_factories.py +++ b/je_auto_control/utils/mcp_server/tools/_factories.py @@ -3410,6 +3410,24 @@ def element_scoring_tools() -> List[MCPTool]: ] +def barcode_tools() -> List[MCPTool]: + return [ + MCPTool( + name="ac_read_barcodes", + description=("Decode 1-D barcodes (EAN / UPC / Code-128) in 'source' " + "(image path; default: screen grab of 'region'). Returns " + "{count, barcodes:[{text, type, points}]}. QR codes have " + "their own tool."), + input_schema=schema({ + "source": {"type": "string"}, + "region": {"type": "array", "items": {"type": "integer"}}}, + required=[]), + handler=h.read_barcodes, + annotations=READ_ONLY, + ), + ] + + def ssim_tools() -> List[MCPTool]: return [ MCPTool( @@ -6919,7 +6937,8 @@ def media_assert_tools() -> List[MCPTool]: motion_regions_tools, window_zorder_tools, soft_assert_tools, perceptual_diff_tools, window_geometry_tools, cua_action_tools, observation_tools, action_grounding_tools, agent_replay_tools, - element_diff_tools, element_scoring_tools, plugin_sdk_tools, governance_tools, + element_diff_tools, element_scoring_tools, barcode_tools, plugin_sdk_tools, + governance_tools, credential_lease_tools, egress_tools, approval_testing_tools, trajectory_eval_tools, compliance_tools, agent_trace_tools, video_report_tools, fuzzy_tools, artifact_store_tools, image_dedup_tools, diff --git a/je_auto_control/utils/mcp_server/tools/_handlers.py b/je_auto_control/utils/mcp_server/tools/_handlers.py index 44c27623..a7f5d136 100644 --- a/je_auto_control/utils/mcp_server/tools/_handlers.py +++ b/je_auto_control/utils/mcp_server/tools/_handlers.py @@ -2350,6 +2350,11 @@ def best_candidate(candidates, want_role=None, want_name=None, anchor=None): return _best_candidate(candidates, want_role, want_name, anchor) +def read_barcodes(source=None, region=None): + from je_auto_control.utils.executor.action_executor import _read_barcodes + return _read_barcodes(source, region) + + def detect_drift(reference, current, threshold=0.25, bins=10): from je_auto_control.utils.executor.action_executor import _detect_drift return _detect_drift(reference, current, threshold, bins) diff --git a/test/unit_test/headless/test_barcode_batch.py b/test/unit_test/headless/test_barcode_batch.py new file mode 100644 index 00000000..65b29987 --- /dev/null +++ b/test/unit_test/headless/test_barcode_batch.py @@ -0,0 +1,52 @@ +"""Headless tests for 1-D barcode decoding. No Qt; decoder is injected.""" +import pytest + +import je_auto_control as ac + +np = pytest.importorskip("numpy") +pytest.importorskip("cv2") + +from je_auto_control.utils.barcode import read_barcodes # noqa: E402 + + +def _image(): + return np.full((40, 120), 255, dtype=np.uint8) + + +def test_injected_decoder_is_used(): + rows = [{"text": "012345678905", "type": "EAN_13", + "points": [[0, 0], [100, 0], [100, 30], [0, 30]]}] + result = read_barcodes(_image(), decoder=lambda image: rows) + assert result == rows + + +def test_decoder_receives_the_image(): + seen = {} + + def decoder(image): + seen["shape"] = getattr(image, "shape", None) + return [] + + read_barcodes(_image(), decoder=decoder) + assert seen["shape"] == (40, 120) + + +def test_default_decoder_blank_image_is_empty(): + # a blank image has no barcodes (graceful, regardless of cv2.barcode presence) + assert read_barcodes(_image()) == [] + + +# --- wiring --------------------------------------------------------------- + +def test_wiring(): + assert "AC_read_barcodes" in set(ac.executor.known_commands()) + from je_auto_control.utils.mcp_server.tools import build_default_tool_registry + names = {t.name for t in build_default_tool_registry()} + assert "ac_read_barcodes" in names + from je_auto_control.gui.script_builder.command_schema import _build_specs + specs = {s.command for s in _build_specs()} + assert "AC_read_barcodes" in specs + + +def test_facade_exports(): + assert hasattr(ac, "read_barcodes") and "read_barcodes" in ac.__all__