Add HSV colour-space segmentation (lighting-robust colour location)

README/WHATS_NEW_zh-CN.md

@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-23) — HSV 色彩空间分割
+
+不论光照都能找出「任一色阶的红色」。完整参考:[`docs/source/Zh/doc/new_features/v139_features_doc.rst`](../docs/source/Zh/doc/new_features/v139_features_doc.rst)。
+
+- **`dominant_hue_regions` / `segment_hsv` / `color_mask`**(`AC_dominant_hue_regions`、`AC_segment_hsv`):`find_color_region` 在 RGB 以各通道 ± 框遮罩——无法匹配「同一颜色但不同亮度」(状态灯、强调色、主题色调)。HSV 把色相与亮度分离,因此「色相带 + 饱和度 / 明度下限」可在不同光照下捕捉所有色阶。`dominant_hue_regions(hue=…)` 自动处理红色 0/180 环绕;`segment_hsv` 接受明确带;两者皆返回 `{x,y,width,height,area,center}` 区块并沿用共用连通元件辅助函数。可注入 haystack → 无头可测。
+
 ## 本次更新 (2026-06-23) — 融合并排序屏幕元素框
 
 把原始的 OCR + 图标 + a11y 框转成一份干净、已编号的元素列表。完整参考:[`docs/source/Zh/doc/new_features/v138_features_doc.rst`](../docs/source/Zh/doc/new_features/v138_features_doc.rst)。

README/WHATS_NEW_zh-TW.md

@@ -1,5 +1,11 @@
 # 本次更新 — AutoControl
 
+## 本次更新 (2026-06-23) — HSV 色彩空間分割
+
+不論光照都能找出「任一色階的紅色」。完整參考:[`docs/source/Zh/doc/new_features/v139_features_doc.rst`](../docs/source/Zh/doc/new_features/v139_features_doc.rst)。
+
+- **`dominant_hue_regions` / `segment_hsv` / `color_mask`**(`AC_dominant_hue_regions`、`AC_segment_hsv`):`find_color_region` 在 RGB 以各通道 ± 框遮罩——無法比對「同一顏色但不同亮度」(狀態燈、強調色、主題色調)。HSV 把色相與亮度分離,因此「色相帶 + 飽和度 / 明度下限」可在不同光照下捕捉所有色階。`dominant_hue_regions(hue=…)` 自動處理紅色 0/180 環繞;`segment_hsv` 接受明確帶;兩者皆回傳 `{x,y,width,height,area,center}` 區塊並沿用共用連通元件輔助函式。可注入 haystack → 無頭可測。
+
 ## 本次更新 (2026-06-23) — 融合並排序螢幕元素框
 
 把原始的 OCR + 圖示 + a11y 框轉成一份乾淨、已編號的元素清單。完整參考:[`docs/source/Zh/doc/new_features/v138_features_doc.rst`](../docs/source/Zh/doc/new_features/v138_features_doc.rst)。

WHATS_NEW.md

@@ -1,5 +1,11 @@
 # What's New — AutoControl
 
+## What's new (2026-06-23) — HSV Colour-Space Segmentation
+
+Find "any shade of red" regardless of lighting. Full reference: [`docs/source/Eng/doc/new_features/v139_features_doc.rst`](docs/source/Eng/doc/new_features/v139_features_doc.rst).
+
+- **`dominant_hue_regions` / `segment_hsv` / `color_mask`** (`AC_dominant_hue_regions`, `AC_segment_hsv`): `find_color_region` masks in RGB with a per-channel ± box — it can't match "the same colour at a different brightness" (status lights, highlights, theme tints). HSV separates hue from brightness, so a hue band + saturation/value floor catches every shade across lighting. `dominant_hue_regions(hue=…)` handles red's 0/180 wrap automatically; `segment_hsv` takes an explicit band; both return `{x,y,width,height,area,center}` blobs reusing the shared connected-components helper. Injectable haystack → headless-testable.
+
 ## What's new (2026-06-23) — Fuse & Order On-Screen Element Boxes
 
 Turn raw OCR + icon + a11y boxes into one clean, numbered element list. Full reference: [`docs/source/Eng/doc/new_features/v138_features_doc.rst`](docs/source/Eng/doc/new_features/v138_features_doc.rst).

docs/source/Eng/doc/new_features/v139_features_doc.rst

@@ -0,0 +1,43 @@
+HSV Colour-Space Segmentation
+=============================
+
+``find_color_region`` masks in RGB with a per-channel ± tolerance box, which fails
+the canonical case: a status light, highlight or theme tint that is "the same colour"
+but at a different *brightness*. HSV separates hue from saturation/value, so a hue
+band with a saturation/value floor catches every shade of a colour across lighting.
+This adds HSV masking and blob boxes, reusing the shared connected-component helper,
+with correct hue-wrap handling for red (which straddles the 0/180 boundary).
+
+Runs on an injectable ``haystack`` (ndarray / path / PIL, RGB), so it is headless-
+testable on synthetic arrays. OpenCV + NumPy come in via ``je_open_cv``. Imports no
+``PySide6``.
+
+Headless API
+------------
+
+.. code-block:: python
+
+    from je_auto_control import dominant_hue_regions, segment_hsv, color_mask
+
+    # Every red region — bright or dark — regardless of lighting (red wrap handled).
+    for r in dominant_hue_regions(hue=0, hue_tol=10, sat_min=80, val_min=80):
+        click(*r["center"])
+
+    # Or an explicit HSV band (H 0-179, S/V 0-255).
+    greens = segment_hsv(lower_hsv=[40, 80, 80], upper_hsv=[80, 255, 255])
+    mask = color_mask(lower_hsv=[40, 80, 80], upper_hsv=[80, 255, 255])
+
+``dominant_hue_regions`` constrains only the hue (± ``hue_tol``) plus a ``sat_min`` /
+``val_min`` floor to skip greys, returning ``{x, y, width, height, area, center}`` per
+blob largest first — so it finds a colour at any brightness, unlike the RGB box.
+``segment_hsv`` takes an explicit ``lower_hsv`` / ``upper_hsv`` band; ``color_mask``
+returns the raw uint8 mask.
+
+Executor commands
+-----------------
+
+``AC_segment_hsv`` (``lower_hsv`` / ``upper_hsv`` / ``min_area`` / ``region``) and
+``AC_dominant_hue_regions`` (``hue`` / ``hue_tol`` / ``sat_min`` / ``val_min`` /
+``min_area`` / ``region``), both returning ``{count, regions, best}``. They are
+exposed as the MCP tools ``ac_segment_hsv`` / ``ac_dominant_hue_regions`` and as
+Script Builder commands under **Image**.

docs/source/Eng/eng_index.rst

@@ -161,6 +161,7 @@ Comprehensive guides for all AutoControl features.
    doc/new_features/v136_features_doc
    doc/new_features/v137_features_doc
    doc/new_features/v138_features_doc
+   doc/new_features/v139_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

docs/source/Zh/doc/new_features/v139_features_doc.rst

@@ -0,0 +1,36 @@
+HSV 色彩空間分割
+================
+
+``find_color_region`` 在 RGB 以各通道 ± 容差框遮罩,這在經典情境會失效:狀態燈、強調色或主題色調是「同一個顏色」
+但*亮度*不同。HSV 把色相與飽和度 / 明度分離,因此「色相帶 + 飽和度 / 明度下限」可在不同光照下捕捉某顏色的所有色階。
+本功能加入 HSV 遮罩與區塊框,沿用共用的連通元件輔助函式,並正確處理紅色的色相環繞(跨越 0/180 邊界)。
+
+在可注入的 ``haystack``(ndarray / 路徑 / PIL,RGB)上執行,因此可對合成陣列做無頭測試。OpenCV + NumPy 透過
+``je_open_cv`` 引入。不匯入 ``PySide6``。
+
+無頭 API
+--------
+
+.. code-block:: python
+
+    from je_auto_control import dominant_hue_regions, segment_hsv, color_mask
+
+    # 每個紅色區域——不論明暗、不論光照(已處理紅色環繞)。
+    for r in dominant_hue_regions(hue=0, hue_tol=10, sat_min=80, val_min=80):
+        click(*r["center"])
+
+    # 或明確的 HSV 帶(H 0-179、S/V 0-255)。
+    greens = segment_hsv(lower_hsv=[40, 80, 80], upper_hsv=[80, 255, 255])
+    mask = color_mask(lower_hsv=[40, 80, 80], upper_hsv=[80, 255, 255])
+
+``dominant_hue_regions`` 只限制色相(± ``hue_tol``)再加 ``sat_min`` / ``val_min`` 下限以略過灰階,為每個區塊回傳
+``{x, y, width, height, area, center}``,由大到小——因此能在任何亮度下找到某顏色,不像 RGB 框。``segment_hsv`` 接受
+明確的 ``lower_hsv`` / ``upper_hsv`` 帶;``color_mask`` 回傳原始 uint8 遮罩。
+
+執行器命令
+----------
+
+``AC_segment_hsv``(``lower_hsv`` / ``upper_hsv`` / ``min_area`` / ``region``)與
+``AC_dominant_hue_regions``(``hue`` / ``hue_tol`` / ``sat_min`` / ``val_min`` / ``min_area`` / ``region``),
+皆回傳 ``{count, regions, best}``。它們以 MCP 工具 ``ac_segment_hsv`` / ``ac_dominant_hue_regions`` 以及 Script
+Builder 中 **Image** 分類下的命令提供。

docs/source/Zh/zh_index.rst

@@ -161,6 +161,7 @@ AutoControl 所有功能的完整使用指南。
    doc/new_features/v136_features_doc
    doc/new_features/v137_features_doc
    doc/new_features/v138_features_doc
+   doc/new_features/v139_features_doc
    doc/ocr_backends/ocr_backends_doc
    doc/observability/observability_doc
    doc/operations_layer/operations_layer_doc

je_auto_control/__init__.py

@@ -316,6 +316,10 @@
 from je_auto_control.utils.element_parse import (
     fuse_elements, iou, merge_boxes, reading_order,
 )
+# HSV colour-space segmentation (lighting-robust colour masking + blob boxes)
+from je_auto_control.utils.hsv_segment import (
+    color_mask, dominant_hue_regions, segment_hsv,
+)
 # CI workflow annotations (GitHub Actions)
 from je_auto_control.utils.ci_annotations import (
     emit_annotations, format_annotation,
@@ -1149,6 +1153,9 @@ def start_autocontrol_gui(*args, **kwargs):
     "merge_boxes",
     "fuse_elements",
     "reading_order",
+    "segment_hsv",
+    "color_mask",
+    "dominant_hue_regions",
     "emit_annotations", "format_annotation",
     "ClipboardHistory", "default_clipboard_history",
     "analyze_heal_log", "heal_stats", "scan_secrets",

je_auto_control/gui/script_builder/command_schema.py

@@ -44,6 +44,7 @@ class CommandSpec:
 
 
 _MOUSE_BUTTONS = ("mouse_left", "mouse_right", "mouse_middle")
+_REGION_PLACEHOLDER = "[left, top, right, bottom]"
 
 
 def _build_specs() -> List[CommandSpec]:
@@ -262,7 +263,7 @@ def _add_image_specs(specs: List[CommandSpec]) -> None:
             FieldSpec("scales", FieldType.STRING, optional=True,
                       placeholder="[0.9, 1.0, 1.1]"),
             FieldSpec("region", FieldType.STRING, optional=True,
-                      placeholder="[left, top, right, bottom]"),
+                      placeholder=_REGION_PLACEHOLDER),
         ),
         description="Locate a template and return its confidence score + scale.",
     ))
@@ -286,7 +287,7 @@ def _add_image_specs(specs: List[CommandSpec]) -> None:
             FieldSpec("min_score", FieldType.FLOAT, optional=True, default=0.9,
                       min_value=0.0, max_value=1.0),
             FieldSpec("region", FieldType.STRING, optional=True,
-                      placeholder="[left, top, right, bottom]"),
+                      placeholder=_REGION_PLACEHOLDER),
         ),
         description="Match counting only opaque/masked pixels (alpha or mask).",
     ))
@@ -310,7 +311,7 @@ def _add_image_specs(specs: List[CommandSpec]) -> None:
             FieldSpec("tolerance", FieldType.INT, optional=True, default=20),
             FieldSpec("min_area", FieldType.INT, optional=True, default=50),
             FieldSpec("region", FieldType.STRING, optional=True,
-                      placeholder="[left, top, right, bottom]"),
+                      placeholder=_REGION_PLACEHOLDER),
         ),
         description="Locate regions by colour (status light / banner / fill).",
     ))
@@ -322,7 +323,7 @@ def _add_image_specs(specs: List[CommandSpec]) -> None:
             FieldSpec("ignore", FieldType.STRING, optional=True,
                       placeholder="[[x, y, w, h], ...]"),
             FieldSpec("region", FieldType.STRING, optional=True,
-                      placeholder="[left, top, right, bottom]"),
+                      placeholder=_REGION_PLACEHOLDER),
         ),
         description="Structural-similarity score (0..1) vs reference / screen.",
     ))
@@ -337,7 +338,7 @@ def _add_image_specs(specs: List[CommandSpec]) -> None:
                       min_value=0.0, max_value=1.0),
             FieldSpec("min_area", FieldType.INT, optional=True, default=50),
             FieldSpec("region", FieldType.STRING, optional=True,
-                      placeholder="[left, top, right, bottom]"),
+                      placeholder=_REGION_PLACEHOLDER),
         ),
         description="Boxes of the regions that structurally changed.",
     ))
@@ -346,7 +347,7 @@ def _add_image_specs(specs: List[CommandSpec]) -> None:
         fields=(
             FieldSpec("template", FieldType.FILE_PATH),
             FieldSpec("region", FieldType.STRING, optional=True,
-                      placeholder="[left, top, right, bottom]"),
+                      placeholder=_REGION_PLACEHOLDER),
             FieldSpec("max_features", FieldType.INT, optional=True, default=500),
             FieldSpec("ratio", FieldType.FLOAT, optional=True, default=0.75,
                       min_value=0.0, max_value=1.0),
@@ -358,7 +359,7 @@ def _add_image_specs(specs: List[CommandSpec]) -> None:
         "AC_find_shapes", "Image", "Find Shapes",
         fields=(
             FieldSpec("region", FieldType.STRING, optional=True,
-                      placeholder="[left, top, right, bottom]"),
+                      placeholder=_REGION_PLACEHOLDER),
             FieldSpec("min_area", FieldType.INT, optional=True, default=400),
             FieldSpec("max_area", FieldType.INT, optional=True),
         ),
@@ -368,7 +369,7 @@ def _add_image_specs(specs: List[CommandSpec]) -> None:
         "AC_find_rectangles", "Image", "Find Rectangles",
         fields=(
             FieldSpec("region", FieldType.STRING, optional=True,
-                      placeholder="[left, top, right, bottom]"),
+                      placeholder=_REGION_PLACEHOLDER),
             FieldSpec("min_area", FieldType.INT, optional=True, default=400),
             FieldSpec("max_area", FieldType.INT, optional=True),
             FieldSpec("aspect_range", FieldType.STRING, optional=True,
@@ -387,10 +388,34 @@ def _add_image_specs(specs: List[CommandSpec]) -> None:
                       placeholder="grayscale,upscale,binarize"),
             FieldSpec("scale", FieldType.FLOAT, optional=True, default=2.0),
             FieldSpec("region", FieldType.STRING, optional=True,
-                      placeholder="[left, top, right, bottom]"),
+                      placeholder=_REGION_PLACEHOLDER),
         ),
         description="Clean up an image for OCR / matching (grayscale/binarize/…).",
     ))
+    specs.append(CommandSpec(
+        "AC_segment_hsv", "Image", "Segment by HSV",
+        fields=(
+            FieldSpec("lower_hsv", FieldType.STRING, placeholder="[40, 80, 80]"),
+            FieldSpec("upper_hsv", FieldType.STRING, placeholder="[80, 255, 255]"),
+            FieldSpec("min_area", FieldType.INT, optional=True, default=50),
+            FieldSpec("region", FieldType.STRING, optional=True,
+                      placeholder=_REGION_PLACEHOLDER),
+        ),
+        description="Locate regions inside an HSV band (lighting-robust).",
+    ))
+    specs.append(CommandSpec(
+        "AC_dominant_hue_regions", "Image", "Find Hue Regions",
+        fields=(
+            FieldSpec("hue", FieldType.INT, placeholder="0=red, 60=green, 120=blue"),
+            FieldSpec("hue_tol", FieldType.INT, optional=True, default=10),
+            FieldSpec("sat_min", FieldType.INT, optional=True, default=80),
+            FieldSpec("val_min", FieldType.INT, optional=True, default=80),
+            FieldSpec("min_area", FieldType.INT, optional=True, default=50),
+            FieldSpec("region", FieldType.STRING, optional=True,
+                      placeholder=_REGION_PLACEHOLDER),
+        ),
+        description="Locate any shade of a hue, any brightness (handles red wrap).",
+    ))
     specs.append(CommandSpec(
         "AC_fuse_elements", "Image", "Fuse Element Boxes",
         fields=(
@@ -684,7 +709,7 @@ def _add_flow_specs(specs: List[CommandSpec]) -> None:
             FieldSpec("min_score", FieldType.FLOAT, optional=True, default=0.8,
                       min_value=0.0, max_value=1.0),
             FieldSpec("region", FieldType.STRING, optional=True,
-                      placeholder="[left, top, right, bottom]"),
+                      placeholder=_REGION_PLACEHOLDER),
         ),
         description="Wait until a target is visible + stable before acting.",
     ))
@@ -742,7 +767,7 @@ def _add_flow_specs(specs: List[CommandSpec]) -> None:
         fields=(
             FieldSpec("target_rgb", FieldType.STRING, placeholder="[0, 200, 0]"),
             FieldSpec("region", FieldType.STRING, optional=True,
-                      placeholder="[left, top, right, bottom]"),
+                      placeholder=_REGION_PLACEHOLDER),
             FieldSpec("tolerance", FieldType.INT, optional=True, default=10),
             FieldSpec("min_fraction", FieldType.FLOAT, optional=True,
                       default=0.5, min_value=0.0, max_value=1.0),

je_auto_control/utils/executor/action_executor.py

@@ -3505,6 +3505,38 @@ def _reading_order(elements: Any, row_tol: Any = 12) -> Dict[str, Any]:
     return {"count": len(ordered), "elements": ordered}
 
 
+def _segment_hsv(lower_hsv: Any, upper_hsv: Any, min_area: Any = 50,
+                 region: Any = None) -> Dict[str, Any]:
+    """Adapter: locate blobs inside an explicit HSV band on the screen."""
+    import json
+    from je_auto_control.utils.hsv_segment import segment_hsv
+    if isinstance(lower_hsv, str):
+        lower_hsv = json.loads(lower_hsv)
+    if isinstance(upper_hsv, str):
+        upper_hsv = json.loads(upper_hsv)
+    if isinstance(region, str):
+        region = json.loads(region) if region.strip() else None
+    boxes = segment_hsv(region=region, lower_hsv=list(lower_hsv),
+                        upper_hsv=list(upper_hsv), min_area=int(min_area))
+    return {"count": len(boxes), "regions": boxes,
+            "best": boxes[0] if boxes else None}
+
+
+def _dominant_hue_regions(hue: Any, hue_tol: Any = 10, sat_min: Any = 80,
+                          val_min: Any = 80, min_area: Any = 50,
+                          region: Any = None) -> Dict[str, Any]:
+    """Adapter: locate any-brightness regions near a hue on the screen."""
+    import json
+    from je_auto_control.utils.hsv_segment import dominant_hue_regions
+    if isinstance(region, str):
+        region = json.loads(region) if region.strip() else None
+    boxes = dominant_hue_regions(region=region, hue=int(hue), hue_tol=int(hue_tol),
+                                 sat_min=int(sat_min), val_min=int(val_min),
+                                 min_area=int(min_area))
+    return {"count": len(boxes), "regions": boxes,
+            "best": boxes[0] if boxes else None}
+
+
 def _with_modifiers(modifiers: Any, actions: Any) -> Dict[str, Any]:
     """Adapter: run nested actions while modifier keys are held down."""
     import json
@@ -5237,6 +5269,8 @@ def __init__(self):
             "AC_wait_actionable": _wait_actionable,
             "AC_fuse_elements": _fuse_elements,
             "AC_reading_order": _reading_order,
+            "AC_segment_hsv": _segment_hsv,
+            "AC_dominant_hue_regions": _dominant_hue_regions,
             "AC_tile_rect": _tile_rect,
             "AC_grid_rects": _grid_rects,
             "AC_cascade_rects": _cascade_rects,

je_auto_control/utils/hsv_segment/__init__.py

@@ -0,0 +1,6 @@
+"""HSV colour-space segmentation (lighting-robust colour masking + blob boxes)."""
+from je_auto_control.utils.hsv_segment.hsv_segment import (
+    color_mask, dominant_hue_regions, segment_hsv,
+)
+
+__all__ = ["color_mask", "dominant_hue_regions", "segment_hsv"]