Skip to content

Commit debb186

Browse files
authored
Merge pull request #406 from Integration-Automation/feat/image-quality-batch
Add image_quality: sharpness/contrast/brightness gate before OCR
2 parents 000a5db + 1d8e86e commit debb186

11 files changed

Lines changed: 341 additions & 0 deletions

File tree

WHATS_NEW.md

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,11 @@
11
# What's New — AutoControl
22

3+
## What's new (2026-06-24) — Image Quality Scoring (sharpness / contrast / brightness gate)
4+
5+
Refuse to OCR a blurry or washed-out frame — score quality and gate before recognition. Full reference: [`docs/source/Eng/doc/new_features/v188_features_doc.rst`](docs/source/Eng/doc/new_features/v188_features_doc.rst).
6+
7+
- **`image_quality` / `is_blurry` / `quality_gate`** (`AC_image_quality`, `AC_quality_gate`): OCR and template matching quietly fail on a blurry, washed-out or too-dark capture, and the caller can't tell a *missing* element from an *unreadable* one. This measures sharpness (variance of the Laplacian), contrast (grayscale stddev) and brightness (mean 0–255); `quality_gate` turns them into `{passed, issues}` flagging `blurry` / `low_contrast` / `too_dark` / `too_bright` so a script can pre-process or re-capture before OCR. Reuses `visual_match`'s grayscale loader (any ndarray / path / PIL image, or the live screen); cv2/numpy lazily imported. No `PySide6`.
8+
39
## What's new (2026-06-24) — Drop Files onto a Window (WM_DROPFILES)
410

511
Complete a drag-and-drop programmatically — drop files onto a target window. Full reference: [`docs/source/Eng/doc/new_features/v187_features_doc.rst`](docs/source/Eng/doc/new_features/v187_features_doc.rst).
Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
Image Quality Scoring (sharpness / contrast / brightness gate)
2+
==============================================================
3+
4+
OCR and template matching quietly fail on a blurry, washed-out or too-dark
5+
capture — the locate returns nothing and the caller can't tell a *missing*
6+
element from an *unreadable* one. ``image_quality`` measures the three things
7+
that wreck recognition and gates on them:
8+
9+
* **sharpness** — variance of the Laplacian (low = blurry / out of focus),
10+
* **contrast** — standard deviation of the grayscale (low = washed out),
11+
* **brightness** — mean grayscale 0–255 (too low = dark, too high = blown out).
12+
13+
:func:`image_quality` returns the raw metrics, :func:`is_blurry` is the common
14+
one-liner, and :func:`quality_gate` turns the metrics into a pass / fail verdict
15+
with named issues, so a script can refuse to OCR a bad frame (or pre-process it
16+
first). It reuses ``visual_match``'s grayscale loader, so the source is any
17+
ndarray / path / PIL image (or the live screen when omitted); cv2 / numpy are
18+
lazily imported. Imports no ``PySide6``.
19+
20+
Headless API
21+
------------
22+
23+
.. code-block:: python
24+
25+
from je_auto_control import image_quality, is_blurry, quality_gate
26+
27+
image_quality("frame.png")
28+
# {"sharpness": 842.1, "contrast": 58.3, "brightness": 131.0}
29+
30+
if is_blurry("frame.png", threshold=100):
31+
... # capture again / sharpen before OCR
32+
33+
gate = quality_gate("frame.png", min_sharpness=100, min_contrast=12)
34+
# {"sharpness": .., "contrast": .., "brightness": .., "passed": False,
35+
# "issues": ["blurry", "too_dark"]}
36+
37+
``quality_gate`` flags ``blurry`` / ``low_contrast`` / ``too_dark`` /
38+
``too_bright``; ``passed`` is True only when no issue fires. ``region`` applies to
39+
a live-screen grab (omit ``source`` to grade the screen). Thresholds are tunable;
40+
the defaults suit typical UI screenshots.
41+
42+
Executor commands
43+
-----------------
44+
45+
``AC_image_quality`` (``source`` / ``region``) and ``AC_quality_gate`` (plus
46+
``min_sharpness`` / ``min_contrast``). They are exposed as read-only ``ac_*`` MCP
47+
tools and as Script Builder commands under **Image**.
Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
影像品質評分(銳利度 / 對比 / 亮度門檻)
2+
=======================================
3+
4+
OCR 與模板比對在模糊、褪色或太暗的擷取畫面上會悄悄失敗——定位回傳空值,呼叫端無法分辨是元素
5+
*不存在*還是畫面*無法辨識*。``image_quality`` 量測三項會破壞辨識的指標並據以把關:
6+
7+
* **sharpness(銳利度)**——Laplacian 的變異數(低 = 模糊 / 失焦),
8+
* **contrast(對比)**——灰階的標準差(低 = 褪色),
9+
* **brightness(亮度)**——灰階平均 0–255(太低 = 太暗,太高 = 過曝)。
10+
11+
:func:`image_quality` 回傳原始指標,:func:`is_blurry` 是常用的一行式,:func:`quality_gate` 把
12+
指標轉成通過 / 失敗的判定並附上具名問題,讓腳本可以拒絕對壞畫面做 OCR(或先做前處理)。它重用
13+
``visual_match`` 的灰階載入器,因此來源可為任何 ndarray / 路徑 / PIL 影像(省略時則為存活螢幕);
14+
cv2 / numpy 為延遲匯入。不匯入 ``PySide6``。
15+
16+
無頭 API
17+
--------
18+
19+
.. code-block:: python
20+
21+
from je_auto_control import image_quality, is_blurry, quality_gate
22+
23+
image_quality("frame.png")
24+
# {"sharpness": 842.1, "contrast": 58.3, "brightness": 131.0}
25+
26+
if is_blurry("frame.png", threshold=100):
27+
... # 在 OCR 前重新擷取 / 銳化
28+
29+
gate = quality_gate("frame.png", min_sharpness=100, min_contrast=12)
30+
# {"sharpness": .., "contrast": .., "brightness": .., "passed": False,
31+
# "issues": ["blurry", "too_dark"]}
32+
33+
``quality_gate`` 會標記 ``blurry`` / ``low_contrast`` / ``too_dark`` /
34+
``too_bright``;只有在沒有任何問題時 ``passed`` 才為 True。``region`` 套用於存活螢幕擷取(省略
35+
``source`` 即評分螢幕)。門檻可調整;預設值適合一般 UI 截圖。
36+
37+
執行器指令
38+
----------
39+
40+
``AC_image_quality``(``source`` / ``region``)與 ``AC_quality_gate``(另加
41+
``min_sharpness`` / ``min_contrast``)。皆以唯讀 ``ac_*`` MCP 工具及 Script Builder 指令
42+
(位於 **Image** 分類下)形式提供。

je_auto_control/__init__.py

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -78,6 +78,10 @@
7878
)
7979
# Drop files onto a window (WM_DROPFILES sender)
8080
from je_auto_control.utils.file_drop import drop_files, plan_file_drop
81+
# Image quality scoring (sharpness / contrast / brightness gate before OCR)
82+
from je_auto_control.utils.image_quality import (
83+
image_quality, is_blurry, quality_gate,
84+
)
8185
# VLM element locator (headless)
8286
from je_auto_control.utils.vision import (
8387
VLMNotAvailableError, click_by_description, locate_by_description,
@@ -1652,6 +1656,7 @@ def start_autocontrol_gui(*args, **kwargs):
16521656
"classify_format", "classify_formats", "diff_formats",
16531657
"list_clipboard_formats", "clipboard_formats",
16541658
"plan_file_drop", "drop_files",
1659+
"image_quality", "is_blurry", "quality_gate",
16551660
# VLM locator
16561661
"VLMNotAvailableError", "locate_by_description", "click_by_description",
16571662
"verify_description",

je_auto_control/gui/script_builder/command_schema.py

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -741,6 +741,28 @@ def _add_image_specs(specs: List[CommandSpec]) -> None:
741741
),
742742
description="Detect a palette/view change vs a reference (illumination-robust).",
743743
))
744+
specs.append(CommandSpec(
745+
"AC_image_quality", "Image", "Image Quality",
746+
fields=(
747+
FieldSpec("source", FieldType.FILE_PATH, optional=True),
748+
FieldSpec("region", FieldType.STRING, optional=True,
749+
placeholder=_REGION_PLACEHOLDER),
750+
),
751+
description="Sharpness / contrast / brightness of an image or the screen.",
752+
))
753+
specs.append(CommandSpec(
754+
"AC_quality_gate", "Image", "Quality Gate (OCR-ready?)",
755+
fields=(
756+
FieldSpec("source", FieldType.FILE_PATH, optional=True),
757+
FieldSpec("region", FieldType.STRING, optional=True,
758+
placeholder=_REGION_PLACEHOLDER),
759+
FieldSpec("min_sharpness", FieldType.FLOAT, optional=True,
760+
default=100.0),
761+
FieldSpec("min_contrast", FieldType.FLOAT, optional=True,
762+
default=12.0),
763+
),
764+
description="Pass/fail an image for OCR readability with named issues.",
765+
))
744766
specs.append(CommandSpec(
745767
"AC_changed_regions", "Image", "Changed Regions (motion)",
746768
fields=(

je_auto_control/utils/executor/action_executor.py

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -4274,6 +4274,30 @@ def _drop_files(hwnd: Any, paths: Any, point: Any = None) -> Dict[str, Any]:
42744274
return {"dropped": bool(dropped), "count": len(coerced)}
42754275

42764276

4277+
def _coerce_region(region: Any):
4278+
"""Normalise a region argument (JSON '[x,y,w,h]' string / list / None)."""
4279+
import json
4280+
if isinstance(region, str):
4281+
return json.loads(region) if region.strip() else None
4282+
return region
4283+
4284+
4285+
def _image_quality(source: Any = None, region: Any = None) -> Dict[str, Any]:
4286+
"""Adapter: sharpness / contrast / brightness of an image or the screen."""
4287+
from je_auto_control.utils.image_quality import image_quality
4288+
return image_quality(source, region=_coerce_region(region))
4289+
4290+
4291+
def _quality_gate(source: Any = None, region: Any = None,
4292+
min_sharpness: Any = 100.0,
4293+
min_contrast: Any = 12.0) -> Dict[str, Any]:
4294+
"""Adapter: pass / fail an image for OCR readability with named issues."""
4295+
from je_auto_control.utils.image_quality import quality_gate
4296+
return quality_gate(source, region=_coerce_region(region),
4297+
min_sharpness=float(min_sharpness),
4298+
min_contrast=float(min_contrast))
4299+
4300+
42774301
def _image_histogram(source: Any = None, bins: Any = 32, space: str = "hsv",
42784302
region: Any = None) -> Dict[str, Any]:
42794303
"""Adapter: per-channel colour histogram of an image / the screen."""
@@ -6496,6 +6520,8 @@ def __init__(self):
64966520
"AC_diff_formats": _diff_formats,
64976521
"AC_plan_file_drop": _plan_file_drop,
64986522
"AC_drop_files": _drop_files,
6523+
"AC_image_quality": _image_quality,
6524+
"AC_quality_gate": _quality_gate,
64996525
"AC_image_histogram": _image_histogram,
65006526
"AC_histogram_changed": _histogram_changed,
65016527
"AC_changed_regions": _changed_regions,
Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
"""Score image quality (sharpness / contrast / brightness) before OCR / matching."""
2+
from je_auto_control.utils.image_quality.image_quality import (
3+
image_quality, is_blurry, quality_gate,
4+
)
5+
6+
__all__ = ["image_quality", "is_blurry", "quality_gate"]
Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
"""Score image quality before OCR / template matching.
2+
3+
OCR and template matching quietly fail on a blurry, washed-out or too-dark
4+
capture — the locate returns nothing and the caller can't tell a *missing*
5+
element from an *unreadable* one. ``image_quality`` measures the three things
6+
that wreck recognition and gates on them:
7+
8+
* **sharpness** — variance of the Laplacian (low = blurry / out of focus),
9+
* **contrast** — standard deviation of the grayscale (low = washed out),
10+
* **brightness** — mean grayscale 0–255 (too low = dark, too high = blown out).
11+
12+
:func:`image_quality` returns the raw metrics, :func:`is_blurry` is the common
13+
one-liner, and :func:`quality_gate` turns the metrics into a pass / fail verdict
14+
with named issues so a script can refuse to OCR a bad frame (or pre-process it
15+
first). It reuses ``visual_match``'s grayscale loader, so the source is any
16+
ndarray / path / PIL image (or the live screen when omitted); ``region`` applies
17+
to a live-screen grab. cv2 / numpy are lazily imported. Imports no ``PySide6``.
18+
"""
19+
from typing import Any, Dict, Optional, Sequence, Tuple
20+
21+
ImageSource = Any
22+
23+
24+
def _gray(source: Optional[ImageSource], region: Optional[Sequence[int]]):
25+
from je_auto_control.utils.visual_match.visual_match import _haystack_gray
26+
return _haystack_gray(source, region)
27+
28+
29+
def image_quality(source: Optional[ImageSource] = None, *,
30+
region: Optional[Sequence[int]] = None) -> Dict[str, float]:
31+
"""Return ``{sharpness, contrast, brightness}`` for an image (or live screen).
32+
33+
``sharpness`` is the variance of the Laplacian, ``contrast`` the grayscale
34+
standard deviation, ``brightness`` the mean grayscale (0–255).
35+
"""
36+
import cv2
37+
gray = _gray(source, region)
38+
return {"sharpness": float(cv2.Laplacian(gray, cv2.CV_64F).var()),
39+
"contrast": float(gray.std()),
40+
"brightness": float(gray.mean())}
41+
42+
43+
def is_blurry(source: Optional[ImageSource] = None, *,
44+
region: Optional[Sequence[int]] = None,
45+
threshold: float = 100.0) -> bool:
46+
"""Return True if the image's Laplacian variance is below ``threshold``."""
47+
return image_quality(source, region=region)["sharpness"] < float(threshold)
48+
49+
50+
def quality_gate(source: Optional[ImageSource] = None, *,
51+
region: Optional[Sequence[int]] = None,
52+
min_sharpness: float = 100.0, min_contrast: float = 12.0,
53+
brightness_range: Tuple[float, float] = (40.0, 220.0),
54+
) -> Dict[str, Any]:
55+
"""Grade an image for OCR readability: ``{..., passed, issues}``.
56+
57+
``issues`` flags ``blurry`` / ``low_contrast`` / ``too_dark`` / ``too_bright``;
58+
``passed`` is True only when no issue fires.
59+
"""
60+
metrics = image_quality(source, region=region)
61+
low, high = brightness_range
62+
issues = []
63+
if metrics["sharpness"] < float(min_sharpness):
64+
issues.append("blurry")
65+
if metrics["contrast"] < float(min_contrast):
66+
issues.append("low_contrast")
67+
if metrics["brightness"] < float(low):
68+
issues.append("too_dark")
69+
elif metrics["brightness"] > float(high):
70+
issues.append("too_bright")
71+
return {**metrics, "passed": not issues, "issues": issues}

je_auto_control/utils/mcp_server/tools/_factories.py

Lines changed: 26 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3385,6 +3385,32 @@ def img_histogram_tools() -> List[MCPTool]:
33853385
handler=h.histogram_changed,
33863386
annotations=READ_ONLY,
33873387
),
3388+
MCPTool(
3389+
name="ac_image_quality",
3390+
description=("Measure image quality of 'source' (image path; default "
3391+
"screen grab of 'region'): {sharpness (Laplacian "
3392+
"variance — low=blurry), contrast (grayscale stddev), "
3393+
"brightness (mean 0-255)}."),
3394+
input_schema=schema({
3395+
"source": {"type": "string"},
3396+
"region": {"type": "array", "items": {"type": "integer"}}}),
3397+
handler=h.image_quality,
3398+
annotations=READ_ONLY,
3399+
),
3400+
MCPTool(
3401+
name="ac_quality_gate",
3402+
description=("Grade 'source' for OCR readability: {sharpness, "
3403+
"contrast, brightness, passed, issues}. 'issues' flags "
3404+
"blurry / low_contrast / too_dark / too_bright. Tune with "
3405+
"'min_sharpness' / 'min_contrast'."),
3406+
input_schema=schema({
3407+
"source": {"type": "string"},
3408+
"region": {"type": "array", "items": {"type": "integer"}},
3409+
"min_sharpness": {"type": "number"},
3410+
"min_contrast": {"type": "number"}}),
3411+
handler=h.quality_gate,
3412+
annotations=READ_ONLY,
3413+
),
33883414
]
33893415

33903416

je_auto_control/utils/mcp_server/tools/_handlers.py

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2509,6 +2509,17 @@ def drop_files(hwnd, paths, point=None):
25092509
return _drop_files(hwnd, paths, point)
25102510

25112511

2512+
def image_quality(source=None, region=None):
2513+
from je_auto_control.utils.executor.action_executor import _image_quality
2514+
return _image_quality(source, region)
2515+
2516+
2517+
def quality_gate(source=None, region=None, min_sharpness=100.0,
2518+
min_contrast=12.0):
2519+
from je_auto_control.utils.executor.action_executor import _quality_gate
2520+
return _quality_gate(source, region, min_sharpness, min_contrast)
2521+
2522+
25122523
def image_histogram(source=None, bins=32, space="hsv", region=None):
25132524
from je_auto_control.utils.executor.action_executor import _image_histogram
25142525
return _image_histogram(source, bins, space, region)

0 commit comments

Comments
 (0)