Skip to content

Edge case generation notebooks 1 and 1.5 - #37

Open
arvinAM17 wants to merge 32 commits into
mainfrom
feat/edge_case_generation_NB1
Open

Edge case generation notebooks 1 and 1.5#37
arvinAM17 wants to merge 32 commits into
mainfrom
feat/edge_case_generation_NB1

Conversation

@arvinAM17

@arvinAM17 arvinAM17 commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Summary

A feature branch containing implementation of notebooks 1 (sample image generation process) and 1.5 (image generation method comparison) of the edge-case image synthesis reference implementation.
This includes a hydra managed config folder that is split by each dataset, with some default values in the default folder in the config.

Clickup Ticket(s):

Type of Change

  • 🐛 Bug fix (non-breaking change that fixes an issue)
  • [ x ] ✨ New feature (non-breaking change that adds functionality)
  • 💥 Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • [ x ] 📝 Documentation update
  • 🔧 Refactoring (no functional changes)
  • ⚡ Performance improvement
  • 🧪 Test improvements
  • 🔒 Security fix

Summary by CodeRabbit

  • New Features

    • Added an end-to-end workflow for generating synthetic edge-case images from real scenes.
    • Supports road debris, traffic cones, potholes, cracks, animals, fog, snow, and dusk scenarios.
    • Added automatic image conditioning, editing, annotation, quality review, retries, and visualization.
    • Added support for Mapillary Vistas, Nordland, and RDD2022 datasets.
    • Added CPU and GPU configuration options with configurable generation settings.
  • Documentation

    • Added setup instructions, workflow guidance, dataset references, evaluation recommendations, and interactive notebook examples.

Arvin Azarmina and others added 30 commits July 22, 2026 21:39
Railway edge-case synthesis demo under implementations/edge_case_image_generation (configs, package, notebook, docs). Dataset payloads and model weights are gitignored.

Co-authored-by: Cursor <cursoragent@cursor.com>
Revert root .gitignore to match origin/main; dataset/output ignores remain in implementations/edge_case_image_generation/.gitignore.

Co-authored-by: Cursor <cursoragent@cursor.com>
Use CLIP-style photo-of queries, a lower score threshold with NMS, and seed paste anomaly masks so edge-case labels still appear.

Co-authored-by: Cursor <cursoragent@cursor.com>
Swap the model stack via Hydra (paste + SD 1.5 on CPU; SDXL inpaint/ControlNet + larger depth/seg/annotators on g2-standard-8).

Co-authored-by: Cursor <cursoragent@cursor.com>
Pass width/height into diffusion calls and resize before blending so L4 runs no longer mix 1024² outputs with source masks.

Co-authored-by: Cursor <cursoragent@cursor.com>
Replace fixed YAML x/y blobs with depth + cab-track corridor anchors, skip cartoon primes on inpaint, and upscale small frames before SDXL.

Co-authored-by: Cursor <cursoragent@cursor.com>
Qwen2.5-VL-3B on CPU (CLIP fallback) and Qwen2.5-VL-7B on L4 score RGB synthetics for accept/retry/reject.

Co-authored-by: Cursor <cursoragent@cursor.com>
Drop paste/cab-view hardcoding; CPU uses depth ControlNet and GPU keeps SDXL inpaint; slim data loaders and notebook.

Co-authored-by: Cursor <cursoragent@cursor.com>
Replace hard-to-read NEU-DET steel textures with open street-level road photos where damage and hazards are sparse, intuitive long-tail cases.

Co-authored-by: Cursor <cursoragent@cursor.com>
Add Notebook 1.5 comparing inpaint, dual ControlNet, and InstructPix2Pix on clean scene seeds (pothole, cone, fog), with CPU/L4 hardware configs and HF range-extract helper for the small Mapillary subset.

Co-authored-by: Cursor <cursoragent@cursor.com>
Extract script now rebuilds the EOCD Range index when missing so GPU boxes
do not need a checked-in 21MB jsonl; prepare_sample_images refuses Nordland
leftovers and can invoke the Mapillary toy extract when samples/ is empty.

Co-authored-by: Cursor <cursoragent@cursor.com>
Separate low controlnet_strength (~0.4) from inpaint strength, raise depth/seg
conditioning scales, and add fidelity prompts / instruct image-guidance so
comparisons stay on the same photo.

Co-authored-by: Cursor <cursoragent@cursor.com>
Transformers' Mask2FormerLoss imports scipy; without it section 2 fails
when hardware=gpu_l4 switches to facebook/mask2former-swin-small-ade.

Co-authored-by: Cursor <cursoragent@cursor.com>
Mask2Former still works if scipy is installed; otherwise section 2 failed
even after adding the dependency when the notebook kernel was not resynced.

Co-authored-by: Cursor <cursoragent@cursor.com>
Transformers caches scipy availability at import time, so %pip install alone
often still fails; auto-fallback unblocks gpu_l4 section 2 either way.

Co-authored-by: Cursor <cursoragent@cursor.com>
Stop prefer_near from collapsing edit holes into crescents, enlarge cone/pothole
masks, switch L4 instruct to SD1.5 InstructPix2Pix, and tighten SDXL ControlNet
denoise so dual edits stay closer to the source photo.

Co-authored-by: Cursor <cursoragent@cursor.com>
Learners choose METHOD_BY_ANOMALY from the comparison notebook, then run
load → edit → annotate → judge → retry without digging into each method.

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
SDXL was rewriting scenes; both hardware profiles now share SD1.5 inpaint with
clean ellipse masks, crop-to-mask, and short object prompts so local inserts
are honestly reliable. ControlNet/instruct stay as comparison baselines.

Co-authored-by: Cursor <cursoragent@cursor.com>
Asphalt potholes are nearly invisible in SD inpaint demos; a cardboard box
on the lane is an honest local-insert target. Also tone down cartoon cone prompts.

Co-authored-by: Cursor <cursoragent@cursor.com>
Dedicated inpaint models already preserve outside the mask; re-blending faded
inserts into smudges. Also enlarge holes, raise strength/steps/CFG, and widen
crop-to-mask padding.

Co-authored-by: Cursor <cursoragent@cursor.com>
Replace weak SD1.5 InstructPix2Pix/inpaint on gpu_l4 with Klein pipelines while keeping SD ControlNet for the depth+seg baseline.

Co-authored-by: Cursor <cursoragent@cursor.com>
Large FLUX.2-klein downloads often die with a masked Xet writer error; fall back to classic hub HTTP.

Co-authored-by: Cursor <cursoragent@cursor.com>
Drop the stale hardcoded pothole/ground_animal list so the compare study matches default.yaml.

Co-authored-by: Cursor <cursoragent@cursor.com>
Fixed ellipses were floating cones/debris; road_patch places the hole on near road pixels and prompts emphasize ground contact.

Co-authored-by: Cursor <cursoragent@cursor.com>
Move shared baselines under configs/default/, per-dataset sources and anomalies under configs/datasets/, and point samples/outputs at data/<dataset>/.

Co-authored-by: Cursor <cursoragent@cursor.com>
CPU hardware no longer sets a pruned null device; synthesize_one accepts seed_offset for judge retries.

Co-authored-by: Cursor <cursoragent@cursor.com>
Open-vocab labeling now uses yolov8s-worldv2 with tight per-anomaly queries; judge still validates bbox summaries without edit-mask seeding.

Co-authored-by: Cursor <cursoragent@cursor.com>
Encode class prompts on CPU before GPU predict, and run anomalies from METHOD_BY_ANOMALY keys instead of only dataset.workshop_anomalies.

Co-authored-by: Cursor <cursoragent@cursor.com>
Lower L4 threshold to 8.0 so overall=8 accepts; use scene-level fog queries with a lower annotation_conf; prefer instruct over brittle inpaint for Mapillary locals.

Co-authored-by: Cursor <cursoragent@cursor.com>
Arvin Azarmina and others added 2 commits July 29, 2026 14:32
High faithfulness/overall with edge_case_present=false no longer hard-rejects a clearly present cone.

Co-authored-by: Cursor <cursoragent@cursor.com>
Document fog as a manual-label teaching case; link stubs from README and NB1 wrap-up.

Co-authored-by: Cursor <cursoragent@cursor.com>
@arvinAM17
arvinAM17 requested a review from fatemetkl July 29, 2026 19:32
@coderabbitai

coderabbitai Bot commented Jul 29, 2026

Copy link
Copy Markdown

Review Change Stack

📝 Walkthrough

Walkthrough

This PR adds a complete edge-case image generation implementation with dataset-driven configuration, multiple image-editing methods, depth and segmentation conditioning, open-vocabulary annotation, VLM-based judging, retry handling, visualization utilities, notebooks, documentation, and environment setup scripts.

Changes

Edge-case image synthesis

Layer / File(s) Summary
Configuration and dataset ingestion
implementations/edge_case_image_generation/configs/*, src/edgecase_synthesis/config.py, src/edgecase_synthesis/data.py, pyproject.toml
Adds shared, hardware, dataset, anomaly, annotation, judge, and path configuration plus loaders for local, URL, VOC, Hugging Face detection, and Hugging Face row sources.
Mapillary extraction and environment setup
scripts/extract_mapillary_toy.py, scripts/setup_notebook_env.sh, scripts/diagnose_python.sh
Adds authenticated range-based Mapillary ZIP extraction, toy sample metadata generation, Python diagnostics, virtual-environment setup, and Jupyter kernel registration.
Conditioning and generation
src/edgecase_synthesis/conditioning.py, generation.py, compare_methods.py
Adds depth and segmentation inference, geometric anomaly masks, and inpaint, dual-ControlNet, and instruction-edit generation paths.
Annotation and judging
src/edgecase_synthesis/annotation.py, judge.py
Adds YOLO-World annotation results and overlays plus Qwen VLM and CLIP fallback judging with accept/retry/reject decisions.
Pipeline, notebooks, visualization, and docs
src/edgecase_synthesis/pipeline.py, viz.py, notebooks/*, docs/*, README.md
Connects method selection, synthesis, annotation, judging, retries, artifact persistence, notebook workflows, and implementation documentation.

Estimated code review effort: 5 (Critical) | ~120 minutes

Suggested labels: enhancement

Suggested reviewers: fatemetkl

Sequence Diagram(s)

sequenceDiagram
  participant Notebook
  participant DataLoader
  participant Conditioning
  participant MethodComparer
  participant Annotator
  participant VLMJudge

  Notebook->>DataLoader: prepare_sample_images()
  DataLoader-->>Notebook: sample images
  Notebook->>Conditioning: predict depth and segmentation
  Conditioning-->>Notebook: conditioning results
  Notebook->>MethodComparer: synthesize anomaly edit
  MethodComparer-->>Notebook: generated image
  Notebook->>Annotator: annotate generated image
  Annotator-->>Notebook: detections and overlay
  Notebook->>VLMJudge: judge image and annotations
  VLMJudge-->>Notebook: accept, retry, or reject decision
Loading
🚥 Pre-merge checks | ✅ 3 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Description check ⚠️ Warning The description covers summary and type of change, but it omits most required template sections. Add the missing sections: Changes Made, Testing, Screenshots/Recordings, Related Issues, Deployment Notes, and Checklist.
Docstring Coverage ⚠️ Warning Docstring coverage is 17.24% which is insufficient. The required threshold is 80.00%. Write docstrings for the functions missing them to satisfy the coverage threshold.
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title is relevant and accurately points to the notebook-focused edge case generation work.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches 💡 2
📝 Generate docstrings 💡
  • Create stacked PR
  • Commit on current branch
🛠️ Fix failing CI checks 💡
  • Fix failing CI checks

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 16

🧹 Nitpick comments (15)
implementations/edge_case_image_generation/notebooks/01.5_method_comparison.ipynb (2)

175-184: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low value

Add strict= to zip().

zip(scene_paths[: len(ANOMALIES)], ANOMALIES) silently truncates to the shorter iterable. Since the code already validates len(scene_paths) >= len(ANOMALIES) just above, this should be a non-issue in practice, but passing strict=True documents that intent and fails loudly if the invariant is ever violated.

As per static analysis hints, Ruff flags zip() without an explicit strict= parameter (B905).

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@implementations/edge_case_image_generation/notebooks/01.5_method_comparison.ipynb`
around lines 175 - 184, Update the jobs construction loop around scene_paths and
ANOMALIES to call zip with strict=True, preserving the existing iterable slicing
and job creation behavior while ensuring length mismatches fail loudly.

Source: Linters/SAST tools


44-49: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick win

Clear notebook outputs before committing.

The committed outputs leak the author's local filesystem path (/Users/arvinazarmina/Documents/Projects/..., line 47) and include a stray failed-download traceback with an "unsafe serialization" fallback warning (lines 371-374) from a broken local cache — neither is meaningful to reviewers and both bloat the diff. Consider stripping outputs (e.g., nbstripout or jupyter nbconvert --clear-output) before committing notebooks.

Also applies to: 371-377

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@implementations/edge_case_image_generation/notebooks/01.5_method_comparison.ipynb`
around lines 44 - 49, Clear all stored outputs from the notebook before
committing, including the Project root stdout containing the author’s local path
and the failed-download traceback and unsafe-serialization warning near the
later output block. Preserve the notebook’s code and markdown cells while
removing execution results and transient error output, using the repository’s
standard notebook-output cleanup tool if available.
implementations/edge_case_image_generation/src/edgecase_synthesis/annotation.py (2)

165-170: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

unload() leaves the object in an unusable state with no guard.

After unload(), any annotate() call fails with AttributeError: 'NoneType' object has no attribute 'set_classes'. A explicit check in _set_classes (raise RuntimeError("annotator unloaded; rebuild via from_config")) makes the notebook lifecycle mistake obvious.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@implementations/edge_case_image_generation/src/edgecase_synthesis/annotation.py`
around lines 165 - 170, Update _set_classes to detect when unload has cleared
self.model and raise RuntimeError("annotator unloaded; rebuild via from_config")
before accessing the model. Preserve normal class-setting behavior when the
annotator remains loaded.

69-72: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Silent except Exception: pass around device moves.

If .to("cpu") fails, the CLIP device-mismatch this guard exists to prevent resurfaces later as an opaque Expected all tensors to be on the same device. Log at debug/warning level so the cause is recoverable. Same pattern at Lines 153-161.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@implementations/edge_case_image_generation/src/edgecase_synthesis/annotation.py`
around lines 69 - 72, Update the exception handling around the device moves in
the annotation flow, including the corresponding block near the later model
transfer, to log the caught exception at debug or warning level instead of
silently passing. Preserve the existing fallback behavior while making failures
from self.model.to("cpu") recoverable in logs.

Source: Linters/SAST tools

implementations/edge_case_image_generation/src/edgecase_synthesis/generation.py (2)

334-364: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

padding_mask_crop support is probed by catching TypeError around a full diffusion call. A TypeError raised anywhere inside the denoise loop triggers a second complete inference pass (minutes of GPU time) and masks genuine bugs. Probe the pipeline signature once instead.

  • implementations/edge_case_image_generation/src/edgecase_synthesis/generation.py#L334-L364: replace both try/except blocks in _inpaint with a shared _supports_kwarg(pipe, "padding_mask_crop") check before calling the pipe.
  • implementations/edge_case_image_generation/src/edgecase_synthesis/compare_methods.py#L530-L563: apply the same helper in the Klein and SD branches of _run_inpaint.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@implementations/edge_case_image_generation/src/edgecase_synthesis/generation.py`
around lines 334 - 364, Replace the full-call TypeError fallbacks in _inpaint
with a shared _supports_kwarg(pipe, "padding_mask_crop") check, removing
padding_mask_crop before invocation when unsupported; apply the same change to
both Klein and SD branches of _run_inpaint in
implementations/edge_case_image_generation/src/edgecase_synthesis/compare_methods.py
(lines 530-563), while preserving all other inference arguments and behavior.

227-243: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Hardcoded strength clamp plus mismatched ControlNet-scale defaults.

Two config-contract wrinkles:

  • Line 227/231: scalar default is 0.55 but the dict path defaults to 0.75, so behavior silently shifts depending on whether YAML wrote a scalar or a mapping.
  • Line 243: np.clip(..., 0.25, 0.60) makes any configured controlnet_strength above 0.60 a no-op. compare_methods.py already exposes controlnet_strength_cap_* keys for this; reusing them here keeps the caps discoverable and overridable.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@implementations/edge_case_image_generation/src/edgecase_synthesis/generation.py`
around lines 227 - 243, The ControlNet configuration handling uses inconsistent
scale defaults and hardcodes the strength upper bound. Update the scale
extraction in the surrounding generation method so scalar and mapping
configurations share the same default, and replace the fixed np.clip upper limit
in the _controlnet call with the existing controlnet_strength_cap_*
configuration value used by compare_methods.py, while preserving the configured
lower bound and override behavior.
implementations/edge_case_image_generation/src/edgecase_synthesis/compare_methods.py (3)

141-213: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Pipeline plumbing duplicated verbatim from AnomalyEditor.

_is_klein_model/inpaint_is_klein, _dtype, _place, _gen, and _fit_klein_image are near-identical to generation.py Lines 82-106 / 427-447. Extract a small mixin or base class so device/dtype/offload policy changes apply to both runners.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@implementations/edge_case_image_generation/src/edgecase_synthesis/compare_methods.py`
around lines 141 - 213, The pipeline utility methods in the current runner
duplicate AnomalyEditor; extract the shared Klein/device helpers into a small
mixin or base class and have both runners reuse it. Move `_is_klein_model`, the
Klein model properties, `_dtype`, `_place`, `_gen`, and `_fit_klein_image` into
the shared abstraction, preserving their existing behavior so future device,
dtype, and offload policy changes apply consistently.

474-479: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Unconditional unload() per method forces repeated multi-GB model reloads.

_build_* already calls _free_other_edit_pipes(keep=...), so this extra unload() mainly guarantees the pipe just used is discarded and re-downloaded/re-deserialized on the next anomaly in the outer loop. Dropping it (and relying on _free_other_edit_pipes) keeps the same VRAM guarantee without the reload cost.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@implementations/edge_case_image_generation/src/edgecase_synthesis/compare_methods.py`
around lines 474 - 479, Remove the conditional self.unload() call from the
per-method loop in the method executing flow, while preserving the existing CUDA
and uses_klein checks only if needed elsewhere. Rely on the _build_* methods’
_free_other_edit_pipes(keep=...) behavior to maintain VRAM management without
discarding the current pipe between methods.

24-31: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Importing five private helpers across modules.

_depth_to_control_image, _ensure_same_size, _fit_for_diffusion, _mask_to_pil, and _composite are private to generation.py yet form the de-facto shared image-prep API. Move them into a small shared module (e.g. imaging.py) with public names so the contract is explicit and refactors in generation.py don't silently break this file.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@implementations/edge_case_image_generation/src/edgecase_synthesis/compare_methods.py`
around lines 24 - 31, Move the shared image-preparation helpers
_depth_to_control_image, _ensure_same_size, _fit_for_diffusion, _mask_to_pil,
and _composite from generation.py into a dedicated imaging module, rename them
as public helpers, and update both generation.py and compare_methods.py to
import and use the new symbols. Keep their existing behavior unchanged while
removing cross-module imports of generation.py’s private helpers.
implementations/edge_case_image_generation/src/edgecase_synthesis/judge.py (1)

322-343: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Dead if/pass branch, and the docstring numbers don't match the defaults.

  • Lines 331-333: the if result.edge_case_present and result.overall < 3.0: pass block is a no-op; keep it as a comment only.
  • Line 339: the worked example says threshold=8.0 while __init__/from_config default to 8.5, so the stated accept/reject bands (≥8 / <6) are off by 0.5.
  • Worth noting explicitly that looks_present overrides an explicit edge_case_present=false from the VLM whenever faithfulness ≥ 7, which turns hard rejects into accept/retry — intentional, but it should be config-gated rather than hardcoded.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@implementations/edge_case_image_generation/src/edgecase_synthesis/judge.py`
around lines 322 - 343, The reconciliation logic around looks_present should
remain, but remove the no-op result.edge_case_present/overall branch and
preserve its intent as a comment only. Update _decide’s docstring example to use
the actual default threshold of 8.5 and corresponding accept/reject bands. Make
the looks_present override of an explicit false edge_case_present
configuration-gated, using the existing configuration mechanism, while
preserving the current behavior when enabled.
implementations/edge_case_image_generation/src/edgecase_synthesis/conditioning.py (1)

337-353: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low value

Single-pixel argmax center can sit on the road boundary.

best is one pixel, so the ellipse frequently straddles the road edge. Averaging the top-k scored pixels (or taking the centroid of the best-scoring connected component) gives a more stable anchor.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@implementations/edge_case_image_generation/src/edgecase_synthesis/conditioning.py`
around lines 337 - 353, Update the center-selection logic after computing score
so it averages the coordinates of the top-scoring pixels, rather than using the
single index from np.argmax. Preserve the existing score calculation and ensure
cx/cy are derived from the selected top-k xs/ys coordinates, producing a stable
anchor within the road region.
implementations/edge_case_image_generation/src/edgecase_synthesis/config.py (1)

124-124: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

PROJECT_ROOT env override is silently discarded. paths.yaml defines project_root: ${oc.env:PROJECT_ROOT,.} and data.py:55-58 tells users to "set PROJECT_ROOT", but this line always replaces it with the discovered root. Honor the env value when present.

♻️ Suggested change
-    cfg.paths.project_root = str(root)
+    env_root = os.environ.get("PROJECT_ROOT")
+    cfg.paths.project_root = str(Path(env_root).resolve() if env_root else root)

Add import os at the top of the module.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@implementations/edge_case_image_generation/src/edgecase_synthesis/config.py`
at line 124, Update the configuration initialization around
cfg.paths.project_root to honor the PROJECT_ROOT environment variable when it is
set, while retaining the discovered root as the fallback. Add the required os
import and only assign the discovered root when no PROJECT_ROOT override is
present.
implementations/edge_case_image_generation/src/edgecase_synthesis/data.py (1)

424-429: 🔒 Security & Privacy | 🔵 Trivial | 💤 Low value

Consider defusedxml for archive-supplied XML. VOC annotations come from a downloaded zip; stdlib ElementTree is vulnerable to entity-expansion attacks (Ruff S314). from defusedxml.ElementTree import parse is a drop-in replacement.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@implementations/edge_case_image_generation/src/edgecase_synthesis/data.py`
around lines 424 - 429, Update _parse_voc_xml to use
defusedxml.ElementTree.parse instead of xml.etree.ElementTree.parse when reading
archive-supplied VOC annotations, preserving the existing path checks and root
extraction behavior.

Source: Linters/SAST tools

implementations/edge_case_image_generation/scripts/extract_mapillary_toy.py (1)

26-40: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Module import performs config I/O. load_config() at import time means the module cannot be imported (for reuse or testing) without a resolvable config tree. Moving this into main() or a small _settings() helper keeps import side-effect free.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@implementations/edge_case_image_generation/scripts/extract_mapillary_toy.py`
around lines 26 - 40, Remove the module-level load_config() call and derived
configuration values in extract_mapillary_toy.py. Move this setup into main() or
a small _settings() helper, and have the extraction flow obtain those settings
explicitly so importing the module performs no configuration I/O while
preserving the existing defaults and mappings.
implementations/edge_case_image_generation/scripts/setup_notebook_env.sh (1)

11-19: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Both setup scripts hardcode an Apple-Silicon Homebrew python3.12 instead of the project's declared Python version. This breaks Intel macOS (/usr/local/bin) and Linux GPU hosts — which the hardware/gpu_l4 profile targets — and makes the diagnostic disagree with the venv the setup script actually creates.

  • implementations/edge_case_image_generation/scripts/setup_notebook_env.sh#L11-L19: fall back to uv venv --python "$(cat .python-version)" (or command -v python3.12) when the Homebrew path is absent, instead of exit 1.
  • implementations/edge_case_image_generation/scripts/diagnose_python.sh#L20-L25: probe $ROOT/.venv/bin/python (the path setup_notebook_env.sh prints) rather than the version-suffixed python3.12.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@implementations/edge_case_image_generation/scripts/setup_notebook_env.sh`
around lines 11 - 19, The setup flow hardcodes an Apple-Silicon Homebrew Python
path and the diagnostic probes a different interpreter. In
implementations/edge_case_image_generation/scripts/setup_notebook_env.sh lines
11-19, fall back to the project’s .python-version via uv venv, or command -v
python3.12, when BREW_PYTHON is unavailable instead of exiting; in
implementations/edge_case_image_generation/scripts/diagnose_python.sh lines
20-25, probe $ROOT/.venv/bin/python to match the environment created by
setup_notebook_env.sh.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@implementations/edge_case_image_generation/configs/datasets/rdd2022/data.yaml`:
- Around line 18-22: Update prefer_classes in the dataset configuration to use
the normalized class names defined by class_map and stem_prefixes—pothole,
alligator_crack, transverse_crack, and longitudinal_crack—in the intended
priority order, so load_sample_images() matches the extracted filename prefixes.

In `@implementations/edge_case_image_generation/docs/context.md`:
- Around line 25-34: Align the documentation with the implemented YOLO-World
box-annotation contract: in
implementations/edge_case_image_generation/docs/context.md lines 25-34, replace
Grounded-SAM 2 with YOLO-World box annotation; in lines 262-283, describe box
outputs and manual or image-level labeling for diffuse effects rather than
guaranteed masks and no manual annotation; in
implementations/edge_case_image_generation/docs/project_summary.md line 11,
replace the Grounded-SAM-2 workflow with the configured YOLO-World path; and in
implementations/edge_case_image_generation/docs/quality_measure.md lines
152-159, replace SAM-mask IoU guidance with a box-compatible annotation-quality
check.

In `@implementations/edge_case_image_generation/docs/project_summary.md`:
- Around line 26-40: Update the reference list in the project summary to use
standard Markdown link syntax without literal ++ wrappers. Correct the malformed
SDXL-Turbo URL so it points to the verified stabilityai/sdxl-turbo repository,
and validate all listed reference links before publishing.

In
`@implementations/edge_case_image_generation/notebooks/01_sample_data_generation.ipynb`:
- Around line 353-367: Persist each retried current result and its refreshed
annotations under an attempt-specific output directory before the _unload call.
Update the retry flow around synthesize_one, annotate_anomaly, and
results[anomaly_id] to save both the image and label artifacts for every
attempt, including accepted retries.

In `@implementations/edge_case_image_generation/pyproject.toml`:
- Around line 22-24: Update the project dependencies in pyproject.toml: replace
the bare clip requirement with a direct PEP 508 URL dependency pointing to the
intended Ultralytics fork so built metadata installs the correct package, and
remove ipykernel from the runtime dependency list, retaining it only in the
existing extras/dev group.

In `@implementations/edge_case_image_generation/scripts/extract_mapillary_toy.py`:
- Around line 286-288: Update the sample cleanup around SAMPLES.mkdir so it does
not unconditionally delete every JPG/PNG in the user-managed samples directory.
Gate cleanup behind an explicit reset flag, or restrict deletion to image files
identified by this script’s toy_meta.json/source_meta.json metadata, preserving
user-provided files and prior samples when no reset is requested.
- Around line 57-65: Update _zip_size and its HEAD call site to prevent the
Hugging Face bearer token from being forwarded across redirects. Resolve the
final URL using Hugging Face’s session/token handling, then issue redirected
requests without Authorization unless the destination remains the same trusted
host, while preserving the existing size fallback behavior.
- Line 340: Update the label-key construction in load_detection_labels so each
labels_out key matches the corresponding ImageSample.name derived from
path.stem; use the stem-based sample name rather than prefixing it with the tag,
while preserving the existing .jpg naming convention.

In
`@implementations/edge_case_image_generation/src/edgecase_synthesis/annotation.py`:
- Around line 191-204: Update _resolve_yolo_weights so it completes the
parent-directory search for the requested weights name before checking any
yolov8s-worldv2.pt fallback. Only perform the fallback search after no requested
file is found, while preserving the existing direct-path resolution and final
detector_model return behavior.

In
`@implementations/edge_case_image_generation/src/edgecase_synthesis/compare_methods.py`:
- Around line 727-752: Normalize scalar and mapping controlnet_scale values
through a shared _as_scale_map(value, *, depth_default, seg_default) helper. In
compare_methods.py lines 727-752, use the normalized map before reading depth
and seg values; in compare_methods.py lines 390-396, handle scalar values
instead of falling back to instance defaults; in generation.py lines 227-243,
reuse the helper and align scalar and mapping defaults, including the current
0.55 versus 0.75 mismatch.
- Around line 390-396: Update the controlnet scale handling around
merged["controlnet_scale"] so scalar values are interpreted consistently with
generation.py rather than silently replaced by instance defaults. Preserve
mapping support for separate depth and seg values, and either apply the scalar
to both runners or explicitly reject it with a clear error.

In
`@implementations/edge_case_image_generation/src/edgecase_synthesis/conditioning.py`:
- Around line 368-373: Update the segmentation handling around the seg_class_ids
branch to apply np.isin(labels, list(class_ids)) before resizing, rather than
resizing the int32 label_map. Resize the resulting boolean support mask when its
shape differs from (height, width), then return it as a boolean mask while
preserving the existing no-class and missing-label-map behavior.

In `@implementations/edge_case_image_generation/src/edgecase_synthesis/data.py`:
- Around line 550-555: Validate the bbox values in the objects-list branch
before calling _box_from_any: normalize or inspect the candidate bbox and skip
entries with fewer than four usable values, matching the existing annot branch
behavior. Apply the same guard to the additional bbox handling around the
corresponding branch, while preserving valid box processing and preventing
malformed entries from aborting ingestion.
- Around line 344-345: Update the archive extraction flow around zipfile.ZipFile
and extractall to validate every member path before extraction, rejecting
absolute paths and normalized paths that escape extract_dir (including traversal
via symlinks if applicable). Only call extractall after all entries pass
validation, preserving extraction for safe archives.

In `@implementations/edge_case_image_generation/src/edgecase_synthesis/judge.py`:
- Around line 109-124: Update the model-loading flow using AutoProcessor and
AutoModelForImageTextToText to remove trust_remote_code=True from both calls,
relying on native Qwen2.5-VL support. Replace the dtype_kw indirection by
passing self.torch_dtype directly to the model loader, while preserving the
existing device_map and pixel configuration.
- Around line 132-143: Keep the configured self.model_id immutable in
_ensure_clip; record the fallback model in a separate _active_model_id instead.
Initialize _active_model_id to None in __init__, clear it in unload(), and have
judge() report _active_model_id when set, otherwise self.model_id, so
_ensure_qwen continues loading the configured model.

---

Nitpick comments:
In
`@implementations/edge_case_image_generation/notebooks/01.5_method_comparison.ipynb`:
- Around line 175-184: Update the jobs construction loop around scene_paths and
ANOMALIES to call zip with strict=True, preserving the existing iterable slicing
and job creation behavior while ensuring length mismatches fail loudly.
- Around line 44-49: Clear all stored outputs from the notebook before
committing, including the Project root stdout containing the author’s local path
and the failed-download traceback and unsafe-serialization warning near the
later output block. Preserve the notebook’s code and markdown cells while
removing execution results and transient error output, using the repository’s
standard notebook-output cleanup tool if available.

In `@implementations/edge_case_image_generation/scripts/extract_mapillary_toy.py`:
- Around line 26-40: Remove the module-level load_config() call and derived
configuration values in extract_mapillary_toy.py. Move this setup into main() or
a small _settings() helper, and have the extraction flow obtain those settings
explicitly so importing the module performs no configuration I/O while
preserving the existing defaults and mappings.

In `@implementations/edge_case_image_generation/scripts/setup_notebook_env.sh`:
- Around line 11-19: The setup flow hardcodes an Apple-Silicon Homebrew Python
path and the diagnostic probes a different interpreter. In
implementations/edge_case_image_generation/scripts/setup_notebook_env.sh lines
11-19, fall back to the project’s .python-version via uv venv, or command -v
python3.12, when BREW_PYTHON is unavailable instead of exiting; in
implementations/edge_case_image_generation/scripts/diagnose_python.sh lines
20-25, probe $ROOT/.venv/bin/python to match the environment created by
setup_notebook_env.sh.

In
`@implementations/edge_case_image_generation/src/edgecase_synthesis/annotation.py`:
- Around line 165-170: Update _set_classes to detect when unload has cleared
self.model and raise RuntimeError("annotator unloaded; rebuild via from_config")
before accessing the model. Preserve normal class-setting behavior when the
annotator remains loaded.
- Around line 69-72: Update the exception handling around the device moves in
the annotation flow, including the corresponding block near the later model
transfer, to log the caught exception at debug or warning level instead of
silently passing. Preserve the existing fallback behavior while making failures
from self.model.to("cpu") recoverable in logs.

In
`@implementations/edge_case_image_generation/src/edgecase_synthesis/compare_methods.py`:
- Around line 141-213: The pipeline utility methods in the current runner
duplicate AnomalyEditor; extract the shared Klein/device helpers into a small
mixin or base class and have both runners reuse it. Move `_is_klein_model`, the
Klein model properties, `_dtype`, `_place`, `_gen`, and `_fit_klein_image` into
the shared abstraction, preserving their existing behavior so future device,
dtype, and offload policy changes apply consistently.
- Around line 474-479: Remove the conditional self.unload() call from the
per-method loop in the method executing flow, while preserving the existing CUDA
and uses_klein checks only if needed elsewhere. Rely on the _build_* methods’
_free_other_edit_pipes(keep=...) behavior to maintain VRAM management without
discarding the current pipe between methods.
- Around line 24-31: Move the shared image-preparation helpers
_depth_to_control_image, _ensure_same_size, _fit_for_diffusion, _mask_to_pil,
and _composite from generation.py into a dedicated imaging module, rename them
as public helpers, and update both generation.py and compare_methods.py to
import and use the new symbols. Keep their existing behavior unchanged while
removing cross-module imports of generation.py’s private helpers.

In
`@implementations/edge_case_image_generation/src/edgecase_synthesis/conditioning.py`:
- Around line 337-353: Update the center-selection logic after computing score
so it averages the coordinates of the top-scoring pixels, rather than using the
single index from np.argmax. Preserve the existing score calculation and ensure
cx/cy are derived from the selected top-k xs/ys coordinates, producing a stable
anchor within the road region.

In `@implementations/edge_case_image_generation/src/edgecase_synthesis/config.py`:
- Line 124: Update the configuration initialization around
cfg.paths.project_root to honor the PROJECT_ROOT environment variable when it is
set, while retaining the discovered root as the fallback. Add the required os
import and only assign the discovered root when no PROJECT_ROOT override is
present.

In `@implementations/edge_case_image_generation/src/edgecase_synthesis/data.py`:
- Around line 424-429: Update _parse_voc_xml to use defusedxml.ElementTree.parse
instead of xml.etree.ElementTree.parse when reading archive-supplied VOC
annotations, preserving the existing path checks and root extraction behavior.

In
`@implementations/edge_case_image_generation/src/edgecase_synthesis/generation.py`:
- Around line 334-364: Replace the full-call TypeError fallbacks in _inpaint
with a shared _supports_kwarg(pipe, "padding_mask_crop") check, removing
padding_mask_crop before invocation when unsupported; apply the same change to
both Klein and SD branches of _run_inpaint in
implementations/edge_case_image_generation/src/edgecase_synthesis/compare_methods.py
(lines 530-563), while preserving all other inference arguments and behavior.
- Around line 227-243: The ControlNet configuration handling uses inconsistent
scale defaults and hardcodes the strength upper bound. Update the scale
extraction in the surrounding generation method so scalar and mapping
configurations share the same default, and replace the fixed np.clip upper limit
in the _controlnet call with the existing controlnet_strength_cap_*
configuration value used by compare_methods.py, while preserving the configured
lower bound and override behavior.

In `@implementations/edge_case_image_generation/src/edgecase_synthesis/judge.py`:
- Around line 322-343: The reconciliation logic around looks_present should
remain, but remove the no-op result.edge_case_present/overall branch and
preserve its intent as a comment only. Update _decide’s docstring example to use
the actual default threshold of 8.5 and corresponding accept/reject bands. Make
the looks_present override of an explicit false edge_case_present
configuration-gated, using the existing configuration mechanism, while
preserving the current behavior when enabled.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro Plus

Run ID: da316329-cb1d-4bc1-ba9e-1ae535d33fe3

📥 Commits

Reviewing files that changed from the base of the PR and between 6ae91c9 and be1ddf1.

⛔ Files ignored due to path filters (1)
  • implementations/edge_case_image_generation/uv.lock is excluded by !**/*.lock
📒 Files selected for processing (71)
  • implementations/edge_case_image_generation/.gitignore
  • implementations/edge_case_image_generation/.python-version
  • implementations/edge_case_image_generation/README.md
  • implementations/edge_case_image_generation/configs/config.yaml
  • implementations/edge_case_image_generation/configs/datasets/_template/annotation.yaml
  • implementations/edge_case_image_generation/configs/datasets/_template/data.yaml
  • implementations/edge_case_image_generation/configs/datasets/_template/dataset.yaml
  • implementations/edge_case_image_generation/configs/datasets/_template/generation/anomalies/example_anomaly.yaml
  • implementations/edge_case_image_generation/configs/datasets/_template/generation/default.yaml
  • implementations/edge_case_image_generation/configs/datasets/mapillary_vistas/annotation.yaml
  • implementations/edge_case_image_generation/configs/datasets/mapillary_vistas/data.yaml
  • implementations/edge_case_image_generation/configs/datasets/mapillary_vistas/dataset.yaml
  • implementations/edge_case_image_generation/configs/datasets/mapillary_vistas/generation/anomalies/fog.yaml
  • implementations/edge_case_image_generation/configs/datasets/mapillary_vistas/generation/anomalies/ground_animal.yaml
  • implementations/edge_case_image_generation/configs/datasets/mapillary_vistas/generation/anomalies/pothole.yaml
  • implementations/edge_case_image_generation/configs/datasets/mapillary_vistas/generation/anomalies/road_debris.yaml
  • implementations/edge_case_image_generation/configs/datasets/mapillary_vistas/generation/anomalies/traffic_cone.yaml
  • implementations/edge_case_image_generation/configs/datasets/mapillary_vistas/generation/default.yaml
  • implementations/edge_case_image_generation/configs/datasets/nordland_hf/annotation.yaml
  • implementations/edge_case_image_generation/configs/datasets/nordland_hf/data.yaml
  • implementations/edge_case_image_generation/configs/datasets/nordland_hf/dataset.yaml
  • implementations/edge_case_image_generation/configs/datasets/nordland_hf/generation/anomalies/dusk.yaml
  • implementations/edge_case_image_generation/configs/datasets/nordland_hf/generation/anomalies/fog.yaml
  • implementations/edge_case_image_generation/configs/datasets/nordland_hf/generation/anomalies/snow.yaml
  • implementations/edge_case_image_generation/configs/datasets/nordland_hf/generation/default.yaml
  • implementations/edge_case_image_generation/configs/datasets/rdd2022/annotation.yaml
  • implementations/edge_case_image_generation/configs/datasets/rdd2022/data.yaml
  • implementations/edge_case_image_generation/configs/datasets/rdd2022/dataset.yaml
  • implementations/edge_case_image_generation/configs/datasets/rdd2022/generation/anomalies/alligator_crack.yaml
  • implementations/edge_case_image_generation/configs/datasets/rdd2022/generation/anomalies/pothole.yaml
  • implementations/edge_case_image_generation/configs/datasets/rdd2022/generation/anomalies/traffic_cone.yaml
  • implementations/edge_case_image_generation/configs/datasets/rdd2022/generation/default.yaml
  • implementations/edge_case_image_generation/configs/default/annotation.yaml
  • implementations/edge_case_image_generation/configs/default/conditioning.yaml
  • implementations/edge_case_image_generation/configs/default/generation.yaml
  • implementations/edge_case_image_generation/configs/default/judge.yaml
  • implementations/edge_case_image_generation/configs/default/paths.yaml
  • implementations/edge_case_image_generation/configs/hardware/cpu.yaml
  • implementations/edge_case_image_generation/configs/hardware/gpu_l4.yaml
  • implementations/edge_case_image_generation/data/l4r_nlb/archives/.gitkeep
  • implementations/edge_case_image_generation/data/mapillary_vistas/.gitkeep
  • implementations/edge_case_image_generation/data/mapillary_vistas/samples/.gitkeep
  • implementations/edge_case_image_generation/data/neu_det/.gitkeep
  • implementations/edge_case_image_generation/data/nordland_hf/.gitkeep
  • implementations/edge_case_image_generation/data/nordland_hf/samples/.gitkeep
  • implementations/edge_case_image_generation/data/rdd2022/.gitkeep
  • implementations/edge_case_image_generation/data/rdd2022/samples/.gitkeep
  • implementations/edge_case_image_generation/docs/context.md
  • implementations/edge_case_image_generation/docs/models_and_science.md
  • implementations/edge_case_image_generation/docs/potential_datasets.md
  • implementations/edge_case_image_generation/docs/project_summary.md
  • implementations/edge_case_image_generation/docs/quality_measure.md
  • implementations/edge_case_image_generation/notebooks/01.5_method_comparison.ipynb
  • implementations/edge_case_image_generation/notebooks/01_sample_data_generation.ipynb
  • implementations/edge_case_image_generation/notebooks/02_batch_dataset_generation.ipynb
  • implementations/edge_case_image_generation/notebooks/03_training_and_evaluation.ipynb
  • implementations/edge_case_image_generation/outputs/.gitkeep
  • implementations/edge_case_image_generation/pyproject.toml
  • implementations/edge_case_image_generation/scripts/diagnose_python.sh
  • implementations/edge_case_image_generation/scripts/extract_mapillary_toy.py
  • implementations/edge_case_image_generation/scripts/setup_notebook_env.sh
  • implementations/edge_case_image_generation/src/edgecase_synthesis/__init__.py
  • implementations/edge_case_image_generation/src/edgecase_synthesis/annotation.py
  • implementations/edge_case_image_generation/src/edgecase_synthesis/compare_methods.py
  • implementations/edge_case_image_generation/src/edgecase_synthesis/conditioning.py
  • implementations/edge_case_image_generation/src/edgecase_synthesis/config.py
  • implementations/edge_case_image_generation/src/edgecase_synthesis/data.py
  • implementations/edge_case_image_generation/src/edgecase_synthesis/generation.py
  • implementations/edge_case_image_generation/src/edgecase_synthesis/judge.py
  • implementations/edge_case_image_generation/src/edgecase_synthesis/pipeline.py
  • implementations/edge_case_image_generation/src/edgecase_synthesis/viz.py

Comment on lines +18 to +22
prefer_classes:
- D40
- D20
- D10
- D00

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Use normalized class names for sample preference ordering.

load_sample_images() passes prefer_classes directly into path-based selection, but this file supplies raw source IDs while its stem_prefixes and class_map use normalized names. Unless extracted filenames retain D40/D20/D10/D00, the preference ordering will not match and the eight selected samples may not prioritize the intended rare classes. Use pothole, alligator_crack, transverse_crack, and longitudinal_crack, or translate IDs before selection.

This follows the supplied load_sample_images contract and this file’s class_map/stem_prefixes.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@implementations/edge_case_image_generation/configs/datasets/rdd2022/data.yaml`
around lines 18 - 22, Update prefer_classes in the dataset configuration to use
the normalized class names defined by class_map and stem_prefixes—pothole,
alligator_crack, transverse_crack, and longitudinal_crack—in the intended
priority order, so load_sample_images() matches the extracted filename prefixes.

Comment on lines +25 to +34
Controlled Image Editing
(ControlNet + Diffusion)
Automatic Annotation
(Grounded SAM 2)
Quality Verification
(VLM Judge)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Align the documentation with the implemented annotation contract.

The current workflow uses YOLO-World bounding boxes and Notebook 1 explicitly states “no SAM”; these documents instead prescribe Grounded-SAM 2 masks. This directs learners toward outputs the pipeline does not produce.

  • implementations/edge_case_image_generation/docs/context.md#L25-L34: replace Grounded-SAM 2 in the pipeline diagram with YOLO-World box annotation.
  • implementations/edge_case_image_generation/docs/context.md#L262-L283: describe box outputs and manual/image-level labeling for diffuse effects instead of guaranteed masks and no manual annotation.
  • implementations/edge_case_image_generation/docs/project_summary.md#L11-L11: replace the Grounded-SAM-2 workflow description with the configured YOLO-World annotation path.
  • implementations/edge_case_image_generation/docs/quality_measure.md#L152-L159: replace SAM-mask IoU guidance with a box-compatible annotation-quality check.
📍 Affects 3 files
  • implementations/edge_case_image_generation/docs/context.md#L25-L34 (this comment)
  • implementations/edge_case_image_generation/docs/context.md#L262-L283
  • implementations/edge_case_image_generation/docs/project_summary.md#L11-L11
  • implementations/edge_case_image_generation/docs/quality_measure.md#L152-L159
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@implementations/edge_case_image_generation/docs/context.md` around lines 25 -
34, Align the documentation with the implemented YOLO-World box-annotation
contract: in implementations/edge_case_image_generation/docs/context.md lines
25-34, replace Grounded-SAM 2 with YOLO-World box annotation; in lines 262-283,
describe box outputs and manual or image-level labeling for diffuse effects
rather than guaranteed masks and no manual annotation; in
implementations/edge_case_image_generation/docs/project_summary.md line 11,
replace the Grounded-SAM-2 workflow with the configured YOLO-World path; and in
implementations/edge_case_image_generation/docs/quality_measure.md lines
152-159, replace SAM-mask IoU guidance with a box-compatible annotation-quality
check.

Comment on lines +26 to +40
Diffusers + ControlNet: ++[https://github.com/huggingface/diffusers](https://github.com/huggingface/diffusers)++

FLUX.1: ++[https://github.com/black-forest-labs/flux](https://github.com/black-forest-labs/flux)++

SDXL-Turbo: ++[https://huggingfGitHub - black-forest-labs/flux: Official inference repo for FLUX.1 modelsace.co/stabilityai/sdxl-turbo](https://huggingfGitHub)++

Grounded-SAM-2: ++[https://github.com/IDEA-Research/Grounded-SAM-2](https://github.com/IDEA-Research/Grounded-SAM-2)++

Depth-Anything-V2: ++[https://github.com/DepthAnything/Depth-Anything-V2](https://github.com/DepthAnything/Depth-Anything-V2)++

VLM judges: ++[https://github.com/QwenLM/Qwen2.5-VL](https://github.com/QwenLM/Qwen2.5-VL)++ ++[https://github.com/OpenGVLab/InternVL](https://github.com/OpenGVLab/InternVL)++

DriveLM / SkyScenes / DiffusionEngine show closely related patterns: ++[https://github.com/OpenDriveLab/DriveLM](https://github.com/OpenDriveLab/DriveLM)++

++[https://github.com/bravegroup/DiffusionEngine](https://github.com/bravegroup/DiffusionEngine)++

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win

Repair the reference links.

The ++ wrappers render literally in standard Markdown, and the SDXL-Turbo target on Line 30 is malformed. Replace these with verified standard Markdown links before publishing.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@implementations/edge_case_image_generation/docs/project_summary.md` around
lines 26 - 40, Update the reference list in the project summary to use standard
Markdown link syntax without literal ++ wrappers. Correct the malformed
SDXL-Turbo URL so it points to the verified stabilityai/sdxl-turbo repository,
and validate all listed reference links before publishing.

Comment on lines +353 to +367
" current = synthesize_one(\n",
" sample.image,\n",
" anomaly_id=anomaly_id,\n",
" method=method_map[anomaly_id],\n",
" cfg=cfg,\n",
" comparer=comparer,\n",
" depth=depth,\n",
" segmentation=seg,\n",
" project_root=PROJECT_ROOT,\n",
" seed_offset=attempt,\n",
" )\n",
" # Re-annotate the new edit so the next judge call sees fresh labels.\n",
" annotations[anomaly_id] = annotate_anomaly(current, anomaly_id, annotator)\n",
" results[anomaly_id] = current\n",
" _unload(\"comparer\", \"annotator\", \"depth_model\", \"segmenter\")\n",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

Persist retry artifacts.

A retried current result is judged but never saved with its refreshed annotation, so an accepted retry has no corresponding image/label artifact on disk. Save both under an attempt-specific directory before unloading the models.

Proposed fix
         annotations[anomaly_id] = annotate_anomaly(current, anomaly_id, annotator)
+        retry_dir = output_dir / "nb1" / anomaly_id / f"attempt_{attempt}"
+        print(save_generation_artifact(sample, current.generated, retry_dir))
+        print(
+            save_annotation_artifact(
+                f"{sample.name}_{anomaly_id}",
+                annotations[anomaly_id],
+                retry_dir,
+            )
+        )
         results[anomaly_id] = current
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
" current = synthesize_one(\n",
" sample.image,\n",
" anomaly_id=anomaly_id,\n",
" method=method_map[anomaly_id],\n",
" cfg=cfg,\n",
" comparer=comparer,\n",
" depth=depth,\n",
" segmentation=seg,\n",
" project_root=PROJECT_ROOT,\n",
" seed_offset=attempt,\n",
" )\n",
" # Re-annotate the new edit so the next judge call sees fresh labels.\n",
" annotations[anomaly_id] = annotate_anomaly(current, anomaly_id, annotator)\n",
" results[anomaly_id] = current\n",
" _unload(\"comparer\", \"annotator\", \"depth_model\", \"segmenter\")\n",
" current = synthesize_one(\n",
" sample.image,\n",
" anomaly_id=anomaly_id,\n",
" method=method_map[anomaly_id],\n",
" cfg=cfg,\n",
" comparer=comparer,\n",
" depth=depth,\n",
" segmentation=seg,\n",
" project_root=PROJECT_ROOT,\n",
" seed_offset=attempt,\n",
" )\n",
" # Re-annotate the new edit so the next judge call sees fresh labels.\n",
" annotations[anomaly_id] = annotate_anomaly(current, anomaly_id, annotator)\n",
" retry_dir = output_dir / \"nb1\" / anomaly_id / f\"attempt_{attempt}\"\n",
" print(save_generation_artifact(sample, current.generated, retry_dir))\n",
" print(\n",
" save_annotation_artifact(\n",
" f\"{sample.name}_{anomaly_id}\",\n",
" annotations[anomaly_id],\n",
" retry_dir,\n",
" )\n",
" )\n",
" results[anomaly_id] = current\n",
" _unload(\"comparer\", \"annotator\", \"depth_model\", \"segmenter\")\n",
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@implementations/edge_case_image_generation/notebooks/01_sample_data_generation.ipynb`
around lines 353 - 367, Persist each retried current result and its refreshed
annotations under an attempt-specific output directory before the _unload call.
Update the retry flow around synthesize_one, annotate_anomaly, and
results[anomaly_id] to save both the image and label artifacts for every
attempt, including accepted retries.

Comment on lines +22 to +24
"ipykernel>=7.3.0",
"clip",
]

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win

clip only resolves correctly under uv, and ipykernel is a runtime dependency. [tool.uv.sources] is not part of published metadata, so pip install . (or anyone consuming the built wheel) will pull the unrelated clip package from PyPI instead of the Ultralytics fork. Prefer a direct PEP 508 URL so the pin travels with the metadata, and move the notebook kernel to the extras/dev group where it already exists.

🔧 Proposed dependency fix
     "requests>=2.31",
-    "ipykernel>=7.3.0",
-    "clip",
+    "clip @ git+https://github.com/ultralytics/CLIP.git",
 ]
-[tool.uv.sources]
-clip = { git = "https://github.com/ultralytics/CLIP.git" }
-

Also applies to: 36-37

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@implementations/edge_case_image_generation/pyproject.toml` around lines 22 -
24, Update the project dependencies in pyproject.toml: replace the bare clip
requirement with a direct PEP 508 URL dependency pointing to the intended
Ultralytics fork so built metadata installs the correct package, and remove
ipykernel from the runtime dependency list, retaining it only in the existing
extras/dev group.

Comment on lines +368 to +373
class_ids = cfg.get("seg_class_ids")
if class_ids and segmentation.label_map is not None:
labels = segmentation.label_map
if labels.shape != (height, width):
labels = cv2.resize(labels.astype(np.int32), (width, height), interpolation=cv2.INTER_NEAREST)
return np.isin(labels, list(class_ids))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🔴 Critical | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Confirm seg_class_ids is actually used by shipped anomaly configs.
fd -e yaml . implementations/edge_case_image_generation/configs --exec rg -n 'seg_class_ids|mode:\s*(seg_intersection|road_patch)|intersect_seg' {} +

Repository: VectorInstitute/synthetic-data-bootcamp

Length of output: 3041


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "Files:"
git ls-files | rg 'implementations/edge_case_image_generation/(src/edgecase_synthesis/conditioning.py|configs/.*\.yaml)$' | sed -n '1,120p'

echo
echo "conditioning outline:"
ast-grep outline implementations/edge_case_image_generation/src/edgecase_synthesis/conditioning.py --view compact | sed -n '1,160p' || true

echo
echo "conditioning lines 330-390:"
cat -n implementations/edge_case_image_generation/src/edgecase_synthesis/conditioning.py | sed -n '330,390p'

echo
echo "conditioning lines 200-330:"
cat -n implementations/edge_case_image_generation/src/edgecase_synthesis/conditioning.py | sed -n '200,330p'

echo
echo "call sites / annotations:"
rg -n 'build_anomaly_edit_mask|Anomaly|conditioning|seg_class_ids|road_patch|intersect_seg|label_map_path|label_map:' implementations/edge_case_image_generation --glob '*.py' --glob '*.yaml'

Repository: VectorInstitute/synthetic-data-bootcamp

Length of output: 19320


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "default conditioning:"
cat -n implementations/edge_case_image_generation/configs/default/conditioning.yaml

echo
echo "dataset conditioning references:"
for f in implementations/edge_case_image_generation/configs/datasets/*/default.yaml implementations/edge_case_image_generation/configs/default/generation.yaml implementations/edge_case_image_generation/configs/config.yaml; do
  [ -f "$f" ] || continue
  echo "--- $f"
  sed -n '1,220p' "$f"
done

echo
echo "generation call context:"
cat -n implementations/edge_case_image_generation/src/edgecase_synthesis/generation.py | sed -n '150,230p'

echo
echo "label map load paths and config snippets:"
cat -n implementations/edge_case_image_generation/src/edgecase_synthesis/conditioning.py | sed -n '160,215p'
for f in implementations/edge_case_image_generation/configs/datasets/*/data.yaml; do
  echo "--- $f"
  sed -n '1,80p' "$f"
done

echo
echo "OpenCV dtype support probe:"
python3 - <<'PY'
try:
    import cv2, numpy as np
    from collections import deque
    vals = []
    for dt in [np.uint8, np.int8, np.uint16, np.int16, np.uint32, np.int32, np.float32, np.float64]:
        try:
            a = np.zeros((10, 10), dtype=dt)
            cv2.resize(a, (5, 5))
            vals.append(f"{dt.__name__}: ok")
        except Exception as e:
            vals.append(f"{dt.__name__}: {type(e).__name__}: {str(e).splitlines()[0]}")
    print("\n".join(vals))
except Exception as e:
    print(f"probe_unavailable: {type(e).__name__}: {e}")
PY

Repository: VectorInstitute/synthetic-data-bootcamp

Length of output: 12156


🌐 Web query:

OpenCV cv2.resize supported data types CV_8U CV_16U CV_16S CV_32F CV_64F CV_32S error code src data type = 4 is not supported

💡 Result:

The error src data type = 4 is not supported in cv2.resize arises because OpenCV does not support 32-bit signed integer (CV_32S) data types for the resize operation [1]. In OpenCV, matrix types are defined by a depth (the data type of the elements) and the number of channels. The depth codes and their corresponding numeric identifiers are defined in the OpenCV interface header [2]: CV_8U = 0 (unsigned 8-bit) CV_8S = 1 (signed 8-bit) CV_16U = 2 (unsigned 16-bit) CV_16S = 3 (signed 16-bit) CV_32S = 4 (signed 32-bit) CV_32F = 5 (32-bit float) CV_64F = 6 (64-bit float) When you see the error message src data type = 4 is not supported, it explicitly refers to CV_32S [1][2]. Other data types, such as CV_16F (often represented as type 23 in error messages [3]) or various signed 8-bit combinations (such as CV_8SC3, often reported as 17 [4][5]), may also trigger unsupported type errors depending on your specific OpenCV build and function overload. To resolve this issue, you must convert your image data to a supported type—typically CV_8U or CV_32F—before calling cv2.resize. For example, if you are working with integer masks or data stored in 32-bit signed integers, you can cast the array to a float32 type using NumPy: import cv2 import numpy as np # Assuming 'img' is your CV_32S image img_resized = cv2.resize(img.astype(np.float32), (new_width, new_height)) # If you need to return to integer format, you can cast back after resizing img_resized = img_resized.astype(np.int32) Note that if you are working with 32-bit unsigned integers (uint32), OpenCV also does not have direct support for these and will often attempt to interpret them as CV_32S, similarly leading to an unsupported type error [1]. In such cases, casting to float32 is likewise the standard workaround [1].

Citations:


Resize the segmentation mask instead of the label_map.

segmentation.label_map is int32 by construction for both modeled and loaded label maps, but cv2.resize does not support CV_32S. In road_patch/seg_intersection paths, label-map shapes are normalized to diffusion input size while generation.py passes the fitted image size, so the resize branch is hit and crashes when seg_class_ids is used. Apply np.isin(labels, ...) first, resize the support mask, then return bool.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@implementations/edge_case_image_generation/src/edgecase_synthesis/conditioning.py`
around lines 368 - 373, Update the segmentation handling around the
seg_class_ids branch to apply np.isin(labels, list(class_ids)) before resizing,
rather than resizing the int32 label_map. Resize the resulting boolean support
mask when its shape differs from (height, width), then return it as a boolean
mask while preserving the existing no-class and missing-label-map behavior.

Comment on lines +344 to +345
with zipfile.ZipFile(zip_path, "r") as zf:
zf.extractall(extract_dir)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Zip-slip: validate member paths before extracting a remotely-fetched archive. archive_url comes from dataset config, so a crafted or compromised zip entry (../../…) can write outside extract_dir.

🛡️ Proposed fix
         with zipfile.ZipFile(zip_path, "r") as zf:
-            zf.extractall(extract_dir)
+            root = extract_dir.resolve()
+            for member in zf.namelist():
+                dest = (root / member).resolve()
+                if not dest.is_relative_to(root):
+                    raise ValueError(f"Refusing unsafe zip entry: {member!r}")
+            zf.extractall(extract_dir)
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
with zipfile.ZipFile(zip_path, "r") as zf:
zf.extractall(extract_dir)
with zipfile.ZipFile(zip_path, "r") as zf:
root = extract_dir.resolve()
for member in zf.namelist():
dest = (root / member).resolve()
if not dest.is_relative_to(root):
raise ValueError(f"Refusing unsafe zip entry: {member!r}")
zf.extractall(extract_dir)
🧰 Tools
🪛 ast-grep (0.45.0)

[error] 344-344: Calling extractall() on a zipfile.ZipFile or tarfile archive without validating member paths lets a crafted entry (e.g. "../../etc/passwd") write outside the destination directory (Zip Slip). Validate each member resolves inside the target directory, or pass a safe filter (tarfile: filter="data" / tarfile.data_filter).
Context: zf.extractall(extract_dir)
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').

(archive-extractall-path-traversal-python)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@implementations/edge_case_image_generation/src/edgecase_synthesis/data.py`
around lines 344 - 345, Update the archive extraction flow around
zipfile.ZipFile and extractall to validate every member path before extraction,
rejecting absolute paths and normalized paths that escape extract_dir (including
traversal via symlinks if applicable). Only call extractall after all entries
pass validation, preserving extraction for safe archives.

Source: Linters/SAST tools

Comment on lines +550 to +555
elif isinstance(objects, list):
for obj in objects:
label = str(obj.get("label") or obj.get("category") or "defect")
bbox = obj.get("bbox") or obj.get("bbox_xyxy") or obj.get("box")
if bbox is not None:
boxes.append(_box_from_any(bbox, label, image))

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

Guard against bboxes with fewer than 4 values. The dict-object branch passes any bbox straight through; _box_from_any then indexes vals[2]/vals[3] and unpacks four values, so a short or malformed box raises IndexError/ValueError and aborts the whole ingest run. The annot branch already checks len(nums) >= 4.

🐛 Proposed fix
 def _box_from_any(
     bbox: Any,
     label: str,
     image: Image.Image | None,
 ) -> DetectionBox:
     vals = [float(x) for x in list(bbox)[:4]]
+    if len(vals) < 4:
+        raise ValueError(f"Malformed bbox for {label!r}: {bbox!r}")

and skip instead of crashing at the call site:

-            if bbox is not None:
-                boxes.append(_box_from_any(bbox, label, image))
+            if bbox is not None and len(list(bbox)) >= 4:
+                boxes.append(_box_from_any(bbox, label, image))

Also applies to: 577-587

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@implementations/edge_case_image_generation/src/edgecase_synthesis/data.py`
around lines 550 - 555, Validate the bbox values in the objects-list branch
before calling _box_from_any: normalize or inspect the candidate bbox and skip
entries with fewer than four usable values, matching the existing annot branch
behavior. Apply the same guard to the additional bbox handling around the
corresponding branch, while preserving valid box processing and preventing
malformed entries from aborting ingestion.

Comment on lines +109 to +124
dtype_kw: dict[str, Any] = {}
# Prefer dtype= for newer transformers; torch_dtype still widely accepted.
dtype_kw["torch_dtype"] = self.torch_dtype

processor = AutoProcessor.from_pretrained(
self.model_id,
min_pixels=self.min_pixels,
max_pixels=self.max_pixels,
trust_remote_code=True,
)
model = AutoModelForImageTextToText.from_pretrained(
self.model_id,
trust_remote_code=True,
device_map="auto" if self.device.type == "cuda" else None,
**dtype_kw,
)

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔒 Security & Privacy | 🟠 Major | ⚡ Quick win

Drop trust_remote_code=True — Qwen2.5-VL is natively supported.

model_id comes from YAML (judge.model_id), so this flag means any config edit or typo pointing at an untrusted Hub repo executes that repo's Python at load time. transformers 4.49 ships native Qwen2_5_VL support, so remote code isn't needed. The dtype_kw dict is also a no-op indirection now.

🔒️ Proposed fix
-        dtype_kw: dict[str, Any] = {}
-        # Prefer dtype= for newer transformers; torch_dtype still widely accepted.
-        dtype_kw["torch_dtype"] = self.torch_dtype
-
         processor = AutoProcessor.from_pretrained(
             self.model_id,
             min_pixels=self.min_pixels,
             max_pixels=self.max_pixels,
-            trust_remote_code=True,
         )
         model = AutoModelForImageTextToText.from_pretrained(
             self.model_id,
-            trust_remote_code=True,
             device_map="auto" if self.device.type == "cuda" else None,
-            **dtype_kw,
+            torch_dtype=self.torch_dtype,
         )
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
dtype_kw: dict[str, Any] = {}
# Prefer dtype= for newer transformers; torch_dtype still widely accepted.
dtype_kw["torch_dtype"] = self.torch_dtype
processor = AutoProcessor.from_pretrained(
self.model_id,
min_pixels=self.min_pixels,
max_pixels=self.max_pixels,
trust_remote_code=True,
)
model = AutoModelForImageTextToText.from_pretrained(
self.model_id,
trust_remote_code=True,
device_map="auto" if self.device.type == "cuda" else None,
**dtype_kw,
)
processor = AutoProcessor.from_pretrained(
self.model_id,
min_pixels=self.min_pixels,
max_pixels=self.max_pixels,
)
model = AutoModelForImageTextToText.from_pretrained(
self.model_id,
device_map="auto" if self.device.type == "cuda" else None,
torch_dtype=self.torch_dtype,
)
🧰 Tools
🪛 ast-grep (0.45.0)

[warning] 116-116: Loading a HuggingFace model, tokenizer, dataset or pipeline with trust_remote_code=True executes arbitrary code from the remote repository. Only enable it for fully trusted sources; prefer artifacts that do not require it.
Context: trust_remote_code=True
Note: [CWE-94] Improper Control of Generation of Code ('Code Injection').

(transformers-trust-remote-code-python)


[warning] 120-120: Loading a HuggingFace model, tokenizer, dataset or pipeline with trust_remote_code=True executes arbitrary code from the remote repository. Only enable it for fully trusted sources; prefer artifacts that do not require it.
Context: trust_remote_code=True
Note: [CWE-94] Improper Control of Generation of Code ('Code Injection').

(transformers-trust-remote-code-python)

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@implementations/edge_case_image_generation/src/edgecase_synthesis/judge.py`
around lines 109 - 124, Update the model-loading flow using AutoProcessor and
AutoModelForImageTextToText to remove trust_remote_code=True from both calls,
relying on native Qwen2.5-VL support. Replace the dtype_kw indirection by
passing self.torch_dtype directly to the model loader, while preserving the
existing device_map and pixel configuration.

Source: Linters/SAST tools

Comment on lines +132 to +143
def _ensure_clip(self) -> None:
if self._clip is not None:
return
from transformers import CLIPModel, CLIPProcessor

clip_id = "openai/clip-vit-base-patch32"
processor = CLIPProcessor.from_pretrained(clip_id)
model = CLIPModel.from_pretrained(clip_id)
model.to(self.device).eval()
self._clip = (processor, model)
self._active_backend = "clip"
self.model_id = clip_id

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Overwriting self.model_id with the CLIP id poisons later Qwen loads.

_ensure_qwen loads self.model_id. Once a single CLIP fallback fires, self.model_id == "openai/clip-vit-base-patch32", so every subsequent judge() call on the same instance tries to load CLIP as an image-text-to-text model, fails, and silently degrades to CLIP forever — while backend still reports qwen_vl intent. Keep the configured id immutable and report the active id separately.

🐛 Proposed fix
     def _ensure_clip(self) -> None:
         if self._clip is not None:
             return
         from transformers import CLIPModel, CLIPProcessor
 
         clip_id = "openai/clip-vit-base-patch32"
         processor = CLIPProcessor.from_pretrained(clip_id)
         model = CLIPModel.from_pretrained(clip_id)
         model.to(self.device).eval()
         self._clip = (processor, model)
         self._active_backend = "clip"
-        self.model_id = clip_id
+        self._active_model_id = clip_id

Then set result.model_id = self._active_model_id or self.model_id in judge() (Line 188), initializing self._active_model_id = None in __init__ and clearing it in unload().

📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
def _ensure_clip(self) -> None:
if self._clip is not None:
return
from transformers import CLIPModel, CLIPProcessor
clip_id = "openai/clip-vit-base-patch32"
processor = CLIPProcessor.from_pretrained(clip_id)
model = CLIPModel.from_pretrained(clip_id)
model.to(self.device).eval()
self._clip = (processor, model)
self._active_backend = "clip"
self.model_id = clip_id
def _ensure_clip(self) -> None:
if self._clip is not None:
return
from transformers import CLIPModel, CLIPProcessor
clip_id = "openai/clip-vit-base-patch32"
processor = CLIPProcessor.from_pretrained(clip_id)
model = CLIPModel.from_pretrained(clip_id)
model.to(self.device).eval()
self._clip = (processor, model)
self._active_backend = "clip"
self._active_model_id = clip_id
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@implementations/edge_case_image_generation/src/edgecase_synthesis/judge.py`
around lines 132 - 143, Keep the configured self.model_id immutable in
_ensure_clip; record the fallback model in a separate _active_model_id instead.
Initialize _active_model_id to None in __init__, clear it in unload(), and have
judge() report _active_model_id when set, otherwise self.model_id, so
_ensure_qwen continues loading the configured model.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant