Edge case generation notebooks 1 and 1.5 - #37
Conversation
Railway edge-case synthesis demo under implementations/edge_case_image_generation (configs, package, notebook, docs). Dataset payloads and model weights are gitignored. Co-authored-by: Cursor <cursoragent@cursor.com>
Revert root .gitignore to match origin/main; dataset/output ignores remain in implementations/edge_case_image_generation/.gitignore. Co-authored-by: Cursor <cursoragent@cursor.com>
Use CLIP-style photo-of queries, a lower score threshold with NMS, and seed paste anomaly masks so edge-case labels still appear. Co-authored-by: Cursor <cursoragent@cursor.com>
Swap the model stack via Hydra (paste + SD 1.5 on CPU; SDXL inpaint/ControlNet + larger depth/seg/annotators on g2-standard-8). Co-authored-by: Cursor <cursoragent@cursor.com>
Pass width/height into diffusion calls and resize before blending so L4 runs no longer mix 1024² outputs with source masks. Co-authored-by: Cursor <cursoragent@cursor.com>
Replace fixed YAML x/y blobs with depth + cab-track corridor anchors, skip cartoon primes on inpaint, and upscale small frames before SDXL. Co-authored-by: Cursor <cursoragent@cursor.com>
Qwen2.5-VL-3B on CPU (CLIP fallback) and Qwen2.5-VL-7B on L4 score RGB synthetics for accept/retry/reject. Co-authored-by: Cursor <cursoragent@cursor.com>
Drop paste/cab-view hardcoding; CPU uses depth ControlNet and GPU keeps SDXL inpaint; slim data loaders and notebook. Co-authored-by: Cursor <cursoragent@cursor.com>
Replace hard-to-read NEU-DET steel textures with open street-level road photos where damage and hazards are sparse, intuitive long-tail cases. Co-authored-by: Cursor <cursoragent@cursor.com>
Add Notebook 1.5 comparing inpaint, dual ControlNet, and InstructPix2Pix on clean scene seeds (pothole, cone, fog), with CPU/L4 hardware configs and HF range-extract helper for the small Mapillary subset. Co-authored-by: Cursor <cursoragent@cursor.com>
Extract script now rebuilds the EOCD Range index when missing so GPU boxes do not need a checked-in 21MB jsonl; prepare_sample_images refuses Nordland leftovers and can invoke the Mapillary toy extract when samples/ is empty. Co-authored-by: Cursor <cursoragent@cursor.com>
Separate low controlnet_strength (~0.4) from inpaint strength, raise depth/seg conditioning scales, and add fidelity prompts / instruct image-guidance so comparisons stay on the same photo. Co-authored-by: Cursor <cursoragent@cursor.com>
Transformers' Mask2FormerLoss imports scipy; without it section 2 fails when hardware=gpu_l4 switches to facebook/mask2former-swin-small-ade. Co-authored-by: Cursor <cursoragent@cursor.com>
Mask2Former still works if scipy is installed; otherwise section 2 failed even after adding the dependency when the notebook kernel was not resynced. Co-authored-by: Cursor <cursoragent@cursor.com>
Transformers caches scipy availability at import time, so %pip install alone often still fails; auto-fallback unblocks gpu_l4 section 2 either way. Co-authored-by: Cursor <cursoragent@cursor.com>
Stop prefer_near from collapsing edit holes into crescents, enlarge cone/pothole masks, switch L4 instruct to SD1.5 InstructPix2Pix, and tighten SDXL ControlNet denoise so dual edits stay closer to the source photo. Co-authored-by: Cursor <cursoragent@cursor.com>
Learners choose METHOD_BY_ANOMALY from the comparison notebook, then run load → edit → annotate → judge → retry without digging into each method. Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
SDXL was rewriting scenes; both hardware profiles now share SD1.5 inpaint with clean ellipse masks, crop-to-mask, and short object prompts so local inserts are honestly reliable. ControlNet/instruct stay as comparison baselines. Co-authored-by: Cursor <cursoragent@cursor.com>
Asphalt potholes are nearly invisible in SD inpaint demos; a cardboard box on the lane is an honest local-insert target. Also tone down cartoon cone prompts. Co-authored-by: Cursor <cursoragent@cursor.com>
Dedicated inpaint models already preserve outside the mask; re-blending faded inserts into smudges. Also enlarge holes, raise strength/steps/CFG, and widen crop-to-mask padding. Co-authored-by: Cursor <cursoragent@cursor.com>
Replace weak SD1.5 InstructPix2Pix/inpaint on gpu_l4 with Klein pipelines while keeping SD ControlNet for the depth+seg baseline. Co-authored-by: Cursor <cursoragent@cursor.com>
Large FLUX.2-klein downloads often die with a masked Xet writer error; fall back to classic hub HTTP. Co-authored-by: Cursor <cursoragent@cursor.com>
Drop the stale hardcoded pothole/ground_animal list so the compare study matches default.yaml. Co-authored-by: Cursor <cursoragent@cursor.com>
Fixed ellipses were floating cones/debris; road_patch places the hole on near road pixels and prompts emphasize ground contact. Co-authored-by: Cursor <cursoragent@cursor.com>
Move shared baselines under configs/default/, per-dataset sources and anomalies under configs/datasets/, and point samples/outputs at data/<dataset>/. Co-authored-by: Cursor <cursoragent@cursor.com>
CPU hardware no longer sets a pruned null device; synthesize_one accepts seed_offset for judge retries. Co-authored-by: Cursor <cursoragent@cursor.com>
Open-vocab labeling now uses yolov8s-worldv2 with tight per-anomaly queries; judge still validates bbox summaries without edit-mask seeding. Co-authored-by: Cursor <cursoragent@cursor.com>
Encode class prompts on CPU before GPU predict, and run anomalies from METHOD_BY_ANOMALY keys instead of only dataset.workshop_anomalies. Co-authored-by: Cursor <cursoragent@cursor.com>
Lower L4 threshold to 8.0 so overall=8 accepts; use scene-level fog queries with a lower annotation_conf; prefer instruct over brittle inpaint for Mapillary locals. Co-authored-by: Cursor <cursoragent@cursor.com>
High faithfulness/overall with edge_case_present=false no longer hard-rejects a clearly present cone. Co-authored-by: Cursor <cursoragent@cursor.com>
Document fog as a manual-label teaching case; link stubs from README and NB1 wrap-up. Co-authored-by: Cursor <cursoragent@cursor.com>
📝 WalkthroughWalkthroughThis PR adds a complete edge-case image generation implementation with dataset-driven configuration, multiple image-editing methods, depth and segmentation conditioning, open-vocabulary annotation, VLM-based judging, retry handling, visualization utilities, notebooks, documentation, and environment setup scripts. ChangesEdge-case image synthesis
Estimated code review effort: 5 (Critical) | ~120 minutes Suggested labels: Suggested reviewers: Sequence Diagram(s)sequenceDiagram
participant Notebook
participant DataLoader
participant Conditioning
participant MethodComparer
participant Annotator
participant VLMJudge
Notebook->>DataLoader: prepare_sample_images()
DataLoader-->>Notebook: sample images
Notebook->>Conditioning: predict depth and segmentation
Conditioning-->>Notebook: conditioning results
Notebook->>MethodComparer: synthesize anomaly edit
MethodComparer-->>Notebook: generated image
Notebook->>Annotator: annotate generated image
Annotator-->>Notebook: detections and overlay
Notebook->>VLMJudge: judge image and annotations
VLMJudge-->>Notebook: accept, retry, or reject decision
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✨ Finishing Touches 💡 2📝 Generate docstrings 💡
🛠️ Fix failing CI checks 💡
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
There was a problem hiding this comment.
Actionable comments posted: 16
🧹 Nitpick comments (15)
implementations/edge_case_image_generation/notebooks/01.5_method_comparison.ipynb (2)
175-184: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low valueAdd
strict=tozip().
zip(scene_paths[: len(ANOMALIES)], ANOMALIES)silently truncates to the shorter iterable. Since the code already validateslen(scene_paths) >= len(ANOMALIES)just above, this should be a non-issue in practice, but passingstrict=Truedocuments that intent and fails loudly if the invariant is ever violated.As per static analysis hints, Ruff flags
zip()without an explicitstrict=parameter (B905).🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@implementations/edge_case_image_generation/notebooks/01.5_method_comparison.ipynb` around lines 175 - 184, Update the jobs construction loop around scene_paths and ANOMALIES to call zip with strict=True, preserving the existing iterable slicing and job creation behavior while ensuring length mismatches fail loudly.Source: Linters/SAST tools
44-49: 🔒 Security & Privacy | 🔵 Trivial | ⚡ Quick winClear notebook outputs before committing.
The committed outputs leak the author's local filesystem path (
/Users/arvinazarmina/Documents/Projects/..., line 47) and include a stray failed-download traceback with an "unsafe serialization" fallback warning (lines 371-374) from a broken local cache — neither is meaningful to reviewers and both bloat the diff. Consider stripping outputs (e.g.,nbstripoutorjupyter nbconvert --clear-output) before committing notebooks.Also applies to: 371-377
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@implementations/edge_case_image_generation/notebooks/01.5_method_comparison.ipynb` around lines 44 - 49, Clear all stored outputs from the notebook before committing, including the Project root stdout containing the author’s local path and the failed-download traceback and unsafe-serialization warning near the later output block. Preserve the notebook’s code and markdown cells while removing execution results and transient error output, using the repository’s standard notebook-output cleanup tool if available.implementations/edge_case_image_generation/src/edgecase_synthesis/annotation.py (2)
165-170: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win
unload()leaves the object in an unusable state with no guard.After
unload(), anyannotate()call fails withAttributeError: 'NoneType' object has no attribute 'set_classes'. A explicit check in_set_classes(raiseRuntimeError("annotator unloaded; rebuild via from_config")) makes the notebook lifecycle mistake obvious.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@implementations/edge_case_image_generation/src/edgecase_synthesis/annotation.py` around lines 165 - 170, Update _set_classes to detect when unload has cleared self.model and raise RuntimeError("annotator unloaded; rebuild via from_config") before accessing the model. Preserve normal class-setting behavior when the annotator remains loaded.
69-72: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueSilent
except Exception: passaround device moves.If
.to("cpu")fails, the CLIP device-mismatch this guard exists to prevent resurfaces later as an opaqueExpected all tensors to be on the same device. Log at debug/warning level so the cause is recoverable. Same pattern at Lines 153-161.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@implementations/edge_case_image_generation/src/edgecase_synthesis/annotation.py` around lines 69 - 72, Update the exception handling around the device moves in the annotation flow, including the corresponding block near the later model transfer, to log the caught exception at debug or warning level instead of silently passing. Preserve the existing fallback behavior while making failures from self.model.to("cpu") recoverable in logs.Source: Linters/SAST tools
implementations/edge_case_image_generation/src/edgecase_synthesis/generation.py (2)
334-364: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win
padding_mask_cropsupport is probed by catchingTypeErroraround a full diffusion call. ATypeErrorraised anywhere inside the denoise loop triggers a second complete inference pass (minutes of GPU time) and masks genuine bugs. Probe the pipeline signature once instead.
implementations/edge_case_image_generation/src/edgecase_synthesis/generation.py#L334-L364: replace both try/except blocks in_inpaintwith a shared_supports_kwarg(pipe, "padding_mask_crop")check before calling the pipe.implementations/edge_case_image_generation/src/edgecase_synthesis/compare_methods.py#L530-L563: apply the same helper in the Klein and SD branches of_run_inpaint.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@implementations/edge_case_image_generation/src/edgecase_synthesis/generation.py` around lines 334 - 364, Replace the full-call TypeError fallbacks in _inpaint with a shared _supports_kwarg(pipe, "padding_mask_crop") check, removing padding_mask_crop before invocation when unsupported; apply the same change to both Klein and SD branches of _run_inpaint in implementations/edge_case_image_generation/src/edgecase_synthesis/compare_methods.py (lines 530-563), while preserving all other inference arguments and behavior.
227-243: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winHardcoded strength clamp plus mismatched ControlNet-scale defaults.
Two config-contract wrinkles:
- Line 227/231: scalar default is
0.55but the dict path defaults to0.75, so behavior silently shifts depending on whether YAML wrote a scalar or a mapping.- Line 243:
np.clip(..., 0.25, 0.60)makes any configuredcontrolnet_strengthabove 0.60 a no-op.compare_methods.pyalready exposescontrolnet_strength_cap_*keys for this; reusing them here keeps the caps discoverable and overridable.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@implementations/edge_case_image_generation/src/edgecase_synthesis/generation.py` around lines 227 - 243, The ControlNet configuration handling uses inconsistent scale defaults and hardcodes the strength upper bound. Update the scale extraction in the surrounding generation method so scalar and mapping configurations share the same default, and replace the fixed np.clip upper limit in the _controlnet call with the existing controlnet_strength_cap_* configuration value used by compare_methods.py, while preserving the configured lower bound and override behavior.implementations/edge_case_image_generation/src/edgecase_synthesis/compare_methods.py (3)
141-213: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winPipeline plumbing duplicated verbatim from
AnomalyEditor.
_is_klein_model/inpaint_is_klein,_dtype,_place,_gen, and_fit_klein_imageare near-identical togeneration.pyLines 82-106 / 427-447. Extract a small mixin or base class so device/dtype/offload policy changes apply to both runners.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@implementations/edge_case_image_generation/src/edgecase_synthesis/compare_methods.py` around lines 141 - 213, The pipeline utility methods in the current runner duplicate AnomalyEditor; extract the shared Klein/device helpers into a small mixin or base class and have both runners reuse it. Move `_is_klein_model`, the Klein model properties, `_dtype`, `_place`, `_gen`, and `_fit_klein_image` into the shared abstraction, preserving their existing behavior so future device, dtype, and offload policy changes apply consistently.
474-479: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winUnconditional
unload()per method forces repeated multi-GB model reloads.
_build_*already calls_free_other_edit_pipes(keep=...), so this extraunload()mainly guarantees the pipe just used is discarded and re-downloaded/re-deserialized on the next anomaly in the outer loop. Dropping it (and relying on_free_other_edit_pipes) keeps the same VRAM guarantee without the reload cost.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@implementations/edge_case_image_generation/src/edgecase_synthesis/compare_methods.py` around lines 474 - 479, Remove the conditional self.unload() call from the per-method loop in the method executing flow, while preserving the existing CUDA and uses_klein checks only if needed elsewhere. Rely on the _build_* methods’ _free_other_edit_pipes(keep=...) behavior to maintain VRAM management without discarding the current pipe between methods.
24-31: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winImporting five private helpers across modules.
_depth_to_control_image,_ensure_same_size,_fit_for_diffusion,_mask_to_pil, and_compositeare private togeneration.pyyet form the de-facto shared image-prep API. Move them into a small shared module (e.g.imaging.py) with public names so the contract is explicit and refactors ingeneration.pydon't silently break this file.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@implementations/edge_case_image_generation/src/edgecase_synthesis/compare_methods.py` around lines 24 - 31, Move the shared image-preparation helpers _depth_to_control_image, _ensure_same_size, _fit_for_diffusion, _mask_to_pil, and _composite from generation.py into a dedicated imaging module, rename them as public helpers, and update both generation.py and compare_methods.py to import and use the new symbols. Keep their existing behavior unchanged while removing cross-module imports of generation.py’s private helpers.implementations/edge_case_image_generation/src/edgecase_synthesis/judge.py (1)
322-343: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winDead
if/passbranch, and the docstring numbers don't match the defaults.
- Lines 331-333: the
if result.edge_case_present and result.overall < 3.0: passblock is a no-op; keep it as a comment only.- Line 339: the worked example says
threshold=8.0while__init__/from_configdefault to8.5, so the stated accept/reject bands (≥8 / <6) are off by 0.5.- Worth noting explicitly that
looks_presentoverrides an explicitedge_case_present=falsefrom the VLM whenever faithfulness ≥ 7, which turns hard rejects into accept/retry — intentional, but it should be config-gated rather than hardcoded.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@implementations/edge_case_image_generation/src/edgecase_synthesis/judge.py` around lines 322 - 343, The reconciliation logic around looks_present should remain, but remove the no-op result.edge_case_present/overall branch and preserve its intent as a comment only. Update _decide’s docstring example to use the actual default threshold of 8.5 and corresponding accept/reject bands. Make the looks_present override of an explicit false edge_case_present configuration-gated, using the existing configuration mechanism, while preserving the current behavior when enabled.implementations/edge_case_image_generation/src/edgecase_synthesis/conditioning.py (1)
337-353: 🎯 Functional Correctness | 🔵 Trivial | 💤 Low valueSingle-pixel argmax center can sit on the road boundary.
bestis one pixel, so the ellipse frequently straddles the road edge. Averaging the top-k scored pixels (or taking the centroid of the best-scoring connected component) gives a more stable anchor.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@implementations/edge_case_image_generation/src/edgecase_synthesis/conditioning.py` around lines 337 - 353, Update the center-selection logic after computing score so it averages the coordinates of the top-scoring pixels, rather than using the single index from np.argmax. Preserve the existing score calculation and ensure cx/cy are derived from the selected top-k xs/ys coordinates, producing a stable anchor within the road region.implementations/edge_case_image_generation/src/edgecase_synthesis/config.py (1)
124-124: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value
PROJECT_ROOTenv override is silently discarded.paths.yamldefinesproject_root: ${oc.env:PROJECT_ROOT,.}anddata.py:55-58tells users to "set PROJECT_ROOT", but this line always replaces it with the discovered root. Honor the env value when present.♻️ Suggested change
- cfg.paths.project_root = str(root) + env_root = os.environ.get("PROJECT_ROOT") + cfg.paths.project_root = str(Path(env_root).resolve() if env_root else root)Add
import osat the top of the module.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@implementations/edge_case_image_generation/src/edgecase_synthesis/config.py` at line 124, Update the configuration initialization around cfg.paths.project_root to honor the PROJECT_ROOT environment variable when it is set, while retaining the discovered root as the fallback. Add the required os import and only assign the discovered root when no PROJECT_ROOT override is present.implementations/edge_case_image_generation/src/edgecase_synthesis/data.py (1)
424-429: 🔒 Security & Privacy | 🔵 Trivial | 💤 Low valueConsider
defusedxmlfor archive-supplied XML. VOC annotations come from a downloaded zip; stdlibElementTreeis vulnerable to entity-expansion attacks (Ruff S314).from defusedxml.ElementTree import parseis a drop-in replacement.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@implementations/edge_case_image_generation/src/edgecase_synthesis/data.py` around lines 424 - 429, Update _parse_voc_xml to use defusedxml.ElementTree.parse instead of xml.etree.ElementTree.parse when reading archive-supplied VOC annotations, preserving the existing path checks and root extraction behavior.Source: Linters/SAST tools
implementations/edge_case_image_generation/scripts/extract_mapillary_toy.py (1)
26-40: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueModule import performs config I/O.
load_config()at import time means the module cannot be imported (for reuse or testing) without a resolvable config tree. Moving this intomain()or a small_settings()helper keeps import side-effect free.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@implementations/edge_case_image_generation/scripts/extract_mapillary_toy.py` around lines 26 - 40, Remove the module-level load_config() call and derived configuration values in extract_mapillary_toy.py. Move this setup into main() or a small _settings() helper, and have the extraction flow obtain those settings explicitly so importing the module performs no configuration I/O while preserving the existing defaults and mappings.implementations/edge_case_image_generation/scripts/setup_notebook_env.sh (1)
11-19: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winBoth setup scripts hardcode an Apple-Silicon Homebrew
python3.12instead of the project's declared Python version. This breaks Intel macOS (/usr/local/bin) and Linux GPU hosts — which thehardware/gpu_l4profile targets — and makes the diagnostic disagree with the venv the setup script actually creates.
implementations/edge_case_image_generation/scripts/setup_notebook_env.sh#L11-L19: fall back touv venv --python "$(cat .python-version)"(orcommand -v python3.12) when the Homebrew path is absent, instead ofexit 1.implementations/edge_case_image_generation/scripts/diagnose_python.sh#L20-L25: probe$ROOT/.venv/bin/python(the pathsetup_notebook_env.shprints) rather than the version-suffixedpython3.12.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@implementations/edge_case_image_generation/scripts/setup_notebook_env.sh` around lines 11 - 19, The setup flow hardcodes an Apple-Silicon Homebrew Python path and the diagnostic probes a different interpreter. In implementations/edge_case_image_generation/scripts/setup_notebook_env.sh lines 11-19, fall back to the project’s .python-version via uv venv, or command -v python3.12, when BREW_PYTHON is unavailable instead of exiting; in implementations/edge_case_image_generation/scripts/diagnose_python.sh lines 20-25, probe $ROOT/.venv/bin/python to match the environment created by setup_notebook_env.sh.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@implementations/edge_case_image_generation/configs/datasets/rdd2022/data.yaml`:
- Around line 18-22: Update prefer_classes in the dataset configuration to use
the normalized class names defined by class_map and stem_prefixes—pothole,
alligator_crack, transverse_crack, and longitudinal_crack—in the intended
priority order, so load_sample_images() matches the extracted filename prefixes.
In `@implementations/edge_case_image_generation/docs/context.md`:
- Around line 25-34: Align the documentation with the implemented YOLO-World
box-annotation contract: in
implementations/edge_case_image_generation/docs/context.md lines 25-34, replace
Grounded-SAM 2 with YOLO-World box annotation; in lines 262-283, describe box
outputs and manual or image-level labeling for diffuse effects rather than
guaranteed masks and no manual annotation; in
implementations/edge_case_image_generation/docs/project_summary.md line 11,
replace the Grounded-SAM-2 workflow with the configured YOLO-World path; and in
implementations/edge_case_image_generation/docs/quality_measure.md lines
152-159, replace SAM-mask IoU guidance with a box-compatible annotation-quality
check.
In `@implementations/edge_case_image_generation/docs/project_summary.md`:
- Around line 26-40: Update the reference list in the project summary to use
standard Markdown link syntax without literal ++ wrappers. Correct the malformed
SDXL-Turbo URL so it points to the verified stabilityai/sdxl-turbo repository,
and validate all listed reference links before publishing.
In
`@implementations/edge_case_image_generation/notebooks/01_sample_data_generation.ipynb`:
- Around line 353-367: Persist each retried current result and its refreshed
annotations under an attempt-specific output directory before the _unload call.
Update the retry flow around synthesize_one, annotate_anomaly, and
results[anomaly_id] to save both the image and label artifacts for every
attempt, including accepted retries.
In `@implementations/edge_case_image_generation/pyproject.toml`:
- Around line 22-24: Update the project dependencies in pyproject.toml: replace
the bare clip requirement with a direct PEP 508 URL dependency pointing to the
intended Ultralytics fork so built metadata installs the correct package, and
remove ipykernel from the runtime dependency list, retaining it only in the
existing extras/dev group.
In `@implementations/edge_case_image_generation/scripts/extract_mapillary_toy.py`:
- Around line 286-288: Update the sample cleanup around SAMPLES.mkdir so it does
not unconditionally delete every JPG/PNG in the user-managed samples directory.
Gate cleanup behind an explicit reset flag, or restrict deletion to image files
identified by this script’s toy_meta.json/source_meta.json metadata, preserving
user-provided files and prior samples when no reset is requested.
- Around line 57-65: Update _zip_size and its HEAD call site to prevent the
Hugging Face bearer token from being forwarded across redirects. Resolve the
final URL using Hugging Face’s session/token handling, then issue redirected
requests without Authorization unless the destination remains the same trusted
host, while preserving the existing size fallback behavior.
- Line 340: Update the label-key construction in load_detection_labels so each
labels_out key matches the corresponding ImageSample.name derived from
path.stem; use the stem-based sample name rather than prefixing it with the tag,
while preserving the existing .jpg naming convention.
In
`@implementations/edge_case_image_generation/src/edgecase_synthesis/annotation.py`:
- Around line 191-204: Update _resolve_yolo_weights so it completes the
parent-directory search for the requested weights name before checking any
yolov8s-worldv2.pt fallback. Only perform the fallback search after no requested
file is found, while preserving the existing direct-path resolution and final
detector_model return behavior.
In
`@implementations/edge_case_image_generation/src/edgecase_synthesis/compare_methods.py`:
- Around line 727-752: Normalize scalar and mapping controlnet_scale values
through a shared _as_scale_map(value, *, depth_default, seg_default) helper. In
compare_methods.py lines 727-752, use the normalized map before reading depth
and seg values; in compare_methods.py lines 390-396, handle scalar values
instead of falling back to instance defaults; in generation.py lines 227-243,
reuse the helper and align scalar and mapping defaults, including the current
0.55 versus 0.75 mismatch.
- Around line 390-396: Update the controlnet scale handling around
merged["controlnet_scale"] so scalar values are interpreted consistently with
generation.py rather than silently replaced by instance defaults. Preserve
mapping support for separate depth and seg values, and either apply the scalar
to both runners or explicitly reject it with a clear error.
In
`@implementations/edge_case_image_generation/src/edgecase_synthesis/conditioning.py`:
- Around line 368-373: Update the segmentation handling around the seg_class_ids
branch to apply np.isin(labels, list(class_ids)) before resizing, rather than
resizing the int32 label_map. Resize the resulting boolean support mask when its
shape differs from (height, width), then return it as a boolean mask while
preserving the existing no-class and missing-label-map behavior.
In `@implementations/edge_case_image_generation/src/edgecase_synthesis/data.py`:
- Around line 550-555: Validate the bbox values in the objects-list branch
before calling _box_from_any: normalize or inspect the candidate bbox and skip
entries with fewer than four usable values, matching the existing annot branch
behavior. Apply the same guard to the additional bbox handling around the
corresponding branch, while preserving valid box processing and preventing
malformed entries from aborting ingestion.
- Around line 344-345: Update the archive extraction flow around zipfile.ZipFile
and extractall to validate every member path before extraction, rejecting
absolute paths and normalized paths that escape extract_dir (including traversal
via symlinks if applicable). Only call extractall after all entries pass
validation, preserving extraction for safe archives.
In `@implementations/edge_case_image_generation/src/edgecase_synthesis/judge.py`:
- Around line 109-124: Update the model-loading flow using AutoProcessor and
AutoModelForImageTextToText to remove trust_remote_code=True from both calls,
relying on native Qwen2.5-VL support. Replace the dtype_kw indirection by
passing self.torch_dtype directly to the model loader, while preserving the
existing device_map and pixel configuration.
- Around line 132-143: Keep the configured self.model_id immutable in
_ensure_clip; record the fallback model in a separate _active_model_id instead.
Initialize _active_model_id to None in __init__, clear it in unload(), and have
judge() report _active_model_id when set, otherwise self.model_id, so
_ensure_qwen continues loading the configured model.
---
Nitpick comments:
In
`@implementations/edge_case_image_generation/notebooks/01.5_method_comparison.ipynb`:
- Around line 175-184: Update the jobs construction loop around scene_paths and
ANOMALIES to call zip with strict=True, preserving the existing iterable slicing
and job creation behavior while ensuring length mismatches fail loudly.
- Around line 44-49: Clear all stored outputs from the notebook before
committing, including the Project root stdout containing the author’s local path
and the failed-download traceback and unsafe-serialization warning near the
later output block. Preserve the notebook’s code and markdown cells while
removing execution results and transient error output, using the repository’s
standard notebook-output cleanup tool if available.
In `@implementations/edge_case_image_generation/scripts/extract_mapillary_toy.py`:
- Around line 26-40: Remove the module-level load_config() call and derived
configuration values in extract_mapillary_toy.py. Move this setup into main() or
a small _settings() helper, and have the extraction flow obtain those settings
explicitly so importing the module performs no configuration I/O while
preserving the existing defaults and mappings.
In `@implementations/edge_case_image_generation/scripts/setup_notebook_env.sh`:
- Around line 11-19: The setup flow hardcodes an Apple-Silicon Homebrew Python
path and the diagnostic probes a different interpreter. In
implementations/edge_case_image_generation/scripts/setup_notebook_env.sh lines
11-19, fall back to the project’s .python-version via uv venv, or command -v
python3.12, when BREW_PYTHON is unavailable instead of exiting; in
implementations/edge_case_image_generation/scripts/diagnose_python.sh lines
20-25, probe $ROOT/.venv/bin/python to match the environment created by
setup_notebook_env.sh.
In
`@implementations/edge_case_image_generation/src/edgecase_synthesis/annotation.py`:
- Around line 165-170: Update _set_classes to detect when unload has cleared
self.model and raise RuntimeError("annotator unloaded; rebuild via from_config")
before accessing the model. Preserve normal class-setting behavior when the
annotator remains loaded.
- Around line 69-72: Update the exception handling around the device moves in
the annotation flow, including the corresponding block near the later model
transfer, to log the caught exception at debug or warning level instead of
silently passing. Preserve the existing fallback behavior while making failures
from self.model.to("cpu") recoverable in logs.
In
`@implementations/edge_case_image_generation/src/edgecase_synthesis/compare_methods.py`:
- Around line 141-213: The pipeline utility methods in the current runner
duplicate AnomalyEditor; extract the shared Klein/device helpers into a small
mixin or base class and have both runners reuse it. Move `_is_klein_model`, the
Klein model properties, `_dtype`, `_place`, `_gen`, and `_fit_klein_image` into
the shared abstraction, preserving their existing behavior so future device,
dtype, and offload policy changes apply consistently.
- Around line 474-479: Remove the conditional self.unload() call from the
per-method loop in the method executing flow, while preserving the existing CUDA
and uses_klein checks only if needed elsewhere. Rely on the _build_* methods’
_free_other_edit_pipes(keep=...) behavior to maintain VRAM management without
discarding the current pipe between methods.
- Around line 24-31: Move the shared image-preparation helpers
_depth_to_control_image, _ensure_same_size, _fit_for_diffusion, _mask_to_pil,
and _composite from generation.py into a dedicated imaging module, rename them
as public helpers, and update both generation.py and compare_methods.py to
import and use the new symbols. Keep their existing behavior unchanged while
removing cross-module imports of generation.py’s private helpers.
In
`@implementations/edge_case_image_generation/src/edgecase_synthesis/conditioning.py`:
- Around line 337-353: Update the center-selection logic after computing score
so it averages the coordinates of the top-scoring pixels, rather than using the
single index from np.argmax. Preserve the existing score calculation and ensure
cx/cy are derived from the selected top-k xs/ys coordinates, producing a stable
anchor within the road region.
In `@implementations/edge_case_image_generation/src/edgecase_synthesis/config.py`:
- Line 124: Update the configuration initialization around
cfg.paths.project_root to honor the PROJECT_ROOT environment variable when it is
set, while retaining the discovered root as the fallback. Add the required os
import and only assign the discovered root when no PROJECT_ROOT override is
present.
In `@implementations/edge_case_image_generation/src/edgecase_synthesis/data.py`:
- Around line 424-429: Update _parse_voc_xml to use defusedxml.ElementTree.parse
instead of xml.etree.ElementTree.parse when reading archive-supplied VOC
annotations, preserving the existing path checks and root extraction behavior.
In
`@implementations/edge_case_image_generation/src/edgecase_synthesis/generation.py`:
- Around line 334-364: Replace the full-call TypeError fallbacks in _inpaint
with a shared _supports_kwarg(pipe, "padding_mask_crop") check, removing
padding_mask_crop before invocation when unsupported; apply the same change to
both Klein and SD branches of _run_inpaint in
implementations/edge_case_image_generation/src/edgecase_synthesis/compare_methods.py
(lines 530-563), while preserving all other inference arguments and behavior.
- Around line 227-243: The ControlNet configuration handling uses inconsistent
scale defaults and hardcodes the strength upper bound. Update the scale
extraction in the surrounding generation method so scalar and mapping
configurations share the same default, and replace the fixed np.clip upper limit
in the _controlnet call with the existing controlnet_strength_cap_*
configuration value used by compare_methods.py, while preserving the configured
lower bound and override behavior.
In `@implementations/edge_case_image_generation/src/edgecase_synthesis/judge.py`:
- Around line 322-343: The reconciliation logic around looks_present should
remain, but remove the no-op result.edge_case_present/overall branch and
preserve its intent as a comment only. Update _decide’s docstring example to use
the actual default threshold of 8.5 and corresponding accept/reject bands. Make
the looks_present override of an explicit false edge_case_present
configuration-gated, using the existing configuration mechanism, while
preserving the current behavior when enabled.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Repository UI
Review profile: CHILL
Plan: Pro Plus
Run ID: da316329-cb1d-4bc1-ba9e-1ae535d33fe3
⛔ Files ignored due to path filters (1)
implementations/edge_case_image_generation/uv.lockis excluded by!**/*.lock
📒 Files selected for processing (71)
implementations/edge_case_image_generation/.gitignoreimplementations/edge_case_image_generation/.python-versionimplementations/edge_case_image_generation/README.mdimplementations/edge_case_image_generation/configs/config.yamlimplementations/edge_case_image_generation/configs/datasets/_template/annotation.yamlimplementations/edge_case_image_generation/configs/datasets/_template/data.yamlimplementations/edge_case_image_generation/configs/datasets/_template/dataset.yamlimplementations/edge_case_image_generation/configs/datasets/_template/generation/anomalies/example_anomaly.yamlimplementations/edge_case_image_generation/configs/datasets/_template/generation/default.yamlimplementations/edge_case_image_generation/configs/datasets/mapillary_vistas/annotation.yamlimplementations/edge_case_image_generation/configs/datasets/mapillary_vistas/data.yamlimplementations/edge_case_image_generation/configs/datasets/mapillary_vistas/dataset.yamlimplementations/edge_case_image_generation/configs/datasets/mapillary_vistas/generation/anomalies/fog.yamlimplementations/edge_case_image_generation/configs/datasets/mapillary_vistas/generation/anomalies/ground_animal.yamlimplementations/edge_case_image_generation/configs/datasets/mapillary_vistas/generation/anomalies/pothole.yamlimplementations/edge_case_image_generation/configs/datasets/mapillary_vistas/generation/anomalies/road_debris.yamlimplementations/edge_case_image_generation/configs/datasets/mapillary_vistas/generation/anomalies/traffic_cone.yamlimplementations/edge_case_image_generation/configs/datasets/mapillary_vistas/generation/default.yamlimplementations/edge_case_image_generation/configs/datasets/nordland_hf/annotation.yamlimplementations/edge_case_image_generation/configs/datasets/nordland_hf/data.yamlimplementations/edge_case_image_generation/configs/datasets/nordland_hf/dataset.yamlimplementations/edge_case_image_generation/configs/datasets/nordland_hf/generation/anomalies/dusk.yamlimplementations/edge_case_image_generation/configs/datasets/nordland_hf/generation/anomalies/fog.yamlimplementations/edge_case_image_generation/configs/datasets/nordland_hf/generation/anomalies/snow.yamlimplementations/edge_case_image_generation/configs/datasets/nordland_hf/generation/default.yamlimplementations/edge_case_image_generation/configs/datasets/rdd2022/annotation.yamlimplementations/edge_case_image_generation/configs/datasets/rdd2022/data.yamlimplementations/edge_case_image_generation/configs/datasets/rdd2022/dataset.yamlimplementations/edge_case_image_generation/configs/datasets/rdd2022/generation/anomalies/alligator_crack.yamlimplementations/edge_case_image_generation/configs/datasets/rdd2022/generation/anomalies/pothole.yamlimplementations/edge_case_image_generation/configs/datasets/rdd2022/generation/anomalies/traffic_cone.yamlimplementations/edge_case_image_generation/configs/datasets/rdd2022/generation/default.yamlimplementations/edge_case_image_generation/configs/default/annotation.yamlimplementations/edge_case_image_generation/configs/default/conditioning.yamlimplementations/edge_case_image_generation/configs/default/generation.yamlimplementations/edge_case_image_generation/configs/default/judge.yamlimplementations/edge_case_image_generation/configs/default/paths.yamlimplementations/edge_case_image_generation/configs/hardware/cpu.yamlimplementations/edge_case_image_generation/configs/hardware/gpu_l4.yamlimplementations/edge_case_image_generation/data/l4r_nlb/archives/.gitkeepimplementations/edge_case_image_generation/data/mapillary_vistas/.gitkeepimplementations/edge_case_image_generation/data/mapillary_vistas/samples/.gitkeepimplementations/edge_case_image_generation/data/neu_det/.gitkeepimplementations/edge_case_image_generation/data/nordland_hf/.gitkeepimplementations/edge_case_image_generation/data/nordland_hf/samples/.gitkeepimplementations/edge_case_image_generation/data/rdd2022/.gitkeepimplementations/edge_case_image_generation/data/rdd2022/samples/.gitkeepimplementations/edge_case_image_generation/docs/context.mdimplementations/edge_case_image_generation/docs/models_and_science.mdimplementations/edge_case_image_generation/docs/potential_datasets.mdimplementations/edge_case_image_generation/docs/project_summary.mdimplementations/edge_case_image_generation/docs/quality_measure.mdimplementations/edge_case_image_generation/notebooks/01.5_method_comparison.ipynbimplementations/edge_case_image_generation/notebooks/01_sample_data_generation.ipynbimplementations/edge_case_image_generation/notebooks/02_batch_dataset_generation.ipynbimplementations/edge_case_image_generation/notebooks/03_training_and_evaluation.ipynbimplementations/edge_case_image_generation/outputs/.gitkeepimplementations/edge_case_image_generation/pyproject.tomlimplementations/edge_case_image_generation/scripts/diagnose_python.shimplementations/edge_case_image_generation/scripts/extract_mapillary_toy.pyimplementations/edge_case_image_generation/scripts/setup_notebook_env.shimplementations/edge_case_image_generation/src/edgecase_synthesis/__init__.pyimplementations/edge_case_image_generation/src/edgecase_synthesis/annotation.pyimplementations/edge_case_image_generation/src/edgecase_synthesis/compare_methods.pyimplementations/edge_case_image_generation/src/edgecase_synthesis/conditioning.pyimplementations/edge_case_image_generation/src/edgecase_synthesis/config.pyimplementations/edge_case_image_generation/src/edgecase_synthesis/data.pyimplementations/edge_case_image_generation/src/edgecase_synthesis/generation.pyimplementations/edge_case_image_generation/src/edgecase_synthesis/judge.pyimplementations/edge_case_image_generation/src/edgecase_synthesis/pipeline.pyimplementations/edge_case_image_generation/src/edgecase_synthesis/viz.py
| prefer_classes: | ||
| - D40 | ||
| - D20 | ||
| - D10 | ||
| - D00 |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
Use normalized class names for sample preference ordering.
load_sample_images() passes prefer_classes directly into path-based selection, but this file supplies raw source IDs while its stem_prefixes and class_map use normalized names. Unless extracted filenames retain D40/D20/D10/D00, the preference ordering will not match and the eight selected samples may not prioritize the intended rare classes. Use pothole, alligator_crack, transverse_crack, and longitudinal_crack, or translate IDs before selection.
This follows the supplied load_sample_images contract and this file’s class_map/stem_prefixes.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@implementations/edge_case_image_generation/configs/datasets/rdd2022/data.yaml`
around lines 18 - 22, Update prefer_classes in the dataset configuration to use
the normalized class names defined by class_map and stem_prefixes—pothole,
alligator_crack, transverse_crack, and longitudinal_crack—in the intended
priority order, so load_sample_images() matches the extracted filename prefixes.
| Controlled Image Editing | ||
| (ControlNet + Diffusion) | ||
| │ | ||
| ▼ | ||
| Automatic Annotation | ||
| (Grounded SAM 2) | ||
| │ | ||
| ▼ | ||
| Quality Verification | ||
| (VLM Judge) |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Align the documentation with the implemented annotation contract.
The current workflow uses YOLO-World bounding boxes and Notebook 1 explicitly states “no SAM”; these documents instead prescribe Grounded-SAM 2 masks. This directs learners toward outputs the pipeline does not produce.
implementations/edge_case_image_generation/docs/context.md#L25-L34: replace Grounded-SAM 2 in the pipeline diagram with YOLO-World box annotation.implementations/edge_case_image_generation/docs/context.md#L262-L283: describe box outputs and manual/image-level labeling for diffuse effects instead of guaranteed masks and no manual annotation.implementations/edge_case_image_generation/docs/project_summary.md#L11-L11: replace the Grounded-SAM-2 workflow description with the configured YOLO-World annotation path.implementations/edge_case_image_generation/docs/quality_measure.md#L152-L159: replace SAM-mask IoU guidance with a box-compatible annotation-quality check.
📍 Affects 3 files
implementations/edge_case_image_generation/docs/context.md#L25-L34(this comment)implementations/edge_case_image_generation/docs/context.md#L262-L283implementations/edge_case_image_generation/docs/project_summary.md#L11-L11implementations/edge_case_image_generation/docs/quality_measure.md#L152-L159
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@implementations/edge_case_image_generation/docs/context.md` around lines 25 -
34, Align the documentation with the implemented YOLO-World box-annotation
contract: in implementations/edge_case_image_generation/docs/context.md lines
25-34, replace Grounded-SAM 2 with YOLO-World box annotation; in lines 262-283,
describe box outputs and manual or image-level labeling for diffuse effects
rather than guaranteed masks and no manual annotation; in
implementations/edge_case_image_generation/docs/project_summary.md line 11,
replace the Grounded-SAM-2 workflow with the configured YOLO-World path; and in
implementations/edge_case_image_generation/docs/quality_measure.md lines
152-159, replace SAM-mask IoU guidance with a box-compatible annotation-quality
check.
| Diffusers + ControlNet: ++[https://github.com/huggingface/diffusers](https://github.com/huggingface/diffusers)++ | ||
|
|
||
| FLUX.1: ++[https://github.com/black-forest-labs/flux](https://github.com/black-forest-labs/flux)++ | ||
|
|
||
| SDXL-Turbo: ++[https://huggingfGitHub - black-forest-labs/flux: Official inference repo for FLUX.1 modelsace.co/stabilityai/sdxl-turbo](https://huggingfGitHub)++ | ||
|
|
||
| Grounded-SAM-2: ++[https://github.com/IDEA-Research/Grounded-SAM-2](https://github.com/IDEA-Research/Grounded-SAM-2)++ | ||
|
|
||
| Depth-Anything-V2: ++[https://github.com/DepthAnything/Depth-Anything-V2](https://github.com/DepthAnything/Depth-Anything-V2)++ | ||
|
|
||
| VLM judges: ++[https://github.com/QwenLM/Qwen2.5-VL](https://github.com/QwenLM/Qwen2.5-VL)++ ++[https://github.com/OpenGVLab/InternVL](https://github.com/OpenGVLab/InternVL)++ | ||
|
|
||
| DriveLM / SkyScenes / DiffusionEngine show closely related patterns: ++[https://github.com/OpenDriveLab/DriveLM](https://github.com/OpenDriveLab/DriveLM)++ | ||
|
|
||
| ++[https://github.com/bravegroup/DiffusionEngine](https://github.com/bravegroup/DiffusionEngine)++ |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟡 Minor | ⚡ Quick win
Repair the reference links.
The ++ wrappers render literally in standard Markdown, and the SDXL-Turbo target on Line 30 is malformed. Replace these with verified standard Markdown links before publishing.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@implementations/edge_case_image_generation/docs/project_summary.md` around
lines 26 - 40, Update the reference list in the project summary to use standard
Markdown link syntax without literal ++ wrappers. Correct the malformed
SDXL-Turbo URL so it points to the verified stabilityai/sdxl-turbo repository,
and validate all listed reference links before publishing.
| " current = synthesize_one(\n", | ||
| " sample.image,\n", | ||
| " anomaly_id=anomaly_id,\n", | ||
| " method=method_map[anomaly_id],\n", | ||
| " cfg=cfg,\n", | ||
| " comparer=comparer,\n", | ||
| " depth=depth,\n", | ||
| " segmentation=seg,\n", | ||
| " project_root=PROJECT_ROOT,\n", | ||
| " seed_offset=attempt,\n", | ||
| " )\n", | ||
| " # Re-annotate the new edit so the next judge call sees fresh labels.\n", | ||
| " annotations[anomaly_id] = annotate_anomaly(current, anomaly_id, annotator)\n", | ||
| " results[anomaly_id] = current\n", | ||
| " _unload(\"comparer\", \"annotator\", \"depth_model\", \"segmenter\")\n", |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win
Persist retry artifacts.
A retried current result is judged but never saved with its refreshed annotation, so an accepted retry has no corresponding image/label artifact on disk. Save both under an attempt-specific directory before unloading the models.
Proposed fix
annotations[anomaly_id] = annotate_anomaly(current, anomaly_id, annotator)
+ retry_dir = output_dir / "nb1" / anomaly_id / f"attempt_{attempt}"
+ print(save_generation_artifact(sample, current.generated, retry_dir))
+ print(
+ save_annotation_artifact(
+ f"{sample.name}_{anomaly_id}",
+ annotations[anomaly_id],
+ retry_dir,
+ )
+ )
results[anomaly_id] = current📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| " current = synthesize_one(\n", | |
| " sample.image,\n", | |
| " anomaly_id=anomaly_id,\n", | |
| " method=method_map[anomaly_id],\n", | |
| " cfg=cfg,\n", | |
| " comparer=comparer,\n", | |
| " depth=depth,\n", | |
| " segmentation=seg,\n", | |
| " project_root=PROJECT_ROOT,\n", | |
| " seed_offset=attempt,\n", | |
| " )\n", | |
| " # Re-annotate the new edit so the next judge call sees fresh labels.\n", | |
| " annotations[anomaly_id] = annotate_anomaly(current, anomaly_id, annotator)\n", | |
| " results[anomaly_id] = current\n", | |
| " _unload(\"comparer\", \"annotator\", \"depth_model\", \"segmenter\")\n", | |
| " current = synthesize_one(\n", | |
| " sample.image,\n", | |
| " anomaly_id=anomaly_id,\n", | |
| " method=method_map[anomaly_id],\n", | |
| " cfg=cfg,\n", | |
| " comparer=comparer,\n", | |
| " depth=depth,\n", | |
| " segmentation=seg,\n", | |
| " project_root=PROJECT_ROOT,\n", | |
| " seed_offset=attempt,\n", | |
| " )\n", | |
| " # Re-annotate the new edit so the next judge call sees fresh labels.\n", | |
| " annotations[anomaly_id] = annotate_anomaly(current, anomaly_id, annotator)\n", | |
| " retry_dir = output_dir / \"nb1\" / anomaly_id / f\"attempt_{attempt}\"\n", | |
| " print(save_generation_artifact(sample, current.generated, retry_dir))\n", | |
| " print(\n", | |
| " save_annotation_artifact(\n", | |
| " f\"{sample.name}_{anomaly_id}\",\n", | |
| " annotations[anomaly_id],\n", | |
| " retry_dir,\n", | |
| " )\n", | |
| " )\n", | |
| " results[anomaly_id] = current\n", | |
| " _unload(\"comparer\", \"annotator\", \"depth_model\", \"segmenter\")\n", |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@implementations/edge_case_image_generation/notebooks/01_sample_data_generation.ipynb`
around lines 353 - 367, Persist each retried current result and its refreshed
annotations under an attempt-specific output directory before the _unload call.
Update the retry flow around synthesize_one, annotate_anomaly, and
results[anomaly_id] to save both the image and label artifacts for every
attempt, including accepted retries.
| "ipykernel>=7.3.0", | ||
| "clip", | ||
| ] |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
clip only resolves correctly under uv, and ipykernel is a runtime dependency. [tool.uv.sources] is not part of published metadata, so pip install . (or anyone consuming the built wheel) will pull the unrelated clip package from PyPI instead of the Ultralytics fork. Prefer a direct PEP 508 URL so the pin travels with the metadata, and move the notebook kernel to the extras/dev group where it already exists.
🔧 Proposed dependency fix
"requests>=2.31",
- "ipykernel>=7.3.0",
- "clip",
+ "clip @ git+https://github.com/ultralytics/CLIP.git",
]-[tool.uv.sources]
-clip = { git = "https://github.com/ultralytics/CLIP.git" }
-Also applies to: 36-37
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@implementations/edge_case_image_generation/pyproject.toml` around lines 22 -
24, Update the project dependencies in pyproject.toml: replace the bare clip
requirement with a direct PEP 508 URL dependency pointing to the intended
Ultralytics fork so built metadata installs the correct package, and remove
ipykernel from the runtime dependency list, retaining it only in the existing
extras/dev group.
| class_ids = cfg.get("seg_class_ids") | ||
| if class_ids and segmentation.label_map is not None: | ||
| labels = segmentation.label_map | ||
| if labels.shape != (height, width): | ||
| labels = cv2.resize(labels.astype(np.int32), (width, height), interpolation=cv2.INTER_NEAREST) | ||
| return np.isin(labels, list(class_ids)) |
There was a problem hiding this comment.
🩺 Stability & Availability | 🔴 Critical | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Confirm seg_class_ids is actually used by shipped anomaly configs.
fd -e yaml . implementations/edge_case_image_generation/configs --exec rg -n 'seg_class_ids|mode:\s*(seg_intersection|road_patch)|intersect_seg' {} +Repository: VectorInstitute/synthetic-data-bootcamp
Length of output: 3041
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Files:"
git ls-files | rg 'implementations/edge_case_image_generation/(src/edgecase_synthesis/conditioning.py|configs/.*\.yaml)$' | sed -n '1,120p'
echo
echo "conditioning outline:"
ast-grep outline implementations/edge_case_image_generation/src/edgecase_synthesis/conditioning.py --view compact | sed -n '1,160p' || true
echo
echo "conditioning lines 330-390:"
cat -n implementations/edge_case_image_generation/src/edgecase_synthesis/conditioning.py | sed -n '330,390p'
echo
echo "conditioning lines 200-330:"
cat -n implementations/edge_case_image_generation/src/edgecase_synthesis/conditioning.py | sed -n '200,330p'
echo
echo "call sites / annotations:"
rg -n 'build_anomaly_edit_mask|Anomaly|conditioning|seg_class_ids|road_patch|intersect_seg|label_map_path|label_map:' implementations/edge_case_image_generation --glob '*.py' --glob '*.yaml'Repository: VectorInstitute/synthetic-data-bootcamp
Length of output: 19320
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "default conditioning:"
cat -n implementations/edge_case_image_generation/configs/default/conditioning.yaml
echo
echo "dataset conditioning references:"
for f in implementations/edge_case_image_generation/configs/datasets/*/default.yaml implementations/edge_case_image_generation/configs/default/generation.yaml implementations/edge_case_image_generation/configs/config.yaml; do
[ -f "$f" ] || continue
echo "--- $f"
sed -n '1,220p' "$f"
done
echo
echo "generation call context:"
cat -n implementations/edge_case_image_generation/src/edgecase_synthesis/generation.py | sed -n '150,230p'
echo
echo "label map load paths and config snippets:"
cat -n implementations/edge_case_image_generation/src/edgecase_synthesis/conditioning.py | sed -n '160,215p'
for f in implementations/edge_case_image_generation/configs/datasets/*/data.yaml; do
echo "--- $f"
sed -n '1,80p' "$f"
done
echo
echo "OpenCV dtype support probe:"
python3 - <<'PY'
try:
import cv2, numpy as np
from collections import deque
vals = []
for dt in [np.uint8, np.int8, np.uint16, np.int16, np.uint32, np.int32, np.float32, np.float64]:
try:
a = np.zeros((10, 10), dtype=dt)
cv2.resize(a, (5, 5))
vals.append(f"{dt.__name__}: ok")
except Exception as e:
vals.append(f"{dt.__name__}: {type(e).__name__}: {str(e).splitlines()[0]}")
print("\n".join(vals))
except Exception as e:
print(f"probe_unavailable: {type(e).__name__}: {e}")
PYRepository: VectorInstitute/synthetic-data-bootcamp
Length of output: 12156
🌐 Web query:
OpenCV cv2.resize supported data types CV_8U CV_16U CV_16S CV_32F CV_64F CV_32S error code src data type = 4 is not supported
💡 Result:
The error src data type = 4 is not supported in cv2.resize arises because OpenCV does not support 32-bit signed integer (CV_32S) data types for the resize operation [1]. In OpenCV, matrix types are defined by a depth (the data type of the elements) and the number of channels. The depth codes and their corresponding numeric identifiers are defined in the OpenCV interface header [2]: CV_8U = 0 (unsigned 8-bit) CV_8S = 1 (signed 8-bit) CV_16U = 2 (unsigned 16-bit) CV_16S = 3 (signed 16-bit) CV_32S = 4 (signed 32-bit) CV_32F = 5 (32-bit float) CV_64F = 6 (64-bit float) When you see the error message src data type = 4 is not supported, it explicitly refers to CV_32S [1][2]. Other data types, such as CV_16F (often represented as type 23 in error messages [3]) or various signed 8-bit combinations (such as CV_8SC3, often reported as 17 [4][5]), may also trigger unsupported type errors depending on your specific OpenCV build and function overload. To resolve this issue, you must convert your image data to a supported type—typically CV_8U or CV_32F—before calling cv2.resize. For example, if you are working with integer masks or data stored in 32-bit signed integers, you can cast the array to a float32 type using NumPy: import cv2 import numpy as np # Assuming 'img' is your CV_32S image img_resized = cv2.resize(img.astype(np.float32), (new_width, new_height)) # If you need to return to integer format, you can cast back after resizing img_resized = img_resized.astype(np.int32) Note that if you are working with 32-bit unsigned integers (uint32), OpenCV also does not have direct support for these and will often attempt to interpret them as CV_32S, similarly leading to an unsupported type error [1]. In such cases, casting to float32 is likewise the standard workaround [1].
Citations:
- 1: [BUG] OpenCV does not support resizing of uint32 data types during model evaluation MouseLand/cellpose#937
- 2: https://github.com/opencv/opencv/blob/master/modules/core/include/opencv2/core/hal/interface.h
- 3: float16 is not supported (src data type = 23 is not supported) jacobgil/pytorch-grad-cam#493
- 4: https://stackoverflow.com/questions/68444848/cv2-error-opencv4-5-3-1-error-5bad-argument-in-function-resize
- 5: https://stackoverflow.com/questions/50420469/trying-to-use-opencv-resize-with-pillow-image
Resize the segmentation mask instead of the label_map.
segmentation.label_map is int32 by construction for both modeled and loaded label maps, but cv2.resize does not support CV_32S. In road_patch/seg_intersection paths, label-map shapes are normalized to diffusion input size while generation.py passes the fitted image size, so the resize branch is hit and crashes when seg_class_ids is used. Apply np.isin(labels, ...) first, resize the support mask, then return bool.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In
`@implementations/edge_case_image_generation/src/edgecase_synthesis/conditioning.py`
around lines 368 - 373, Update the segmentation handling around the
seg_class_ids branch to apply np.isin(labels, list(class_ids)) before resizing,
rather than resizing the int32 label_map. Resize the resulting boolean support
mask when its shape differs from (height, width), then return it as a boolean
mask while preserving the existing no-class and missing-label-map behavior.
| with zipfile.ZipFile(zip_path, "r") as zf: | ||
| zf.extractall(extract_dir) |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Zip-slip: validate member paths before extracting a remotely-fetched archive. archive_url comes from dataset config, so a crafted or compromised zip entry (../../…) can write outside extract_dir.
🛡️ Proposed fix
with zipfile.ZipFile(zip_path, "r") as zf:
- zf.extractall(extract_dir)
+ root = extract_dir.resolve()
+ for member in zf.namelist():
+ dest = (root / member).resolve()
+ if not dest.is_relative_to(root):
+ raise ValueError(f"Refusing unsafe zip entry: {member!r}")
+ zf.extractall(extract_dir)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| with zipfile.ZipFile(zip_path, "r") as zf: | |
| zf.extractall(extract_dir) | |
| with zipfile.ZipFile(zip_path, "r") as zf: | |
| root = extract_dir.resolve() | |
| for member in zf.namelist(): | |
| dest = (root / member).resolve() | |
| if not dest.is_relative_to(root): | |
| raise ValueError(f"Refusing unsafe zip entry: {member!r}") | |
| zf.extractall(extract_dir) |
🧰 Tools
🪛 ast-grep (0.45.0)
[error] 344-344: Calling extractall() on a zipfile.ZipFile or tarfile archive without validating member paths lets a crafted entry (e.g. "../../etc/passwd") write outside the destination directory (Zip Slip). Validate each member resolves inside the target directory, or pass a safe filter (tarfile: filter="data" / tarfile.data_filter).
Context: zf.extractall(extract_dir)
Note: [CWE-22] Improper Limitation of a Pathname to a Restricted Directory ('Path Traversal').
(archive-extractall-path-traversal-python)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@implementations/edge_case_image_generation/src/edgecase_synthesis/data.py`
around lines 344 - 345, Update the archive extraction flow around
zipfile.ZipFile and extractall to validate every member path before extraction,
rejecting absolute paths and normalized paths that escape extract_dir (including
traversal via symlinks if applicable). Only call extractall after all entries
pass validation, preserving extraction for safe archives.
Source: Linters/SAST tools
| elif isinstance(objects, list): | ||
| for obj in objects: | ||
| label = str(obj.get("label") or obj.get("category") or "defect") | ||
| bbox = obj.get("bbox") or obj.get("bbox_xyxy") or obj.get("box") | ||
| if bbox is not None: | ||
| boxes.append(_box_from_any(bbox, label, image)) |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Guard against bboxes with fewer than 4 values. The dict-object branch passes any bbox straight through; _box_from_any then indexes vals[2]/vals[3] and unpacks four values, so a short or malformed box raises IndexError/ValueError and aborts the whole ingest run. The annot branch already checks len(nums) >= 4.
🐛 Proposed fix
def _box_from_any(
bbox: Any,
label: str,
image: Image.Image | None,
) -> DetectionBox:
vals = [float(x) for x in list(bbox)[:4]]
+ if len(vals) < 4:
+ raise ValueError(f"Malformed bbox for {label!r}: {bbox!r}")and skip instead of crashing at the call site:
- if bbox is not None:
- boxes.append(_box_from_any(bbox, label, image))
+ if bbox is not None and len(list(bbox)) >= 4:
+ boxes.append(_box_from_any(bbox, label, image))Also applies to: 577-587
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@implementations/edge_case_image_generation/src/edgecase_synthesis/data.py`
around lines 550 - 555, Validate the bbox values in the objects-list branch
before calling _box_from_any: normalize or inspect the candidate bbox and skip
entries with fewer than four usable values, matching the existing annot branch
behavior. Apply the same guard to the additional bbox handling around the
corresponding branch, while preserving valid box processing and preventing
malformed entries from aborting ingestion.
| dtype_kw: dict[str, Any] = {} | ||
| # Prefer dtype= for newer transformers; torch_dtype still widely accepted. | ||
| dtype_kw["torch_dtype"] = self.torch_dtype | ||
|
|
||
| processor = AutoProcessor.from_pretrained( | ||
| self.model_id, | ||
| min_pixels=self.min_pixels, | ||
| max_pixels=self.max_pixels, | ||
| trust_remote_code=True, | ||
| ) | ||
| model = AutoModelForImageTextToText.from_pretrained( | ||
| self.model_id, | ||
| trust_remote_code=True, | ||
| device_map="auto" if self.device.type == "cuda" else None, | ||
| **dtype_kw, | ||
| ) |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Drop trust_remote_code=True — Qwen2.5-VL is natively supported.
model_id comes from YAML (judge.model_id), so this flag means any config edit or typo pointing at an untrusted Hub repo executes that repo's Python at load time. transformers 4.49 ships native Qwen2_5_VL support, so remote code isn't needed. The dtype_kw dict is also a no-op indirection now.
🔒️ Proposed fix
- dtype_kw: dict[str, Any] = {}
- # Prefer dtype= for newer transformers; torch_dtype still widely accepted.
- dtype_kw["torch_dtype"] = self.torch_dtype
-
processor = AutoProcessor.from_pretrained(
self.model_id,
min_pixels=self.min_pixels,
max_pixels=self.max_pixels,
- trust_remote_code=True,
)
model = AutoModelForImageTextToText.from_pretrained(
self.model_id,
- trust_remote_code=True,
device_map="auto" if self.device.type == "cuda" else None,
- **dtype_kw,
+ torch_dtype=self.torch_dtype,
)📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| dtype_kw: dict[str, Any] = {} | |
| # Prefer dtype= for newer transformers; torch_dtype still widely accepted. | |
| dtype_kw["torch_dtype"] = self.torch_dtype | |
| processor = AutoProcessor.from_pretrained( | |
| self.model_id, | |
| min_pixels=self.min_pixels, | |
| max_pixels=self.max_pixels, | |
| trust_remote_code=True, | |
| ) | |
| model = AutoModelForImageTextToText.from_pretrained( | |
| self.model_id, | |
| trust_remote_code=True, | |
| device_map="auto" if self.device.type == "cuda" else None, | |
| **dtype_kw, | |
| ) | |
| processor = AutoProcessor.from_pretrained( | |
| self.model_id, | |
| min_pixels=self.min_pixels, | |
| max_pixels=self.max_pixels, | |
| ) | |
| model = AutoModelForImageTextToText.from_pretrained( | |
| self.model_id, | |
| device_map="auto" if self.device.type == "cuda" else None, | |
| torch_dtype=self.torch_dtype, | |
| ) |
🧰 Tools
🪛 ast-grep (0.45.0)
[warning] 116-116: Loading a HuggingFace model, tokenizer, dataset or pipeline with trust_remote_code=True executes arbitrary code from the remote repository. Only enable it for fully trusted sources; prefer artifacts that do not require it.
Context: trust_remote_code=True
Note: [CWE-94] Improper Control of Generation of Code ('Code Injection').
(transformers-trust-remote-code-python)
[warning] 120-120: Loading a HuggingFace model, tokenizer, dataset or pipeline with trust_remote_code=True executes arbitrary code from the remote repository. Only enable it for fully trusted sources; prefer artifacts that do not require it.
Context: trust_remote_code=True
Note: [CWE-94] Improper Control of Generation of Code ('Code Injection').
(transformers-trust-remote-code-python)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@implementations/edge_case_image_generation/src/edgecase_synthesis/judge.py`
around lines 109 - 124, Update the model-loading flow using AutoProcessor and
AutoModelForImageTextToText to remove trust_remote_code=True from both calls,
relying on native Qwen2.5-VL support. Replace the dtype_kw indirection by
passing self.torch_dtype directly to the model loader, while preserving the
existing device_map and pixel configuration.
Source: Linters/SAST tools
| def _ensure_clip(self) -> None: | ||
| if self._clip is not None: | ||
| return | ||
| from transformers import CLIPModel, CLIPProcessor | ||
|
|
||
| clip_id = "openai/clip-vit-base-patch32" | ||
| processor = CLIPProcessor.from_pretrained(clip_id) | ||
| model = CLIPModel.from_pretrained(clip_id) | ||
| model.to(self.device).eval() | ||
| self._clip = (processor, model) | ||
| self._active_backend = "clip" | ||
| self.model_id = clip_id |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Overwriting self.model_id with the CLIP id poisons later Qwen loads.
_ensure_qwen loads self.model_id. Once a single CLIP fallback fires, self.model_id == "openai/clip-vit-base-patch32", so every subsequent judge() call on the same instance tries to load CLIP as an image-text-to-text model, fails, and silently degrades to CLIP forever — while backend still reports qwen_vl intent. Keep the configured id immutable and report the active id separately.
🐛 Proposed fix
def _ensure_clip(self) -> None:
if self._clip is not None:
return
from transformers import CLIPModel, CLIPProcessor
clip_id = "openai/clip-vit-base-patch32"
processor = CLIPProcessor.from_pretrained(clip_id)
model = CLIPModel.from_pretrained(clip_id)
model.to(self.device).eval()
self._clip = (processor, model)
self._active_backend = "clip"
- self.model_id = clip_id
+ self._active_model_id = clip_idThen set result.model_id = self._active_model_id or self.model_id in judge() (Line 188), initializing self._active_model_id = None in __init__ and clearing it in unload().
📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| def _ensure_clip(self) -> None: | |
| if self._clip is not None: | |
| return | |
| from transformers import CLIPModel, CLIPProcessor | |
| clip_id = "openai/clip-vit-base-patch32" | |
| processor = CLIPProcessor.from_pretrained(clip_id) | |
| model = CLIPModel.from_pretrained(clip_id) | |
| model.to(self.device).eval() | |
| self._clip = (processor, model) | |
| self._active_backend = "clip" | |
| self.model_id = clip_id | |
| def _ensure_clip(self) -> None: | |
| if self._clip is not None: | |
| return | |
| from transformers import CLIPModel, CLIPProcessor | |
| clip_id = "openai/clip-vit-base-patch32" | |
| processor = CLIPProcessor.from_pretrained(clip_id) | |
| model = CLIPModel.from_pretrained(clip_id) | |
| model.to(self.device).eval() | |
| self._clip = (processor, model) | |
| self._active_backend = "clip" | |
| self._active_model_id = clip_id |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@implementations/edge_case_image_generation/src/edgecase_synthesis/judge.py`
around lines 132 - 143, Keep the configured self.model_id immutable in
_ensure_clip; record the fallback model in a separate _active_model_id instead.
Initialize _active_model_id to None in __init__, clear it in unload(), and have
judge() report _active_model_id when set, otherwise self.model_id, so
_ensure_qwen continues loading the configured model.
Summary
A feature branch containing implementation of notebooks 1 (sample image generation process) and 1.5 (image generation method comparison) of the edge-case image synthesis reference implementation.
This includes a hydra managed config folder that is split by each dataset, with some default values in the default folder in the config.
Clickup Ticket(s):
Type of Change
Summary by CodeRabbit
New Features
Documentation