optimize coordinate validation, handling both -ve and OOB error

[musaqlain]

Achieved an ~8.2x speedup (9.5 min -> 1.1 min) in dataset validation by reducing disk I/O and vectorizing coordinate checks.

Changes

• Optimized I/O: Caches image dimensions to avoid repeated file opens.
• Vectorization: Replaced iterrows with Pandas boolean masking.
• Unified Error Reporting: Merged checks for negative/OOB coordinates into a single report that lists all invalid boxes to aid debugging.
• Polygon Support: Automatically converts non-rectangular geometries to valid bounding boxes.

Fixes #1244

AI-Assisted Development

• I used AI tools (e.g., GitHub Copilot, ChatGPT, etc.) in developing this PR
• I understand all the code I'm submitting, thanks

[bw4sz]

Thanks for your contribution, lets wait for the tests to pass.

[bw4sz]

@jveitchmichaelis, my general philosophy has always been to focus DeepForest on novice users and help avoid errors before they happen. So in this case there is a raise if there are bad coordinates in the annotations. I think that is still right and not too annoying, instead of just printing or warning. I think we want to prioritize, 'hey there is a problem here' versus streamlined and betting the user understands what they are doing. Agreed?

[bw4sz]

Agreed, thanks.

[bw4sz]

have a look at test failures.

[jveitchmichaelis]

Requested some of the LLM-style comments be cleaned up.

It would be interesting to run this with a profiler. I assume 90+% of the time is spent opening images (and so caching is the real win to avoid reloading images). Hence why I don't think vectorizing is the most important aspect here, even though it's definitely an improvement.

I tested filtering a 100M row numpy array on Colab, takes a couple of seconds. So minutes of runtime is likely I/O.

[jveitchmichaelis]

@bw4sz sorry to belabor this review. If the goal is to show users where issues are, I think it's important that the output is actually useful for debugging. Seeing that some number of boxes are malformed, or only their coordinates, doesn't help me fix what's wrong with my dataset. What do you think about reporting a complete list of image name / box issues? e.g.

Errors:
image_name, box coords, error

In the existing code we print an error string for all invalid boxes, while this PR reports a number only for negative coords:

errors.append(f"Found {bad_count} annotations with negative coordinates.")

@musaqlain If you move the logic for negative coordinates into the oob_mask I think that would be better.

[musaqlain]

completely makes sense, i have updated the logic to catch both the -ve coordinates as well as out of bound coordinates to give users a detailed report. thanks cc @jveitchmichaelis

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.87%. Comparing base (408e150) to head (aa9ad5a).
⚠️ Report is 4 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1285      +/-   ##
==========================================
- Coverage   86.87%   86.87%   -0.01%     
==========================================
  Files          24       24              
  Lines        3064     3184     +120     
==========================================
+ Hits         2662     2766     +104     
- Misses        402      418      +16     

Flag	Coverage Δ
unittests	86.87% <100.00%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[musaqlain]

PTAL cc @jveitchmichaelis

[jveitchmichaelis]

Could you resolve the conflicts and squash this one, please?

Then we can also have a look at merging #1373

[Copilot]

Pull request overview

This PR targets speeding up BoxDataset annotation validation for large datasets by reducing repeated image opens and switching coordinate checks from row-wise iteration to vectorized Pandas masking, while updating tests to reflect the new error reporting and polygon-to-bounds behavior.

Changes:

• Cache per-image dimensions to avoid repeated disk I/O during coordinate validation.
• Replace per-row coordinate validation with vectorized boolean masks and a unified “invalid bounding boxes” report.
• Update dataset tests to match the new error message and to accept non-rectangular geometries by deriving bounding-box bounds.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.

File	Description
src/deepforest/datasets/training.py	Refactors _validate_coordinates() to cache image sizes, add bounds columns when needed, and perform vectorized negative/OOB checks.
tests/test_datasets_training.py	Updates expected validation error text, changes polygon behavior test to assert derived bounds, and adds negative/OOB validation tests.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[musaqlain]

Could you resolve the conflicts and squash this one, please?

hi @jveitchmichaelis you can review it, conflicts are resolved and all the commits are squashed into 1 commit. thanks