LangSplatV2 LERF OVS Evaluation#56
Merged
Merged
Conversation
- Added `train_eval.sh` and `train_scenes.sh` for streamlined training and evaluation processes. - Introduced `download_data.py` for downloading the LERF-OVS dataset, along with `eval_lerf.py` for model evaluation on this dataset. - Created `environment.yml` to define the Conda environment for LangSplatV2, including necessary dependencies. - Updated `.gitignore` to exclude new log and result directories. - Enhanced `train_langsplatv2.py` to save final checkpoints and log paths for better tracking of training progress. - Various alterations to the training, data generation and evaluation to be more inline with the original implementation results. These changes improve the usability and functionality of the LangSplatV2 framework for open-vocabulary segmentation tasks. Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a LERF-OVS evaluation workflow and updates the LangSplatV2 preprocessing/training pipeline to better match the original LangSplatV2 feature/mask generation and checkpointing conventions (Fixes #44).
Changes:
- Add LERF-OVS dataset download utilities and a full evaluation script (OpenCLIP relevancy, mIoU + localization).
- Add SAM1 multi-scale mask generation, shared mask post-processing utilities, and an “import original LangSplatV2 features” transform for A/B comparisons.
- Improve training/preprocessing ergonomics (top-level
final_checkpoint.pt, GPU-based CLIP crop/resize, codebook init from all levels, updated deps/docs).
Reviewed changes
Copilot reviewed 27 out of 27 changed files in this pull request and generated 8 comments.
Show a summary per file
| File | Description |
|---|---|
| open_vocabulary_segmentation/langsplatv2/scripts/train_langsplatv2.py | Saves a top-level final checkpoint for easier evaluation consumption. |
| open_vocabulary_segmentation/langsplatv2/pyproject.toml | Updates Python requirement and adds evaluation/preprocess dependencies. |
| open_vocabulary_segmentation/langsplatv2/langsplatv2/vq_utils.py | Adds helper to load CLIP features from all scales for codebook init. |
| open_vocabulary_segmentation/langsplatv2/langsplatv2/training/trainer.py | Enables codebook initialization from all levels via config flag. |
| open_vocabulary_segmentation/langsplatv2/langsplatv2/training/langsplatv2_writer.py | Adds save_final_checkpoint() writing final_checkpoint.pt at run root. |
| open_vocabulary_segmentation/langsplatv2/langsplatv2/training/dataset.py | Adds cache warmup diagnostics; changes GT feature-map init to zeros. |
| open_vocabulary_segmentation/langsplatv2/langsplatv2/scene_transforms/multi_scale_sam_masks.py | Refactors to shared mask utils, adds diagnostics, updates cache versioning/log suppression. |
| open_vocabulary_segmentation/langsplatv2/langsplatv2/scene_transforms/multi_scale_sam1_masks.py | New SAM1 multi-scale mask transform matching original LangSplatV2 settings. |
| open_vocabulary_segmentation/langsplatv2/langsplatv2/scene_transforms/mask_utils.py | New shared post-processing utilities (cross-crop NMS, mask NMS, small-region cleanup). |
| open_vocabulary_segmentation/langsplatv2/langsplatv2/scene_transforms/import_original_features.py | New transform to import original repo’s _f.npy/_s.npy features into cache. |
| open_vocabulary_segmentation/langsplatv2/langsplatv2/scene_transforms/clip_feature_encoding.py | Moves crop/mask/pad/resize onto GPU to better match original feature extraction. |
| open_vocabulary_segmentation/langsplatv2/langsplatv2/scene_transforms/init.py | Exposes new transforms (SAM1 + original-feature import). |
| open_vocabulary_segmentation/langsplatv2/langsplatv2/loss.py | Updates loss to match original “masked then mean over all pixels” behavior; adds valid-only metric. |
| open_vocabulary_segmentation/langsplatv2/langsplatv2/evaluation/openclip_relevancy.py | Adds OpenCLIP relevancy computation matching original eval logic. |
| open_vocabulary_segmentation/langsplatv2/langsplatv2/evaluation/datasets/util.py | Adds Google Drive download helper for evaluation datasets. |
| open_vocabulary_segmentation/langsplatv2/langsplatv2/evaluation/datasets/lerf.py | Adds LERF-OVS dataset download + path helpers. |
| open_vocabulary_segmentation/langsplatv2/langsplatv2/evaluation/datasets/init.py | Adds dataset-root management for evaluation utilities. |
| open_vocabulary_segmentation/langsplatv2/langsplatv2/evaluation/init.py | Initializes evaluation package. |
| open_vocabulary_segmentation/langsplatv2/evaluation/lerf_ovs/eval_lerf.py | New end-to-end LERF-OVS evaluation script (render, relevancy, mIoU, localization, viz). |
| open_vocabulary_segmentation/langsplatv2/evaluation/lerf_ovs/download_data.py | CLI to download LERF-OVS dataset via tyro. |
| open_vocabulary_segmentation/langsplatv2/evaluation/lerf_ovs/batch_train_eval_langsplat.sh | Batch training script for LERF scenes/levels and checkpoint collection. |
| open_vocabulary_segmentation/langsplatv2/evaluation/lerf_ovs/batch_reconstruct_eval_scenes.sh | Batch reconstruction script for LERF scenes. |
| open_vocabulary_segmentation/langsplatv2/evaluation/lerf_ovs/README.md | Documentation for dataset download, reconstruction, training, and evaluation. |
| open_vocabulary_segmentation/langsplatv2/environment.yml | Adds a conda env capturing dependencies for evaluation/preprocessing. |
| open_vocabulary_segmentation/langsplatv2/README.md | Updates top-level docs for SAM1/SAM2, scripts path, outputs, and references. |
| open_vocabulary_segmentation/langsplatv2/.gitignore | Ignores reconstruction logs and results directories. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
…nsforms/mask_utils.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
…nsforms/clip_feature_encoding.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
add a comment for a small logic issue found in a PR review Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
…dataset.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
swahtz
force-pushed
the
langsplatv2_lerf_eval
branch
from
f55f66d to
90801e3
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 29 out of 29 changed files in this pull request and generated 8 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
You can also share your feedback on Copilot code review. Take the survey.
…n/openclip_relevancy.py Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
Signed-off-by: Jonathan Swartz <jonathan@jswartz.info>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #44