Scene package cooking pipeline#2544
Conversation
Nabla7
force-pushed
the
pim/feat/scene-cooking
branch
from
3514b0b to
fc150e3
Compare
Nabla7
changed the title
…into tmp/scene-cooking-ff
…into tmp/scene-cooking-ff
Zero callers in the repo; the runtime path is MujocoSimModule._compose_model, which calls add_entities_to_spec directly on the MjSpec. Drop the now-unused ScenePackage import.
Add reset_runtime_state to the ControlTask Protocol with a no-op default on BaseControlTask, so the coordinator can call it directly instead of probing each task with inspect.signature. All concrete tasks inherit BaseControlTask, so the direct call is uniform. Drops the now-unused import inspect.
The MujocoEngine in test_engine_request_reset_to_applies_pose_in_sim_loop was torn down via a hand-rolled try/finally. Move creation and disconnect into a fixture so teardown runs even on failure.
Quadric decimation collapses triangles but leaves the original input vertices in the buffer; exporting without dropping them ships millions of orphan vertices (~25x larger GLB, no geometry change).
…t.py json is stdlib and scipy is a core dependency (collision_policy.py already imports it at top level); the four inline imports had no justifying "lazy heavy dep" comment.
mujoco lives in the sim extra, not scene, so keeping the import inline is intentional (browser/rerun-only cooks shouldn't require it) -- just missing the comment explaining why, unlike other lazy imports in this package.
open3d, an equally heavy dependency, is already imported at module scope in this file, so keeping trimesh lazy in _write_glb with no comment was inconsistent.
Explain why this one import stays function-local: it avoids pulling open3d/trimesh into every other test in this module at collection time.
sidecar.py, cook.py, and normalize.py used %s-style logger calls while structured key/value logging dominates the rest of the repo.
_load_usd_mesh and load_scene_prims each ran an identical nested Python loop (per-face, per-triangle) to fan-triangulate USD n-gons -- slow on large meshes. Extract one shared _fan_triangulate() that does the same work with vectorized numpy indexing. Verified against the original nested-loop logic on 200 randomized face_counts/face_verts inputs (including degenerate 0/1/2-gons) and against a live pxr.Usd stage with a mixed quad+triangle mesh.
Verified against the full-project mypy run (ScenePrimMesh.vertices/ triangles are consumed across the whole scene_cooking package) and against a live pxr.Usd smoke test.
Traced each function's actual dtype from its call sites (float32 for hull geometry, float64 for the visual passthrough / scene-bounds path, mixed float precision in _write_mesh_obj which both feed). Verified with the full-project mypy run and a runtime smoke test of the hull/box/OBJ-writer helpers.
Traced each function's dtype from decide_for_prim's single call site (collision_export.py always passes float64 vertices / int32 triangles). Hull-producing paths (CoACD, sheet prisms, single-hull pass-through) mix float32/float64 vertex precision by design, so PrimDecision.hulls and _coacd_decompose keep NDArray[np.floating[Any]] where that's genuine. Verified with the full-project mypy run and a runtime smoke test of decide_for_prim.
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| mesh_cached = out_path.exists() and not rebake | ||
| objects_cached = objects_path.exists() and not rebake | ||
| if mesh_cached and objects_cached: | ||
| return BrowserCollisionCookResult( | ||
| path=out_path, | ||
| stats=inspect_scene_asset(out_path).to_json_dict(), | ||
| objects_path=objects_path, | ||
| ) |
There was a problem hiding this comment.
This cache gate still only checks whether collision.glb and objects.json already exist. When a scene is recooked into the same --output-dir after the cook sidecar changes, the effective collision policy from plan.collision_spec can change, but this branch returns the old browser collision artifacts. The new scene.meta.json can then describe the updated cook plan while browser picking and raycast geometry still use the previous skipped prims, split settings, or face targets. The cache needs to include the effective collision inputs, or these artifacts need to be rebuilt when the plan changes.
…' into pim/feat/scene-cooking
The scene-cooking tests need both but the tests group never installed them, so they always skipped in CI. The full scene extra can't be used: usd-core ships no linux-aarch64 wheel.
Importing coacd into a process that later loads open3d crashes open3d's extension module (clashing vendored native libs). Order the test-module imports open3d-first and force the same order in _coacd_decompose, which otherwise only needs numpy.
# Conflicts: # uv.lock
Satisfies main's new import-from-source codebase check.
| if mesh_cached and objects_cached: | ||
| return BrowserCollisionCookResult( | ||
| path=out_path, | ||
| stats=inspect_scene_asset(out_path).to_json_dict(), | ||
| objects_path=objects_path, | ||
| ) |
There was a problem hiding this comment.
--output-dir with rebake=False, this branch returns as soon as collision.glb and objects.json exist. It never checks the fresh collision_spec or BrowserCollisionSpec passed by the caller, so sidecar changes such as skipped prims, component splitting, or target_faces can be recorded in the newly written package metadata while browser picking and object lookup still use the previous artifacts. The cache needs to be keyed by the effective collision inputs, or these files need to be rebuilt when those inputs change.
has_entities lost to the more precise needs_blender visual-path check; the runtime floor probe was superseded by cook-time alignment. Keep make_raycasting_scene with a note: the pimsim mesh-camera port consumes it.
| mesh_cached = out_path.exists() and not rebake | ||
| objects_cached = objects_path.exists() and not rebake | ||
| if mesh_cached and objects_cached: | ||
| return BrowserCollisionCookResult( | ||
| path=out_path, | ||
| stats=inspect_scene_asset(out_path).to_json_dict(), | ||
| objects_path=objects_path, | ||
| ) |
There was a problem hiding this comment.
--output-dir with rebake=False, this branch returns as soon as collision.glb and objects.json exist. It does not compare the cached files against the current plan.collision_spec, split settings, skip overrides, source input, or BrowserCollisionSpec values such as target_faces. The cook can then write fresh package metadata for the new plan while browser picking and raycast geometry still use the previous artifacts. The cache needs to validate the effective cook inputs, or these artifacts need to be rebuilt when those inputs change.
| if mesh_cached: | ||
| stats = inspect_scene_asset(out_path).to_json_dict() | ||
| else: | ||
| mesh = _build_fused_collision_mesh( | ||
| prims, collision_spec or CollisionSpec.auto_discover(source) | ||
| ) | ||
| original_triangles = len(mesh.triangles) | ||
| target_faces = int(browser_spec.target_faces) | ||
| if target_faces > 0 and original_triangles > target_faces: | ||
| logger.info( | ||
| "browser collision: simplifying %s triangles -> %s", | ||
| original_triangles, | ||
| target_faces, | ||
| ) | ||
| mesh = mesh.simplify_quadric_decimation(target_number_of_triangles=target_faces) | ||
| mesh.remove_degenerate_triangles() | ||
| mesh.remove_duplicated_triangles() | ||
| mesh.remove_duplicated_vertices() | ||
| mesh.remove_non_manifold_edges() | ||
| _write_glb(mesh, out_path) | ||
| stats = inspect_scene_asset(out_path).to_json_dict() | ||
| stats["source_triangles"] = original_triangles | ||
| stats["target_faces"] = target_faces | ||
|
|
||
| objects = extract_scene_objects(prims) | ||
| if not objects_cached: |
There was a problem hiding this comment.
collision.glb exists, the function loads the new prims but keeps the old mesh. If only objects.json exists, it rebuilds the mesh from the new spec but keeps the old object sidecar because _write_objects_json is skipped. A recook after changing split, skip, or entity extraction policy can therefore produce a collision mesh and object lookup file that disagree with each other. These two outputs should be invalidated together, or each file should be checked against the exact inputs that produced it.
Wire SceneCookPlan.to_json_dict into cook stats (replacing the inline copy that had orphaned it), drop EntityCookPlan.to_json_dict now that the plan dump records an entity count, and drop make_raycasting_scene until the pimsim mesh-camera port brings its caller.
The cache gate only checked that collision.glb/objects.json existed, so recooking after a sidecar edit kept stale browser picking geometry while scene.meta.json claimed the new plan. Write a manifest of the effective inputs (source signature, alignment, browser + collision specs) next to the artifacts and rebuild both when it mismatches.
| st = source.stat() | ||
| return json.loads( # type: ignore[no-any-return] | ||
| json.dumps( | ||
| { | ||
| "manifest_version": _MANIFEST_VERSION, | ||
| "source": { |
There was a problem hiding this comment.
This manifest still identifies the source by name, size, and mtime only. If scene.glb is replaced in the same explicit --output-dir by a timestamp-preserving copy, rsync, or restored asset with the same byte size, _manifest_matches() can accept the old manifest and return the existing collision.glb and objects.json. The package can then describe the new cook while browser picking and raycast geometry still come from the previous source. Use a content digest or another content-based source identity before reusing these artifacts.
There was a problem hiding this comment.
fixed in cecae0b: the cook now writes a manifest of the effective inputs (source signature, alignment, browser + collision specs) next to the artifacts and rebuilds collision.glb + objects.json when it mismatches. covered by tests in browser/test_collision.py.
github-actions
Bot
added
the
ready-to-merge
2 participants
Problem
Runtime scene packages give DimOS a clean way to load simulated environments, but they still need to come from somewhere.
Raw authored assets are not runtime-ready. A
.blend,.glb, or.usdmay contain visual geometry, floors, walls, fixtures, repeated product meshes, source-frame transforms, and objects that should become runtime entities. Different consumers also need different outputs: Rerun wants a conservative GLB, browser tools need picking/raycast geometry, and MuJoCo needs collision geometry in world coordinates.This PR adds scene cooking: an offline pipeline for turning authored environment assets into DimOS scene packages.
Scene Cooking
Scene cooking takes a source asset, an alignment, and an optional sidecar, then writes a scene package with
scene.meta.jsonplus backend-specific artifacts.The sidecar is where scene-specific intent lives. It can say which source prims are floors, walls, fixtures, dynamic entities, repeated entity groups, or geometry that should be skipped from static collision.
The cook resolves that authored intent into a
SceneCookPlan. Artifact writers then consume the plan instead of repeating source-scene matching themselves.A cooked package can include:
browser/visual.rerun.glborbrowser/visual.babylon.glbobjects.json.mjbbinaries for faster loadingThe package model is intentionally extensible: splats, articulated assets, richer dynamic objects, and new simulator or viewer targets should fit as additional package artifacts.
Runtime API
Runtime code continues to consume
ScenePackage; it should not import the cooking pipeline.For MuJoCo, the normal cooked artifact is scene-only. The runtime attaches the robot MJCF when building the simulation, so the package remains robot-agnostic. Large scenes may additionally ship composed robot+scene
.mjbfiles, but those are specific to robot, spawn pose, entity policy, and scene revision.What This PR Adds
Adds the scene cooking package under
dimos.experimental.scene_cooking.Adds source normalization and inspection, including headless Blender export for
.blendsources so evaluated Geometry Nodes and collection instances are realized before cooking.Adds sidecar parsing for collision policy, explicit interactables, and repeated entity groups.
Adds
SceneCookPlanas the resolved internal boundary between authored scene intent and backend artifact writers.Adds browser visual cooking with explicit target profiles for
rerun,babylon, andgeneric.Adds browser collision/raycast cooking and object metadata output.
Adds MuJoCo static collision export, entity collision hull cooking, shared prototype collision reuse, and optional scene-only
.mjbcompilation.Adds focused docs:
dimos/experimental/scene_cooking/README.mdfor usagedimos/experimental/scene_cooking/ARCHITECTURE.mdfor the system overviewNon-Goals
This PR does not change the runtime scene package contract introduced by the scene loading PR.
It also does not make every large scene cheap to simulate as fully dynamic geometry. The current path supports static cooked collision plus separately represented entities. Very large dynamic object sets still need careful entity selection, prototype reuse, or precomposed MuJoCo binaries.
How To Test
Cook the office package for Rerun:
Run the cooked package through the G1 MuJoCo/Rerun path:
Prefer headless MuJoCo with Rerun native for normal testing.
mujocosimmodule.headless=falseopens the MuJoCo viewer but can run much slower.Review Guide
Start with:
dimos/experimental/scene_cooking/README.mddimos/experimental/scene_cooking/ARCHITECTURE.mddimos/experimental/scene_cooking/cook.pyThen review the core cook pieces:
package_config.pysidecar.pyplanning.pysource_assets/Then review the artifact writers:
Checks
Focused checks run locally across the branch included
ruff, mypy on the scene cooking/runtime metadata surface, scene-cooking pytest coverage, commit hooks, doclinks, and LFS checks.Recent focused commands:
Contributor License Agreement