feat(canvas) - Image To 3D #9342
Open
kappacommit wants to merge 21 commits into
Open
Conversation
…rotatable 3D splat
Adds a "Convert to 3D" canvas feature: rasterize a layer, generate a 3D Gaussian
splat with TripoSplat (MIT), inspect/adjust perspective in an in-canvas 3D viewport
(orbit camera, floor grid, axis gizmo), and commit the framed view back as a new
raster layer.
Backend:
- image_to_3d invocation wrapping vendored TripoSplat (snapshot_download + RawModel
cache wrapper, mixed-dtype device moves); remove_background toggle.
- Asset3DField/Asset3DOutput primitives.
- asset_files disk service + /api/v1/assets/i/{name} route for serving .ply assets.
Frontend:
- SplatOverlay (three.js + @sparkjsdev/spark): orbit/grid/axes/ViewHelper gizmo,
transparent WYSIWYG capture, commit to a raster layer at the source rect.
- "Convert to 3D" raster-layer submenu (isolate subject / keep background).
- runGraphAndReturnOutput for non-image node output; AbortSignal-based cancel.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…iposplat Resolved conflicts in 9 files: kept triposplat additions (Asset3D* fields/outputs, ImageTo3DInvocation, three/@sparkjsdev/spark/@types/three deps) while taking upstream for shared code and version bumps. Regenerated schema.ts from the merged backend and pnpm-lock.yaml from the merged package.json. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… live canvas Replace the modal 3D "studio" with a world-pinned transparent viewport so the splat composites live over the actual canvas content (WYSIWYG placement): - Footprint rect is now mutable session state: drag the frame edges to move, corner handles to resize (shift = keep aspect), orbit inside to angle the object. Stage pan/zoom stays live outside the frame for precise placement. - SplatScene renders transparent always (studio bg/grid/axes removed); the ViewHelper gizmo shows only on hover. Drawing buffer folds the stage scale into the pixel ratio (quantized, capped 4096) so the view is crisp at any canvas zoom. - Overlay pins to canvas world coords via $stageAttrs (same pattern as CanvasTextOverlay); mount moved inside CanvasManagerProviderGate. - Non-blocking loading state (frame + toolbar spinner instead of a backdrop), Escape cancels, commit bakes at the user-positioned rect. - rectTransforms: pure move/corner-resize helpers with vitest coverage. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… in Pillow 13) Pillow infers L / RGB / RGBA from the uint8 array shape, so behavior is identical; verified against Pillow 11.3 with warnings-as-errors. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
With invalidationBehavior 'immediately', RTK Query drops a forced refetch when the target query is already in flight. A session served entirely by the invocation cache completes within the lifetime of the SessionQueueStatus refetch triggered by its own enqueue/pending events, so the terminal event's invalidation was dropped and the stale mid-run counts stuck — pulsing progress bar and a phantom queued item until the next queue activity. Capture any in-flight getQueueStatus fetch before invalidating; on terminal statuses, invalidate once more after it settles so the final counts always land. Applies to both the owner and sanitized (multiuser) event branches. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…sposal, tests - Register TripoSplatModel in calc_model_size_by_data's dispatch (class moved to backend/image_util/triposplat/triposplat_model.py so model_util can import it) — previously the ~4-5GB pipeline was cached as size 0, invisible to RAM/VRAM room-making, OOMing next to a resident diffusion model. + regression test. - Add the new required asset_files arg at the four InvocationServices test construction sites that were missed (their modules errored at fixture setup). - Splat overlay session integrity: per-session token gates every completion write (no cross-session clobber), starting a new conversion aborts the prior one, the abort slot only clears if still owned, and completion preserves the user's current rect instead of the start-time rect. - Dispose the SparkRenderer in SplatScene.dispose() — it owns sort/LOD web workers and GPU-side state that leaked on every overlay session. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A fixed seed made "Convert to 3D" on an unchanged layer hit the invocation cache and return the identical splat — there was no way to try again from the canvas. Full determinism isn't on offer anyway (the vendored decode stage uses the unseeded global RNG), so trade the fixed seed for a fresh roll per click. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…100% three's ViewHelper.handleClick mixes visual coordinates (getBoundingClientRect, event.clientX/Y) with layout sizes (offsetWidth, fixed 128px dim). The splat viewport is CSS-scaled by the canvas stage transform, so those spaces disagree whenever stage zoom != 1 and the gizmo's click hotspot drifts off the drawn gizmo — axis clicks raycast into empty space and are silently ignored. Invert ViewHelper's mixed-space formula (pure helper + tests) and feed handleClick remapped coordinates so its NDC comes out correct at any zoom. Identity at scale 1. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… hints OrbitControls keeps the horizon level and clamps at the poles, so camera orbit alone cannot reach poses like a diagonal lean — and commit bakes what the camera sees, so unreachable angles were unbakeable. Alt+drag now arcball- rotates the splat root about its center on the camera's screen axes (successive two-axis increments compose to any orientation, including roll), claimed in the container's capture phase so OrbitControls never sees the gesture. Toolbar hint now lists the full control set. Also clear the gizmo-click anchor on pointerup so a blocked pointerdown can't pair with a stale one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… wrapping Alt+drag collided with the color-picker hotkey and holding a modifier while rotating was awkward — replace it with an Orbit view / Rotate object segmented toggle in the overlay toolbar. In object mode a plain left-drag rotates the splat (wheel zoom and right-drag pan still reach OrbitControls; gizmo-corner clicks pass through so view snapping keeps working). Mode resets per session and survives scene remounts on asset swap. Toolbar now wraps gracefully on narrow panels: hint text breaks only between segments and the button pair drops to its own row instead of clipping. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… hint contextmenu fires on right-button release, so every right-drag camera pan ended by popping the canvas context menu over the overlay. Swallow the event inside the viewport (preventDefault for the browser menu, stopPropagation for the app's canvas menu trigger) and list right-drag pan in the control hints. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
While orbiting/panning/zooming (or rotating the object), a small crosshair fades in at the point the gesture rotates around — controls.target for camera gestures, the object origin in rotate-object mode. Constant screen size (scaled by camera distance), drawn on top of the splat (no depth test — the pivot usually sits inside the body), faded via the render loop, and always hidden from capture() so it never bakes into a layer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…iposplat # Conflicts: # invokeai/frontend/web/src/services/api/schema.ts
…he object center Panning moves controls.target, but object rotation always spun the splat about its own origin — the panned pivot felt inert in rotate-object mode and the pivot marker meant different things per mode. Apply the rotation about controls.target (position swings via R·(pos − pivot) + pivot), so every gesture — orbit, pan, zoom, rotate-object — shares one pivot, which is what the crosshair marker now always shows. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The "transformers>=5.1.0" override predates upstream's own transformers 5.5 migration (it forced uv past compel 2.1.1's <5.0 cap, long since obsolete — upstream now ships compel>=2.4 with transformers>=5.5,<5.6). Worse, uv overrides REPLACE all declared requirements for the package, so this silently erased upstream's <5.6 pin and let resolution pick 5.6+, which breaks diffusers single-file SD loads. Restore upstream's line; pyproject now matches upstream exactly. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
kappacommit
requested review from
JPPhoto,
Pfannkuchensack,
blessedcoolant,
dunkeroni and
lstein
as code owners
github-actions
Bot
added
api
python
…he hint Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… generated schemas - python-checks: exclude the two vendored TripoSplat files from ruff (same "External code" pattern as mediapipe_face/mlsd/etc.) and fix import sorting in dependencies.py. - openapi-checks: the committed openapi.json was never regenerated with the triposplat nodes; regenerate it (in a fresh CI-equivalent 3.11 env — the dev venv's older pydantic drops plain-class docstrings from the schema) and run prettier over it like CI does. - typegen-checks: regenerate schema.ts from that openapi.json — restores the CacheStats @description block CI expects. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
… merge tests/app/routers/test_session_queue_workflow_call.py arrived in the latest upstream merge, written against upstream's InvocationServices signature — this branch adds a required asset_files parameter, so all 10 tests errored at fixture setup on every CI platform. Same fix as the other four fixtures. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Collaborator
|
Very cool! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Utilizes Triposplat to enable converting a 2D image to 3D. You can then move the 3D object around the canvas, resize it, adjust the viewing angle, and then recommit it as a new flat 2D raster layer. This enables the ability to make angle adjustments that historically have been difficult to do.
Triposplat is far from perfect. The general workflow that you should expect to follow in regards to this process is:
This PR vendors the triposplat library. It is two small .py files, MIT licensed so compatible with our repo to be vendored. The triposplat library has NO dependencies. Not even transformers or diffusers, making it a clean and simple addition
To render the 3D object on the canvas, 2 new dependencies are required:
Demos
16x speed - run on my RTX 4070 8gb Laptop GPU
Video.Project3.mp4
Video.Project2.mp4
Video.Project4.mp4
Video.Project.mp4
Related Issues / Discussions
N/A
QA Instructions
Triposplat's models are of varying quality. Sometimes they're nearly perfect, and sometimes they're extremely deformed. If you get a bad roll, simply discard and run the conversion again, each conversion is non-deterministic and will produce a new result (this is a limitation of the triposplat library)
Merge Plan
Checklist
What's Newcopy (if doing a release after this PR)