There is a noticeable difference in speed: Spark's orbitControl is much slower than MKK's GaussianSplats3D.

[cloris]

Using the same ply of SPLAT(about 2,878,840 splats) and default threejs Control three/addons/controls/OrbitControls.js, the performance difference is obvious: Spark's orbitControl is much slower than MKK's GaussianSplats3D.

[mrxz]

It's unclear to me what the relevance of the OrbitControls is when comparing performance. Just to be sure, you aren't talking about the movement speed of the camera/controls nor using OrbitControls for rendering on demand, right?

There are many factors that influence performance, so it would help if you could provide:

• Device used (including which GPU)
• Browser + version
• Spark version

And ideally provide a runnable example with the .ply in question or a substitute that exhibits the same issue.

Worth pointing out that Spark does the spherical harmonics and sorting asynchronous to the rendering. This means that large camera movements/rotations might show incorrect rendering results for a few frames. Especially on mobile GPUs (tile-based) in smartphones or Apple M series devices this sorting tends to take longer. See #205 (comment)

Using an FPS counter should tell you whether your scene is rendering at a lower rate or not.

[cloris]

@mrxz Thanks your feedback. I used Windows 11 PC (12th Gen Intel(R) Core(TM) i7-12700H 2.30 GHz 4GB GPU), Chrome 141.0.7390.123, Spark "@sparkjsdev/spark": "^0.1.10", and render continuously. Ply size is 1.288GB, splats count is 18,942,182. I recorded comparison videos — one with Spark and one with Mkk, FPS is shown in the left-top corner. Spark performance is noticeably lower than Mkk's

Spark
https://github.com/user-attachments/assets/46a3396b-77ab-4894-aec6-39f013030330

Mkk
https://github.com/user-attachments/assets/97434200-61ed-48ba-bda5-4d1bcb1aef80

.

[notthesame23]

First of all, thank you for developing Spark, it looks really promising feature and implementation wise and the documentation is also top notch!


The glitches after Orbit rotation is started, can be severe depending on scene...you can see really large splats, covering approx. 30% of the screen, really giving buggy/glitchy impressions.



To me, after much experimentation, it feels like on fast camera moves, the sorting algorithm might get busy with outdated camera positions and therefore has no time to prepare a camera position that would be much closer to the current position of the camera. On extreme fast moves this gets more pronounced. 


Is there no way to implement an algorithm that would detect fast camera moves and start droping outdated sorts or favouring a sort that is closer to the current camera position? Could you give advice on how to handle updates better from outside Spark (when autoUpdate is disabled)?
I would personally prefer to stick with Spark 1.x for now, until the build size is reduced back to an acceptable level (it grew from 700kB -> 5MB!), therefore I didn't test LOD yet. I don't expect this to be fixed by using LOD...as I don't think my relatively lightweight splats (below 2Mio) need LOD. But I would certainly test, if you think this had good chances of getting rid of the glitches.

As this is important to me, I thought I would bump on this and provide additional examples that make it easy to reproduce the visual issue.



I tried it with the latest spark 2.0.0. (your git commit de76c7e)(I built the dist myself)
and when I noticed the issue also could reproduce it with the Spark 1.1.10 (your git commit 792d6d1)(used your dist). So this must be a deeply rooted issue, not introduced by LOD with 2.0 or similar.


For testing I used a beefy old HP z820 Workstation with dual Xeon CPU, 128GB RAM, Win 10, Chrome browser and a 2080 TI 11GB VRAM GPU and I have to admit I also used a Macbook Air M1, 8GB, OSX, Chrome browser. But I truly tested, the issue is definitely not only present on the M1, and the workstation should definitely deliver enough horsepowers for this.

Steps to reproduce:

In SPARK:

(Sorry zipped PLY upload was too large for free GitHub: https://we.tl/t-r2oEGF6EdFexkJsn)

1. Go to your own editor example: https://sparkjs.dev/examples/#editor
Upload the provided uncompressed (MKK needs this) PLY relatively lightweight model with SH0 and approx only 2Mio Splats (Country_Hotel_Le_Querce_Italy_SH0_Z-Flipped.ply)

2. Open settings (top right) enable "Orbit controls"

3. Center model meaningfully and start orbiting....you will see the model in the center disappear sometimes, and sometimes large Splats covering up large parts of the viewing area. Both effects are only visible for split seconds. But definitely long enough to be very noticeable.



For reference, to see it working correctly: MKK Splat Viewer:



1. Go to
https://projects.markkellogg.org/threejs/demo_gaussian_splats_3d.php



2. On the lower left of the screen: under “View a .ply, .ksplat, or .splat file
Select “Choose file” and upload the same model (Country_Hotel_Le_Querce_Italy_SH0_Z-Flipped.ply)



3. Center model meaningfully and start orbiting…you will see only minor glitches, in quite small parts of the model. Really nowhere as disturbing as the glitches we currently see with Spark.




I also provide two video examples, to demonstrate the difference of what I would consider acceptable and what not.



MKK Screenrecord
https://github.com/user-attachments/assets/7df52aca-8075-476d-a67b-5882ccde91ba

Spark Screenrecord
https://github.com/user-attachments/assets/a5a187bc-600e-4c18-b6e1-18a34dd4322a

IMPORTANT: MKKs OrbitControls uses damping…yours does not. But, this does not help, believe me, I’ve implemented spark also with a damped variant of the OrbitControls and the issue with Spark is exactly the same.

It would be cool if Spark could be as snappy as DOOM without these glitches :)

Thank you!


Source for splat:
"Country Hotel Le Querce, Italy" by Dok11 (https://superspl.at/user?id=dok11)
Source: https://superspl.at/scene/8f4c0957
Licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/)

[mrxz]

To me, after much experimentation, it feels like on fast camera moves, the sorting algorithm might get busy with outdated camera positions and therefore has no time to prepare a camera position that would be much closer to the current position of the camera. On extreme fast moves this gets more pronounced.

In-flight sort operations aren't cancelled, but any queued sort operation is updated to reflect the most recent state. So at most one stale camera position is "blocking" the sorting. For extreme fast camera movements there will be a large delta between the camera pose the current splat ordering was computed for and the new camera pose.

Is there no way to implement an algorithm that would detect fast camera moves and start droping outdated sorts or favouring a sort that is closer to the current camera position?

As mentioned above, there is at most one scheduled sort operation that is being updated to the latest state, which can be seen as form of "dropping" sorts. Cancelling the in-flight sort operation isn't supported. There are effectively two atomic steps in the sorting process (on the GPU and the CPU), so cancelling the in-flight sort tends to not be beneficial. It will still consume these resources and still delay the next sort operation. And with the sort result being dropped, rendering will continue to happen with even staler data.

That said, if the cancellation can happen in between the two, it might help improve the situation. So this might be worth trying. Alternatively two sort operations could theoretically be in-flight if they were staggered, which could also help bring down latency.

Could you give advice on how to handle updates better from outside Spark (when autoUpdate is disabled)?

Spark has always been quite opinionated about the render loop and how it drives the updates and sort operations. I don't believe there are any examples or documentation that really goes into detail how to manually update things properly. @asundqui might be able to go into more detail, as v2 also introduced onDirty callbacks to close a gap in manually updating.

That said, you'd only indirectly influence the sorting logic this way, so I doubt it'll give you the control you're looking for.

I would personally prefer to stick with Spark 1.x for now, until the build size is reduced back to an acceptable level (it grew from 700kB -> 5MB!), therefore I didn't test LOD yet. I don't expect this to be fixed by using LOD...as I don't think my relatively lightweight splats (below 2Mio) need LOD. But I would certainly test, if you think this had good chances of getting rid of the glitches.

The increased bundle size should see a significant reduction in the next release (see #321). It's likely still going to be bigger than Spark 0.1 simply by the fact that it contains additional features like the LoD system, paged splats and a new splat representation.

Though, I don't see a reason why the LoD system would improve the issue at hand, as it's purely bound by the sorting. In theory LoD could mean rendering at a lower splat budget, meaning less splats to sort and thus faster sorting, but that's assuming you'd not be rendering the entire model (i.e. higher LoD levels).

For testing I used a beefy old HP z820 Workstation with dual Xeon CPU, 128GB RAM, Win 10, Chrome browser and a 2080 TI 11GB VRAM GPU and I have to admit I also used a Macbook Air M1, 8GB, OSX, Chrome browser. But I truly tested, the issue is definitely not only present on the M1, and the workstation should definitely deliver enough horsepowers for this.

Not to claim that either of these devices are underpowered or the issue at hand doesn't exist, but interestingly enough both you're test devices have characteristics that make Spark's sorting slow. For sorting Spark employs both the GPU and CPU, though more out of necessity than optimization. To enable global sorting and dyno modifiers, splats are accumulated and transformed on the GPU. Since GPU sorting is infeasible with WebGL2, a readback is performed to get per-splat depth which then gets sorted in a WebWorker.

This readback tends to be expensive on tile based GPUs, like in Apple M-series devices. Whereas the CPU sorting is single-threaded and thus benefits from CPUs with high single-core performance.

---

I'll try and see if either staggering sorts and/or cancelling in-flight sorts for large viewpoint changes improve the situation. The ideal solution would of course be fully GPU based sorting, especially with the way Spark updates splats, but that is still a ways away.

[notthesame23]

Thank you very much for the in-depth answer @mrxz !

Good to know that "any queued sort operation is updated to reflect the most recent state”. Still as you say, for very fast camera moves this might not be enough.
I haven’t tested spark on lower end mobile phones yet…but I fear the plopping effect might even be more pronounced. As other splat viewers don't have this issue, I really hope Spark can address this somehow (maybe with a flag for scene's that need highspeed cam algorithm?)



There’s on thing I noticed and I would like to share with you. Maybe there’s also something else not going quite as good for highspeed camera moves.


I noticed that with Spark 1.x the below render algorithm makes the glitches seemingly disappear completely:



(Spark autoUpdate = false)



this.renderer.setAnimationLoop(async () => {
    this.orbitControls.update();

    await this.spark.defaultView.prepare({
      scene: this.scene,
      camera: this.camera
    });

    this.renderer.render(this.scene, this.camera);
  });




The big downside of this is of course fps performance. On my MacBook M1 it drops from 58fps down to 20fps.
But visually this feels way better, despite the low frame rate. Plopping is gone.

Unfortunately this throws a ton of errors because it completely spams the sorting. This does not work anymore with Spark 2.x because, Spark 2.x checks if sorting is already running, and prevents adding a new sort.



Still, to me it isn’t obvious why this would work and result in the glitches disappear. But maybe it will give a hint to you of what might get improved to allow for super fast camera moves. 



One other interesting observation I made yesterday: on my Macbook M1, if I switch to use the correct devicePixelRatio for this device, which is 2, the glitches are quite a bit more pronounced.



this.renderer.setPixelRatio(this.devicePixelRatio);



Of course, for the splats, I don’t need a HDPI pixel ratio, 1 looks acceptable and performance is of course better.
Still, if this were a sole sorting logic issue, I wouldn’t necessarily expect devicePixelRatio to have such a big effect on the plopping.



Off-topic:

I also have a question regarding how to selectively antialias with Spark. I would like to combine splats with other opaque objects. These objects should get decent antialiasing….but just these objects, not the splats, ofc for performance and also visual reasons.
Could you please give me a hint were to see a working example of this maybe, or something that would point me i the right direction.?I tried for days by now…it is apparently not that simple as one would expect…or maybe me having zero formal education on WebGL2 and Three.js does not help either :)


[mrxz]

I haven’t tested spark on lower end mobile phones yet…but I fear the plopping effect might even be more pronounced. As other splat viewers don't have this issue, I really hope Spark can address this somehow (maybe with a flag for scene's that need highspeed cam algorithm?)

Generally you'd render less splats on a mobile phone than you would on other device classes. This in turn means that there's less work to do for both the GPU readback and the CPU sorting. Not an exact correlation, but this tends to even out with the lower performance of the device. In other words the behaviour is mostly comparable. Though, mobile phones have their own unique quirks, as some do boast high resolutions and refresh rates relative to their performance.

Spark is designed with global sorting and dynamic effects for splats in mind. As a result the current sorting approach is unfortunately more expensive to execute on many devices. There is some work to improve the CPU part of sorting (see #327), though without a full GPU based sorting solution the GPU work + readback remains. The main criterium is the number of frames of latency before a new sort result is available. This is always going to be at least 1, as that'll be the frame that kicks off a new sort operation.

Cancelling the in-flight sort operation might improve on this latency, though I haven't had the time to experiment with it yet.

The big downside of this is of course fps performance. On my MacBook M1 it drops from 58fps down to 20fps.
But visually this feels way better, despite the low frame rate. Plopping is gone.


Awaiting the prepare ensures that both an update and sort have taken place, essentially turning the asynchronous process into a sequence. As you've noticed this comes at a high performance cost. Also worth pointing out that the code as posted, isn't quite correct. Ideally all rendering work happens synchronously within a rAF callback, which is now broken up by the await. What's worse is that there is no guarantee that the next rAF callback happens after the work started by prior ones has completed. This is probably why you're seeing the errors regarding sorting.

So you'd actually have to schedule a rAF once you are ready to render (i.e. prepare has finished) and once done schedule the next sequence (prepare followed by rAF).

It goes without saying that this isn't really a solution, though it is interesting to consider the possibility of deliberately missing frame(s) if the camera delta is too large to await a sort operation. The main drawback is that Spark would need full control of the render loop, as by default it reacts to the Three.js render call, at which point the main colour buffer has likely already been cleared/overwritten. It remains subjective whether one would want a lag spike over a (few) frame(s) with stale sorting, but it wouldn't hurt if this could be implemented on the user-side.

This does not work anymore with Spark 2.x because, Spark 2.x checks if sorting is already running, and prevents adding a new sort.



Spark never allowed concurrent sorts, though the way this is handled has changed. As described above, the code snippet as posted has a problem where it would actually attempt triggering a prepare, and thus a new sort, without properly awaiting the completion of the previous one. If handled properly, the same pattern should work with Spark v2 as well.

Still, to me it isn’t obvious why this would work and result in the glitches disappear. But maybe it will give a hint to you of what might get improved to allow for super fast camera moves. 



Simply put: it awaits sorting to complete before rendering.

Of course, for the splats, I don’t need a HDPI pixel ratio, 1 looks acceptable and performance is of course better.
Still, if this were a sole sorting logic issue, I wouldn’t necessarily expect devicePixelRatio to have such a big effect on the plopping.



It really is purely a sorting issue, though there are two important factors:

1. Spark's sorting requires both GPU and CPU work
2. The issue can be observed in terms of "frames" not actual sorting latency

By increasing the pixel ratio, you're increase the GPU workload of rendering the frames. If this causes the framerate to drop below the refresh rate, the GPU might have little to no idle time. So it can take longer for the GPU step of the sorting process to even take place.

Additionally a lower frame rate, means the time a frame is shown becomes longer. So even if the actual sorting latency isn't much longer time-wise, it could still present itself visually for longer.

I also have a question regarding how to selectively antialias with Spark. I would like to combine splats with other opaque objects. These objects should get decent antialiasing….but just these objects, not the splats, ofc for performance and also visual reasons.

It's not possible to selectively apply antialiassing in WebGL, as this is a property of the framebuffer. Depending on how the opaque objects are intended to interact with the splats, there are some tricks possible (e.g. render splats to render target, blit to MSAA framebuffer and render opaque objects on top). But if you want proper depth testing and MSAA, that's sadly not possible.

Using post-processing to apply a screen-space antialiasing technique is an option, though not always ideal in terms of performance.