For reference: glTF Extension KHR_gaussian_splatting for 3DGS scene representation

[jo-chemla]

Is your feature request related to a problem? Please describe.

OGC 3D Tiles is a great spec for tiling massive meshes, pointclouds or large instance arrays. Some great people eg from Cesium and jdultra are starting to extend the glTF spec to allow for the representation of a gaussian splat scene. This then allows for the tiling and streaming of a gaussian-splat tiled scene.

I'm opening this discussion thread here in case you'd like to keep track of the current state of things outside the 3DTilesRendererJS repo. Feel free to close it and watch the universal splat thread instead, since this 3DGS feature seems out-of-scope for this library at the moment.

Describe the solution you'd like

Here mkkellogg/GaussianSplats3D#47 is a mega discussion regarding a universal gaussian-splat format, with a lot of great takes and mentions - mostly regarding compression, like Niantic SPZ, and standardization.
A recent comment by lilleyse (Cesium team member) redirects to the draft KHR_gaussian_splatting glTF extension as well as an implementation within CesiumJS in the experimental splat-shader branch

Describe alternatives you've considered

CesiumJS viewer will probably be the first reference implementation for that KHR_gaussian_splatting extension + OGC-3D-tiles

[gkjohnson]

Thanks for the heads up - if this is looking like it's going to become a real KHR_ extension then I think it would be worth making an issue in the three.js repo. This is something that has far more use and interest beyond this project - and mostly I don't want to maintain a gaussian splat loader and renderer in this project since it can get fairly complex. One hurdle I see is that as far as I understand all splats need to be sorted together so it can be difficult to build a single "self-contained" renderable object for splats.

[jo-chemla]

Thanks for the feedback, and indeed the three.js repo would probably be a better place for this. I'll cross-post there and see if anyone interested can take part in this standardization effort! - cross-post here mrdoob/three.js#30101

And it is correct that depending on the 3DGS renderer implementation, also to my understanding, splats have to be sorted from the camera perspective for the integration of the splats transparency to be accounted for physically correctly. I think there are more naive implementations where splats ordering is not necessary, not sure if they made their way to an existing JS renderer implementation.

If or once that KHR_gaussian_splatting extension matures, it is possible there will be some specificities for a 3D-tiles-of-3dgs scene support, for the data structure or renderer, but this is probably a few more months too early. It is also possible there won't be a need for any adaptation if the 3D-tiles is used merely as a container of gltf splats, if a renderer for splats is integrated.