Gaussian splat spherical harmonics support #12790

keyboardspecialist · 2025-08-01T08:16:12Z

Description

Adds spherical harmonic support for Gaussian splats in the SPZ format. Supports degrees 1-3 and auto detects what is available. This brings full SPZ support to CesiumJS.

Issue number and link

Fixes #12756

Testing plan

View the same asset tiled with 3 degrees of SH in the current main build versus the CI brand build:

No SH Support
SH Support

Author checklist

I have submitted a Contributor License Agreement
I have added my name to CONTRIBUTORS.md
I have updated CHANGES.md with a short summary of my change
I have added or updated unit tests to ensure consistent code coverage
I have updated the inline documentation, and included code examples where relevant
I have performed a self-review of my code

github-actions · 2025-08-01T08:16:33Z

Thank you for the pull request, @keyboardspecialist!

✅ We can confirm we have a CLA on file for you.

javagl · 2025-08-01T10:05:47Z

What is the SH degree of that example asset?

In any case, I think that proper SH support can hardly be implemented (and certainly not tested properly) based on the current SPZ loader, because it does not properly decode the SH data - unless the CesiumJS code is anticipating that wrong encoding, and compensating manually, in which case we just have to hope that it will never be fixed in the SPZ loader.

keyboardspecialist · 2025-08-01T13:30:44Z

The underlying SPZ wasm module decodes it. The raw decoded SH buffer is all we need.

Example is degree 3

javagl · 2025-08-01T14:24:26Z

Asked for clarification at drumath2237/spz-loader#36 (comment)

There is an effect that is clearly visible:

Differences to BabylonJS may be due to coordinate system differences (it's difficult)

Test data:

Shs Example.zip

keyboardspecialist · 2025-08-04T15:32:25Z

Babylon makes their lives simpler by rotating the world around the model.

The model's inverse rotation matrix is applied to the view angle which should give us the correct sampling.

One thing to keep in mind is we do apply a axis correction matrix which may contribute to what you are seeing. When I load our tiled SPZ into Babylon its sideways. That 90 degree rotation might be what you are seeing?

javagl · 2025-08-04T16:16:00Z

I think that the different coordinate system conventions can likely explain some differences.

Until now, I didn't dive into the maths behind the spherical harmonics. Yes, it's a bunch of coefficients that are mushed together with the view direction and eventually yield a "color". But I don't thoroughly understand the 'meaning of the values'. For example, I'd really like to create such a unit cube with spherical harmonics that looks

red from the right, and cyan from the left
green from the top and magenta from the bottom
blue from the front and orange from the back

This could help to see whether the SHs are taken into account correctly, and whether there are any orientation issues. (If someone knows, from the tip of the head, what the SH values would have to be for that, I'd create an example - until then, this is scheduled in the 'When I have way too much time on my hands'-section of my TODO list).

jjhembd · 2025-08-04T16:26:29Z

I'll put in a +1 for the "unit cube" test data. Without that, we don't have any way of knowing if the SH are rendered correctly, or wrongly but canceled by an opposite error in the data generation, or just wrong.

Here's what I see for the 2 Sandcastles.
No SH (before the PR):

With SH (after this PR):

At first glance it looks like a change in the scene lighting direction/intensity.

javagl · 2025-08-04T19:22:30Z

Creating predefined test data for this is a bit tricky, though. Deriving the right values from the shader code is close to impossible - there's a reason why there is some magic "iterative learner" blackbox generating these coefficients in "real" applications. I considered a lazy/sneaky approach, and just created a GLB of such a (solid) cube, recorded a video of that spinning it in all directions, and uploaded that to poly.cam, but ... that simply refuses to generate splats from that. (Even if it worked, it might not even possible to derive the right values from that).

A bit of trial and error gave the "red-cyan" effect, but only that, and only along the diagonal...

... so I think that it might be possible, and maybe even "easy" for someone who knows what he's doing. Not me, at least, not at the moment.

ggetz · 2025-08-08T16:04:51Z

Hi @keyboardspecialist, what's the status here? Any input on the testing data discussion above?

jjhembd · 2025-08-12T21:51:15Z

CHANGES.md

@@ -23,6 +23,7 @@
 - Expand the CustomShader Sample to support real-time modification of CustomShader. [#12702](https://github.com/CesiumGS/cesium/pull/12702)
 - Add wrapR property to Sampler and Texture3D, to support the newly added third dimension wrap.[#12701](https://github.com/CesiumGS/cesium/pull/12701)
 - Added the ability to load a specific changeset for iTwin Mesh Exports using `ITwinData.createTilesetFromIModelId` [#12778](https://github.com/CesiumGS/cesium/issues/12778)
+- Added spherical harmonics support for Gaussian Splats. Supports degrees 1, 2, and 3 in the SPZ format.


Where can we find a spec or description of the SPZ format? Is the format stable?
A link to the spec, perhaps in GltfVertexBufferLoader, could go a long way to shortening the spin-up time for future maintainers.

The thing that is closest to a spec is the README of https://github.com/nianticlabs/spz . The actual truth is in the code. Request for clarification are in nianticlabs/spz#42 . This does not yet cover the details of SHs and their layout, but maybe what's in the README is enough... (?)

I should reword that... we load them from SPZ currently, but the rendering side is not tied to SPZ.

wongshek · 2025-08-13T11:24:03Z

@jjhembd Here is a small ply data where the correctness of higher-order SH rendering can be clearly perceived. It would be great if it could help.

simple_board.ply.zip

Below is a comparison screenshot from Supersplat.

javagl · 2025-08-13T13:59:17Z

For example, I'd really like to create such a unit cube with spherical harmonics that looks

red from the right, and cyan from the left
green from the top and magenta from the bottom
blue from the front and orange from the back

This could help to see whether the SHs are taken into account correctly, and whether there are any orientation issues. (If someone knows, from the tip of the head, what the SH values would have to be for that, I'd create an example - until then, this is scheduled in the 'When I have way too much time on my hands'-section of my TODO list).

I guess that it's hard to just "come up" with the proper SH coefficients for that.

So I ported the GLSL code from the GaussianSplattingViewer to Java, took the view- and projection matrix and the camera positions for the top/bottom/left/right/front/back view configuration as the inputs, the desired color as the outputs, and interpreted the whole thing as an optimization problem that I fed into the Apache BOBYQAOptimizer.
Exactly what any sane person would do 🤡

The result as ASCII PLY, SPZ, GLB, and a matching tileset JSON and Sandcastle:

Splat SH orientation experiment 2025-08-13.zip

This is what it looks like:

blue from the front, yellow from the back
green from the top, magenta from the bottom
red from the right, cyan from the left

(This is rendered with the viewer from JSplat. BabylonJS shows the same for SPZ. For PLY, there seems to be an orientation issue, or I'm writing the coefficients in the wrong order... who knows what's "right" or "wrong" here...)

I tried to check the different views in CesiumJS:

Looks about right. The bottom is missing - let's just assume it's magenta.

ggetz · 2025-08-26T19:54:02Z

@keyboardspecialist @javagl @jjhembd Are we comfortable that the test case above proves out the CesiumJS implementation? If so, can we add it as a unit test?

weegeekps

Found some places where you need to update the SH prefixes to support the new and the old extensions.

packages/engine/Source/Scene/GaussianSplat3DTileContent.js

keyboardspecialist · 2025-08-27T15:00:40Z

@keyboardspecialist @javagl @jjhembd Are we comfortable that the test case above proves out the CesiumJS implementation? If so, can we add it as a unit test?

I think this will work. I can put it into a test.

ggetz · 2025-08-27T19:53:07Z

Thanks @keyboardspecialist, I'm taking a review pass on the code now.

For context, my understanding that this PR now also accounts for the new standard where glTF attribute names use extension-specific namespace prefixes. And this in additional to supporting the older style of prefixing attribute names with an underscore _. So everything here should be backwards compatible and now just account for cases where Spherical Harmonic data is already present, and when using the new attribute naming.

One thing to keep in mind is we do apply a axis correction matrix which may contribute to what you are seeing. When I load our tiled SPZ into Babylon its sideways. That 90 degree rotation might be what you are seeing?

@keyboardspecialist I have now seen some confusion about this from a few users. I assume this axis correction is accounting for the up-axis difference from glTF to 3D Tiles. I want to make sure I'm clear on this so I'm responding to users correctly.

Is it indeed the glTF to 3D Tiles up-axis correction?
Is the axis correction "baked-in" via the tiler, or a correction we're applying at runtime? I believe we are doing the latter everywhere else and I'd want to make sure we're being consistent.

ggetz · 2025-08-27T21:22:23Z

@keyboardspecialist It looks like this PR contains some changes from #12843. Could you please merge main into this branch so we're reviewing code specific to SH?

packages/engine/Source/Scene/GaussianSplat3DTileContent.js

ggetz · 2025-08-27T20:02:15Z

packages/engine/Source/Scene/GaussianSplat3DTileContent.js

+   * @type {number}
+   * @private
+   */
+  this.shDegree = 0;


Do we need both degree and coefficient as independent properties? It's possible to determine one based on the other, no?

You are correct it can be derived, but it's nice to not have to and keeps things more explicit. They are used in different ways when making checks and indexing.

Could we set one and use a getter function for the other? I'm mostly concerned about these values getting out of sync since they can both be set.

Oh, I see now they are set privately and only exposed via the getter. Should all be fine then. 👍

javagl · 2025-08-28T11:26:15Z

One thing to keep in mind is we do apply a axis correction matrix which may contribute to what you are seeing. When I load our tiled SPZ into Babylon its sideways. That 90 degree rotation might be what you are seeing?

I have now seen some confusion about this from a few users. I assume this axis correction is accounting for the up-axis difference from glTF to 3D Tiles. I want to make sure I'm clear on this so I'm responding to users correctly.

I think that the context of the first quote was very specifically the different appearances of the SH colors. When something is supposed to be "red from the front and green from the top", everything depends on what "front" and "top" are.

But more generally, there are two aspects of ~"coordinate systems" standing out:

there are cases where a whole model is rotated by 90°. This was reported at different places and in different contexts (sometimes in the forum, referring to ion output, and to some extent tracked in internal issues). This comes on top of the usual differences of up-axis conventions of glTF and 3D Tiles, and one root cause is that there already are different conventions for PLY.
once upon a time, there was a state where the splats themself had been rendered incorrectly, depending on certain matrices in the glTF and the tileset. Specifically, it was once necessary to 1. insert an axis conversion in glTF and 2. undo this axis conversion in the tileset (!). An example is shown in Jagged rendering for SPZ Gaussian Splats #12749 (comment) (but I haven't checked with the latest state)

Now one could argue that this is independent of this PR. After all, this PR is only about spherical harmonics. When there's an issue with the orientation, then you can just rotate the splats. Just use the right matrix at the right place. Well... it's not that easy. On top of the (always confusing and error prone) differences of up-axis-conventions, "rotating splats" also involves "rotating the spherical harmonics (~'coefficients')". Have a look at this rotateSH function to see what this might entail.

Checks 3 rotations around the central splat in the cube.

keyboardspecialist · 2025-08-28T22:03:53Z

Some of the axis issues were due to the old SPZ loader that always rotated the incoming data around X.

Indeed, rotating spherical harmonics is prohibitively complex and expensive at runtime which is why we avoid it. When we sample we apply the model rotation to the view, so we sample like it's in its original orientation.

…ate and added getters.

keyboardspecialist · 2025-08-28T23:06:23Z

Thanks @keyboardspecialist, I'm taking a review pass on the code now.

For context, my understanding that this PR now also accounts for the new standard where glTF attribute names use extension-specific namespace prefixes. And this in additional to supporting the older style of prefixing attribute names with an underscore _. So everything here should be backwards compatible and now just account for cases where Spherical Harmonic data is already present, and when using the new attribute naming.

One thing to keep in mind is we do apply a axis correction matrix which may contribute to what you are seeing. When I load our tiled SPZ into Babylon its sideways. That 90 degree rotation might be what you are seeing?

@keyboardspecialist I have now seen some confusion about this from a few users. I assume this axis correction is accounting for the up-axis difference from glTF to 3D Tiles. I want to make sure I'm clear on this so I'm responding to users correctly.

Is it indeed the glTF to 3D Tiles up-axis correction?

Is the axis correction "baked-in" via the tiler, or a correction we're applying at runtime? I believe we are doing the latter everywhere else and I'd want to make sure we're being consistent.

Yes
It's applied at runtime same as the Model pipeline

packages/engine/Source/Scene/GaussianSplat3DTileContent.js

CHANGES.md

packages/engine/Source/Scene/VertexAttributeSemantic.js

ggetz · 2025-08-29T15:47:39Z

packages/engine/Source/Scene/GaussianSplatPrimitive.js

+              switch (tile.content.sphericalHarmonicsDegree) {
+                case 1:
+                  coefs = 9;
+                  break;
+                case 2:
+                  coefs = 24;
+                  break;
+                case 3:
+                  coefs = 45;
+              }


Should we throw an error here in case the SH is not 1, 2, or 3? Or is that unlikely to happen (both with our tiler or third-party ones)?

I think it's ok not to check here since the data should be checked and ready for rendering. 3 is the max anyone has ever trained that I'm aware of. I could see adding a check in the loader and notifying the user.

With our data coming from SPZ it shouldn't be an issue.

packages/engine/Source/Scene/GaussianSplatPrimitive.js

packages/engine/Source/Scene/GaussianSplat3DTileContent.js

ggetz · 2025-08-29T16:04:10Z

packages/engine/Source/Scene/GaussianSplat3DTileContent.js

+  const totalLength = tileContent.pointsLength * (coefs * (2 / 3)); //3 packs into 2
+  const packedData = new Uint32Array(totalLength);
+
+  const shAttributes = tileContent.splatPrimitive.attributes.filter((attr) =>


I don't think the attributes property was properly defined in the GaussianSplatPrimitive class. Looks like it was getting set as a side-effect of a helper function in ModelUtility. Please add and document it.

It's defined in the GS content class. Very ambiguous, but it's a reference to the glTF primitive. I renamed it for better clarity.

packages/engine/Source/Scene/GaussianSplat3DTileContent.js

Fix param list description

keyboardspecialist added 2 commits August 1, 2025 02:57

clean up

7c6a2a8

SH changes

a78ab94

keyboardspecialist requested a review from ggetz August 1, 2025 13:43

ggetz assigned keyboardspecialist Aug 1, 2025

jjhembd added the category - gaussian splatting label Aug 6, 2025

jjhembd reviewed Aug 12, 2025

View reviewed changes

keyboardspecialist added 2 commits August 26, 2025 14:07

Merge branch 'main' into gsplat-sh-release

94165d2

update changes for clarity

faf0111

keyboardspecialist added 2 commits August 26, 2025 15:59

gltf spec updates

2448e21

fix extension check

f9f99aa

weegeekps reviewed Aug 26, 2025

View reviewed changes

packages/engine/Source/Scene/GaussianSplat3DTileContent.js Outdated Show resolved Hide resolved

packages/engine/Source/Scene/GaussianSplat3DTileContent.js Outdated Show resolved Hide resolved

packages/engine/Source/Scene/GaussianSplat3DTileContent.js Outdated Show resolved Hide resolved

spec update support for SH content

42e0c8d

keyboardspecialist added 4 commits August 27, 2025 10:12

wording

81d252b

Merge branch 'main' into gsplat-sh-release

9026e4a

extension spelling

1d03ac0

spz extension check clean up and sync

40e4fe6

ggetz mentioned this pull request Aug 27, 2025

Add support for latest 3DGS extensions and deprecate original extension #12843

Merged

6 tasks

ggetz requested changes Aug 27, 2025

View reviewed changes

keyboardspecialist added 3 commits August 28, 2025 16:02

Add spherical harmonic unit test.

8ecb57c

Checks 3 rotations around the central splat in the cube.

sh unit test data

0b099c3

Fixed re-enabled tests

3e0ac2b

keyboardspecialist added 2 commits August 28, 2025 17:39

Spherical harmonic naming clarity. Moved read only properties to priv…

c3cc59c

…ate and added getters.

Merge branch 'main' into gsplat-sh-release

abc8209

keyboardspecialist requested a review from ggetz August 29, 2025 12:30

ggetz requested changes Aug 29, 2025

View reviewed changes

ggetz added the priority - next release label Aug 29, 2025

keyboardspecialist added 7 commits August 29, 2025 13:17

comment cleanup

a18a935

rename gaussianSplatSHTexture sphericalHarmonicsTexture

a0baf6c

Fix param list description

param description

b725db4

object param case

2836299

VertexAttributeSemantic param description

38bba9a

renamed splatPrimitive to gltfPrimitive for clarity

032fb1f

Fix bad merge

ae0adcc

Gaussian splat spherical harmonics support #12790

Are you sure you want to change the base?

Gaussian splat spherical harmonics support #12790

Uh oh!

Conversation

keyboardspecialist commented Aug 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issue number and link

Testing plan

Author checklist

Uh oh!

github-actions bot commented Aug 1, 2025

Uh oh!

javagl commented Aug 1, 2025

Uh oh!

keyboardspecialist commented Aug 1, 2025

Uh oh!

javagl commented Aug 1, 2025

Uh oh!

keyboardspecialist commented Aug 4, 2025

Uh oh!

javagl commented Aug 4, 2025

Uh oh!

jjhembd commented Aug 4, 2025

Uh oh!

javagl commented Aug 4, 2025

Uh oh!

ggetz commented Aug 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wongshek commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

javagl commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ggetz commented Aug 26, 2025

Uh oh!

weegeekps left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

keyboardspecialist commented Aug 27, 2025

Uh oh!

ggetz commented Aug 27, 2025

Uh oh!

ggetz commented Aug 27, 2025

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

javagl commented Aug 28, 2025

Uh oh!

keyboardspecialist commented Aug 28, 2025

Uh oh!

keyboardspecialist commented Aug 28, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

keyboardspecialist commented Aug 1, 2025 •

edited

Loading

wongshek commented Aug 13, 2025 •

edited

Loading

javagl commented Aug 13, 2025 •

edited

Loading