GLTFLoader support KHR_texture_procedurals

[DennisSmolek]

Description

GLTF is gearing up to be the final mile delivery system for 3D platforms. One of the key goals expressed is better interoperability between GLTF and OpenUSD. One of the primary ways to do that is supporting MaterialX.

To do that, GLTF first wants to support procedural materials (including MaterialX) through a JSON defined node graph system for materials: https://github.com/KhronosGroup/glTF/tree/KHR_texture_procedurals/extensions/2.0/Khronos/KHR_texture_procedurals

From there, different systems/versions of procedural materials can be loaded, (however this primarily seems to be aimed at specifying versions of MaterialX, though I could see it supporting other systems)

Outline of GLTF/MaterialX 1.39

Link to the PR

At Siggraph 2024 Asia this was highlighted as done/near done and also one of the key goals within gltf. This is 100% happening and I think threejs should be a leader of it's implementation or at a minimum prepared for it.

This issue is first to mark the item within threeJS and hopefully could act as a holder of other issues related to it. I searched (a little) and found some issues mentioning parts of this, but it's hard to understand what is happening without following all open issues.

Solution

The first task would be parsing the GLTF Json into a ThreeJS material using TSL and the correct material type.

I'm not sure if the GLTF system can tag in a way if a material would need to be a basic, standard, or Physical material, however it could be built as a standard material and then rebuilt as a basic if it only has the basic inputs or a physical if it has the more complex ones. (implementation discussion)

I know there is currently a materialX loader, however, that is parsing the XML matx files and not this converted JSON. A new parser would have to be put together but could use that as a starting point.

I am willing to start working on this in my desire to do more for things within gltf, however I didn't want to start if @sunag already had something in the works

Alternatives

Additional context

There are a million links to MaterialX stuff. But MaterialXLearn might be the easiest to link to all the other resources, see graphs, editors, etc.

[sunag]

Would the proposal be to have a MaterialXLoader extension for GLTFLoader as happened with DRACOLoader?

[DennisSmolek]

Would the proposal be to have a MaterialXLoader extension for GLTFLoader as happened with DRACOLoader?

This is actually the direction I was thinking.
Expand the parse functionality of the loader to read json and use the same nodes.

the question I have is if the work should be done as a plugin/extension add on or as a pr modification to the core loader itself.

I attended many talks where they discussed this being a priority but in at least reading what’s public on github it’s a bit early still.

there are a few changes to the spec even I want to propose.

If the work was done as an add-on (or even totally external) the changes in the spec wouldn’t be as impactful to core.

—-

This is something I’m super interested in and helps me learn the core of Nodes/TSL so I’m willing to champion the effort but if you (@sunag) would prefer to oversee the loader I’d understand.

there are a handful of major GLTF efforts ongoing and other platforms already have working versions , this felt like a good place Three could also be involved and lead on

[DennisSmolek]

@sunag if you get the chance can you take a look at the spec?
You made a MatX parser and a JSON format for nodes so I think your feedback would be helpful.

The issues I have:

1. Node references are indexes not names outlined here this isn’t so much of an issue persay, but the matx graph uses name references making the order of the nodes less important. This system is a bit brittle to changes. Output and other references are by index too.
2. UI data is dropped. MaterialX graphs (can) include many optional UI data points like position (x,y), min, max, etc. which is useful for graph systems. If the goal is interopt, stripping this makes it a pain to load a matx from a gltf into a graph editor. (Be it a web one or any of the existing ones) The spec already allows for a description which is only relevant for editing. I think if a node has this data we should include it. A ui: { min: 0, max: 2 } prop makes sense to me.

Probably more but that’s what stands out…

[sunag]

1. I prefer nodes in a single array, they are easier to control when serializable, you could have a single array to handle the inputs and nodes, the uniforms would just be a different type, in a node system uniforms should also be a node.
   I like indexes, in a compiled environment this works well, the label is an optional or utility for UI. If it is added with dependency precedence, where those that have more dependencies load first, this helps performance in the Loader. Now if the idea is that a person will edit/create the JSON, then numeric indexing shouldn't help.

2. ui: { min: 0, max: 2 } What do these values ​​mean?
   I think it can be useful, if we have the bounds of each Node in the graph too {x,y,width,height} considering some scale standard, we have to consider that the bound of a Node is different from another editor, with this data the Loader can simply centralize the nodes individually in the UI.

It would be great to have some help with this, I'm not currently working on it, any help is welcome.

[DennisSmolek]

in a node system uniforms should also be a node.

Uniforms are an option for input nodes, it's a boolean tag I believe.

2. ui: { min: 0, max: 2 } What do these values ​​mean?

consider:

<input name="dirt_color" type="color3" value="0.56372, 0.56372, 0.56372" uiname="Dirt Color" uifolder="Dirt" />
    <input name="dirt_amount" type="float" value="0.248" uimin="0" uimax="1" uiname="Dirt Amount" uifolder="Dirt" />
    <input name="uvtiling" type="float" value="3" uisoftmin="1" uisoftmax="16" uiname="UV Tiling" uifolder="Texturing" />

The inputs have a variety of possible parameters related to how they would be used in a UI system. Min, max, step, etc.

I could have sworn they had UI XY positions for in the editor but maybe that was a different system

[DennisSmolek]

I could have sworn they had UI XY positions for in the editor but maybe that was a different system

Found it. It's in the spec and any node can have it:

• xpos (float attribute): X-position of the upper-left corner of the node when drawn in a UI.

• ypos (float attribute): Y-position of the upper-left corner of the node when drawn in a UI.

• width (float attribute): the relative width of the node when drawn in a UI; default is 1.0.

• height (float attribute): the relative height of the node when drawn in a UI; default is 1.0.

• uicolor (color3 attribute): the display-referred color of the node as drawn in the UI, normalized to 0.0-1.0 range;