WebGLRenderer: Add support for Node Materials

[gkjohnson]

Description

I wanted to open a discussion to talk about the possibility of adding Node Materials to WebGLRenderer - as far as I understand Node Materials are only supported with WebGPURenderer which poses a lot of issues for community adoption, practical testing, and library support:

WebGPURenderer looks like it's coming along great but until it's more fully featured and mature I cannot use it as a platform for any of my professional work. Likewise many tools I've developed and work on are already built on WebGLRenderer and cannot ported to WebGPURenderer. I assume many developers are in the same boat. This means I haven't been able to invest any time in learning and trying the new node material system despite being very interested in the benefits after years of trying to solve shader and material problems.

Adding support to WebGLRenderer would help afford the well-experienced three.js community the ability to test the system in large scale, complex projects and give practical feedback sooner rather than trying it to WebGPURenderer. This will also help ease the transition projects and developers that currently need to use WebGLRenderer. Currently all material development currently being done for WebGL will be rendered useless in WebGPU which this can help avoid.

And from the library perspective - I need to be able to develop something that will work in both WebGLRenderer and WebGPURenderer. Asking developers to maintain two versions of their materials or libraries isn't great and I think will otherwise just make this renderer transition significantly more difficult than it needs to be.

I appreciate all the work done on the new material systems - it looks like a large improvement over what we have now so it would be great to make this transition as smooth as possible.

cc @sunag @RenaudRohlinger

Solution

Add support for NodeMaterials to WebGLRenderer.

Alternatives

Community feedback is significantly delayed, developers and libraries are fragmented.

Additional context

No response

[DennisSmolek]

Correct me if I'm wrong, but doesn't switching to the WebGPURenderer (which supports nodes) work on WebGL only instances as the fallback?

Or are you wanting a way to mix the existing GLSL code with the outputs of the node system?

Are you just wanting the output of the visual side of things or are you wanting things like the compute and functional nodes as well?

[gkjohnson]

I understand that WebGPURenderer has a webgl fallback but this is about using Node Materials with WebGLRenderer. In this case I don't care about the underlying browser graphics API being used - I care about the compatibility with the WebGLRenderer class that many projects are heavily invested in already and cannot or are not ready to switch for a variety of reasons.

Or are you wanting a way to mix the existing GLSL code with the outputs of the node system?

I just want to use node materials as they are with WebGLRenderer and leverage all the benefits that the node system already promises and provide feedback. I'm not proposing new features otherwise.

[sunag]

Thanks for the analysis @gkjohnson, it's a very interesting subject. In these last months where WebGPURenderer is gaining more users, it became clear that we need a migration guide to TSL, not just the a specification. I believe that just as it was done in previous years to design the new Nodes System and TSL, now it will be to smooth the transition.

I fear that it will not be possible to migrate all the TSL features, except for the basic functions already known in the old node system, but I believe that this is not enough. We had to discontinue the new node system for WebGLRenderer because it was falling far behind in features compared to WebGPURenderer (#28167), the renderer ended up being re-architected not only because of WebGPU but also to align with the new node system, and bring new features like backdrop for example, things that would not be possible using GLSL with simple functions in WebGLRenderer without complex work involved, only in this specific case WebGPURenderer does not render the opaque materials twice when using transmission materials like WebGLRenderer, instead it copies the g-buffer to a texture, for this it is also necessary that the color transform is done in post-production internally, this backdrop effect can be accessed on any material using viewportTexture(). The event system like '.updateBefore()' that allows nodes to create pre-pass, at the time of rendering the object, which can also be shared with other related ones, in WebGPURenderer the light and shadow system can be extended using nodes, and this allows users create new shadows and shadow filters as well as the light, light model and the light system without needing to modify the core for this. Including other things like compute()... surely this is not all just what I remembered now.

Alternatively, I would like to know if there is anything else we can improve like the TSL transpiler, and a guide for this migration.

[Mugen87]

I'd like to understand in more detail why migrating to the new renderer is any issue. To me, the real migration task is to move from GSLS to TSL. We want to support this process with more documentation and guides and also technically with a TSL transpiler. But WebGPURenderer itself is not that different compared to WebGLRenderer, at least API-wise. Most public methods are identical, it is required in some cases to switch to an async/await pattern though. As long as you didn't use an extensively forked version of the renderer, exchanging the renderers should not require a lot of migration effort.

Generally speaking, TSL and WebGPURenderer are superior compared to WebGLRenderer with its hardwired materials and lights. As outlined by @sunag supporting node materials in WebGLRenderer is essentially developing a new WebGLRenderer. I've said it earlier but the project should not make the mistake to invest too many resources in maintaining things which are in fact legacy. From a strategic point of view, it's better to focus on the future by adding missing features to the new renderer, improving performance and fixing bugs. I understand when companies (or individuals) have built solutions around WebGLRenderer but it should be clear that a migration to TSL and WebGPURenderer is inevitable. I also understand there are some reservations about the maturity level of WebGPU but it's important to highlight that a WebGPU usage isn't mandatory when using the new renderer. WebGL 2 can always be forced with a renderer parameter.

TBH, I have expected this discussion sooner or later since we always had it when moving to new approaches. E.g. Geometry to BufferGeometry, WebGL 1 to WebGL 2 or legacy to physically correct lighting. There were always developers who wanted to retain the old structures for different reasons but the project consistently deprecated (and eventually removed) legacy code paths. Which was in retrospective the right decision. It should be clear that at some point in the future WebGLRenderer will be deprecated as well. To me, the renderer is now in some sort of pre-stage. We still provide bug fixes and smaller additions, but implement no big features or larger refactorings anymore. And I think that's the right direction.

[gkjohnson]

Thanks Sunag - I understand the new nodes system includes a lot of "magic" that can't necessarily be easily back ported to WebGLRenderer without significant work and I think that's okay. A more standard subset of features akin to what would have been supported previously or with a more traditional nodes system would be sufficient imo and serve as an on-ramp for users - then more complex nodes and TSL systems can be "better with WebGPURenderer".

I'm mostly looking for a way to make sure (some very complex) materials and shaders being written right now can be used later down the line or a porting process can be done progressively rather than all at once, possibly fragmenting the community in the process (libraries will have to decide to support their huge userbase of WebGLRenderer users, abandon them and only support WebGPURenderer, or support two completely different codebases).

---

Mugen -

E.g. Geometry to BufferGeometry, WebGL 1 to WebGL 2 or legacy to physically correct lighting.

I'm having a hard time seeing how these changes can be compared to upgrading to WebGPURenderer at all. WebGL1 > 2 was transparent for almost everyone since it was enabled by default and physically correct lighting was a flag and multiplying light intensities by some constants. Deprecating Geometry is maybe the only significant one but even when I started using the project almost 10 years ago, using "Geometry" already did not feel recommended. There have been a lot of deprecations and whatnot over the lifespan of three.js and for what it's worth none of them have really been concerning. I think it needs to be recognized how much more significant this change is.

Most public methods are identical, it is required in some cases to switch to an async/await pattern though

This alone can mean significant architectural changes for user projects, guarding against race conditions, etc.

I also understand there are some reservations about the maturity level of WebGPU but it's important to highlight that a WebGPU usage isn't mandatory

To be clear I wasn't expressing concern about the underlying webgpu browser API - I'm expressing concern about the three.js WebGPURenderer implementation. WebGLRenderer has over 10 years of bug fixes, API refinement, memory leak fixes, etc. Of course a lot's been learned in that time and is applied to WebGPURenderer but I've written and used enough code to know that a complete rewrite of anything needs some time to be proven before it should be relied on for any large scale projects.

Dependencies on ecosystem libraries is also something that needs to be considered. Tens of thousands of projects using the "postprocessing" package every week, for example, and it will not work with WebGPURenderer which means any project that depends on it will not be able to upgrade. I know this project has post processing effects, as well, but historically the three.js screen effects have not been comparable in terms of performance or quality - which is fine. It should be expected that dedicated package is higher quality.

That project has been in the process of upgrading its architecture for almost 4 years. How long until WebGPURenderer will be supported? Again no knock on the project; it's great work and we all know this stuff takes time especially when it's free contributions. But the reality is that many people rely on projects like this and it's going to impact the sense of "readiness" and ability to migrate to a newer three.js renderer.

I in-part created this issue because another package, three-geospatial, is creating some amazing, complex effects and because of the above limitations the codebase is stuck in WebGLRenderer and I foresee the maintainer not having the time, availability, or interest in porting it forward. Any project depending on this one now will not be able to migrate forward now, either. This may be a small aside but the notation of TSL also makes writing the scale of shaders such as the ones in that project feel completely unapproachable, which is one of the areas I'm hoping more feedback can be surfaced.

Three.js should be promoting building modern, future-proof shaders which should ultimately help accelerate us to a WebGPURenderer-only world rather than continuing to have to build using the old system simply because dependencies and renderer maturity are preventing it. I feel like the inability to write any cross-renderer shaders is already having a larger impact than people seem to be aware of. I'm curious to hear the thoughts on the above issues.

To be clear I'm very excited for WebGPURenderer and the future of node materials. To be honest I've been excited for node materials for the last several years and truly I'd be hoping that I'd have been able to take more practical advantage of them by now 😭

Also happy new years everyone! 🎊

[vanruesc]

That project has been in the process of upgrading its architecture for almost 4 years.

Has it really been that long already 💀

How long until WebGPURenderer will be supported?

I'm excited for node materials but haven't used them for the same reasons you've described. If WebGPURenderer supports shader-based materials alongside node materials, it should be a drop-in replacement for postprocessing.

The part that would need to be changed in postprocessing is the manipulation of three's built-in materials to render additional scene data during the main geometry pass via MRT. From my point of view, node materials just provide a better way of manipulating those built-in materials and while I've been curious about that aspect in the past (#16440 (comment), #16440 (comment)) I've not had the chance yet to actually try it out.

WebGLRenderer's materials can only be tweaked via onBeforeCompile string manipulation which is fragile. A while ago, I figured out that those manipulations can largely be avoided for the MRT use-case if three would define explicit and predictable shader outputs (#27808). And I think node materials already provide something similar. However, WebGLRenderer is still widely used, so I'll try to get back to finishing that PR because it would make the switch to node materials easier later on.