Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WIP - IBL shadow material blending #15544

Closed

Conversation

MiiBond
Copy link
Contributor

@MiiBond MiiBond commented Sep 9, 2024

I'm working on how to combine the generated shadow image with the actual material properties.
Shadows should only apply to view-independent lighting and so don't work really well for things like specular reflections.
Therefore, I want to apply them 100% to diffuse (indirect) lighting and scale their affect on specular (direct) lighting based on surface roughness. The more rough the surface, the less view-dependent the specular component becomes.

I'm struggling with the best way to do this in Babylon. Currently, I'm trying to use prepass buffers to write out this view-independent/dependent lighting and then I combine them with shadows in a post process. This is simple but has some issues:

  1. I ended up hijacking fragData[0] for the indirect lighting component because, otherwise, this data is wasted and I end up using an additional buffer. Then, anything else (like UI) that was written to the fragData[0] is lost when I apply the shadows.
  2. Some things, like emissive, might be technically view-independent but they shouldn't have shadows applied.

It might be better to apply the shadows in the material shader directly to handle all the material properties as needed. However, this implies rendering all the prepass buffers before the final material render. This means everything will be drawn an extra time. This is how I've implemented this effect before but in, WebGL, I worry about the performance implications of rendering all the geometry another time...

@MiiBond MiiBond marked this pull request as draft September 9, 2024 20:46
@bjsplat
Copy link
Collaborator

bjsplat commented Sep 9, 2024

Please make sure to label your PR with "bug", "new feature" or "breaking change" label(s).
To prevent this PR from going to the changelog marked it with the "skip changelog" label.

@deltakosh
Copy link
Contributor

Thanks a lot @MiiBond ! But before moving on with more changes I would need for the current version to work on WebGPU with tint so I can merge my webgpu shaders and thus upcoming new PR can update glsl and wgsl simultaneously

@bjsplat
Copy link
Collaborator

bjsplat commented Sep 9, 2024

@bjsplat
Copy link
Collaborator

bjsplat commented Sep 9, 2024

@bjsplat
Copy link
Collaborator

bjsplat commented Sep 9, 2024

@Popov72
Copy link
Contributor

Popov72 commented Sep 10, 2024

  1. I ended up hijacking fragData[0] for the indirect lighting component because, otherwise, this data is wasted and I end up using an additional buffer. Then, anything else (like UI) that was written to the fragData[0] is lost when I apply the shadows.

You should still use a specific index and not fragData[0]. This is because we're developing a render graph, which has a GeometryRenderer task that reuses existing prepass rendering shader code, and fragData[0] is not special in this context (it's no longer the final color), but one of the buffers you want to generate.

Another more immediate reason for fragData[0] to remain the final color is if additional post-processing is added, since in this case the effect of the post-processing is applied to this buffer.

It might be better to apply the shadows in the material shader directly to handle all the material properties as needed. However, this implies rendering all the prepass buffers before the final material render. This means everything will be drawn an extra time. This is how I've implemented this effect before but in, WebGL, I worry about the performance implications of rendering all the geometry another time...

You'll be able to do this when the new render graph is officially supported in Babylon.js, but in the meantime I think you should keep the existing code, as an additional geometry pass would be too heavy!

@MiiBond
Copy link
Contributor Author

MiiBond commented Sep 12, 2024

The issue that I'm having is that this isn't really a simple post-process effect. It doesn't really have use for the textureSampler input.
The shadows should composite as:
indirectLighting * shadows + directLighting * shadows_modified_based_on_roughness
It doesn't involve the previously rendered, fully lit material.
Can we enforce that the iblShadowsRenderPipeline is always the first pipeline? Even then, the UI won't be rendered into the prepass buffers and will be lost...
Maybe some new flags on the prepass renderer could control some of these things?

@Popov72
Copy link
Contributor

Popov72 commented Sep 13, 2024

It's a limitation of the prepass renderer, the geometry textures can only be used in a post-process.

You can use the geometry buffer renderer instead. This will render the scene once again, but with a simplified shader (no final color calculation). This corresponds better to what will be available in the frame graph, where a geometry renderer task will render geometry textures, which you can use in shaders as input textures.

In your case, I can't see how this can be done without two passes. You need to generate the geometry textures first, then use them when rendering the meshes. You'll need to write a material plugin to implement the “indirectLighting * shadows + directLighting * shadows_modified_based_on_roughness” calculation part in a PBR material.

This is pretty much what we do to implement RSM GI. We use the geometry buffer renderer to generate the geometry textures we need, then we use a material plugin which can merge the GI contribution when rendering a mesh with a standard or PBR material.

As far as the pre-pass renderer is concerned, we don't want to add any new features, since it won't be used anymore once we have full frame graph support in the engine.

Copy link

This pull request has been marked as stale because it has been inactive for more than 14 days. Please update to "unstale".

@github-actions github-actions bot added the stale label Oct 12, 2024
@Popov72
Copy link
Contributor

Popov72 commented Oct 21, 2024

Should this PR be closed?

@MiiBond
Copy link
Contributor Author

MiiBond commented Oct 21, 2024

This is replaced by #15634

@MiiBond MiiBond closed this Oct 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants