Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

vae tiling improvements: encoding support and adaptative overlap #484

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

stduhpf
Copy link
Contributor

@stduhpf stduhpf commented Nov 26, 2024

  • Added a boolean argument sd_tiling() function, specifying if the output tile is scaled up (for decoding) or if the input is (encoding).
  • Added a mechanism to automatically fix the tile overlaps in cases of mismatch between image size and tile size.

With these changes, vae encoding can now be tiled, and a small (unreported?) issue with the upscaler (or tae) over-brightening some parts of the image is now fixed. This could also be a step toward tiled diffusion support.

Fixes #588 and #564

@stduhpf stduhpf changed the title Implement vae tiling for encoding vae tiling improvements: encoding support and adaptative overlap Nov 27, 2024
@Green-Sky
Copy link
Contributor

  • Added a mechanism to automatically fix the tile overlaps in cases of mismatch between image size and tile size.

With these changes, vae encoding can now be tiled, and a small (unreported?) issue with the upscaler over-brightening some parts of the image is now fixed. This could also be a step toward tiled diffusion support.

Nice! It would also be interesting to see larger tile sizes. Depending on VAE, larger tiles would still fit into the memory budget and would speed things up.

@masamaru-san
Copy link

I am not familiar with AI programming, and this is a machine translation, so please ignore me if I say something strange.

The last_x used in the loop at line 568 (or line 629 in this PR) in ggml_extend.hpp is only initialized outside the loop.
Locally, it seems that this PR with initializing last_x = false; before the for (int x = 0; ~ loop would improve the generation results.

However, the brightness issue (issue #588) seems to remain.

@stduhpf
Copy link
Contributor Author

stduhpf commented Feb 13, 2025

The last_x used in the loop at line 568 (or line 629 in this PR) in ggml_extend.hpp is only initialized outside the loop.
Locally, it seems that this PR with initializing last_x = false; before the for (int x = 0; ~ loop would improve the generation results.

I'm sorry, I don't quite understand what you mean. last_x is already set to false when the loop starts.

However, the brightness issue (issue #588) seems to remain.

You mean this PR doesn't fix it for you? If so this is strange, I can't reproduce the issue with this PR...

master PR
tae_tiling_576 tae_tiling_576

As you can see the image made with the master branch is much brighter than the one with this PR.
(ignore the slight variation of the composition, it's just a little unrelated bug of the vulkan backend that happens at non standard resolutions, that's fixed by updating GGML)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Over-exposed parts of generated images at some resolutions with VAE (or TAE) tiling, and upscaler
3 participants