Using a more accurate model trained at a low resolution to feed a control map to ControlNet

[scarbain]

Could we theoretically train a muse model using lucidrains implementation at a low resolution (64x64 for example) to only produce like segmentation maps (or M-LSD lines or even a mix of both) that we would upscale to 512x512 and then feed that to the control image of the ControlNet Segmentation model of SD ?
This way we would get the benefits of better accurate results from Muse trained at a quick low resolution and the diversity of custom models from SD1.5 ?

@lllyasviel what do you think ?

https://github.com/lucidrains/muse-maskgit-pytorch
https://github.com/Sygil-Dev/muse-maskgit-pytorch

[geroldmeisinger]

This sounds like it could be good idea but I have difficulty following your concept. Could you elaborate please!

• What is Muse?
• What would be different from deriving the current segmentation?