Can Not Find Pretrained CLIP Implementation

[echelon2718]

Hey there! Firstly, thank you for publishing this wonderful work. I am fascinated by your research and I'm trying to understand every component of Scenimefy. However, I could not find where you implement the pretrained CLIP's code for the PatchNCE loss (I read your paper, and you mentioned that you're using a pretrained CLIP for extracting the image features). I would greatly appreciate it if you want to elaborate more about this component based on your paper. Thank you very much!

[Yuxinn-J]

Hi, thanks for saying that I'm glad you liked it. Regarding your question, yes, we use the pretrained CLIP model in the first stage to help preserve content while fine-tuning the StyleGAN to generate pseudo paired data. As for the PatchNCE loss, we actually extract features directly from the GAN generator itself, rather than using CLIP. You can find the relevant code here.

Given the impressive generative capabilities of diffusion models nowadays, an alternative approach could be to use a diffusion model to generate pseudo paired data in the first stage instead of fine-tuning StyleGAN. This might yield better results for supervising GAN training.

[echelon2718]

Thank you very much! I was thinking of the same thing, I am also using Diffusion models to generate the pseudo-paired dataset. certainly, it leverages the quality of generation a lot.