about smirk_generator

[ak01user]

Hi!I'm very interested in smirk_generator.

it could re-generate the entire face very well.I noticed that the covered part is not completely black. It randomly samples the relevant information of the original image. I want to know what the training process of this model is.

[filby89]

Hey @ak01user thanks for your interest in SMIRK ! You can find more info on exactly how we sample points from the face in our publication: https://arxiv.org/abs/2404.04104 (Section 3.2.1 and A.2.
This process in the code starts here: https://github.com/georgeretsi/smirk/blob/main/src/smirk_trainer.py#L262

[airpdev]

Hi, @filby89 @ak01user
I am very interested in smirk_generator.
The reconstructed image seems like to be blurry.
Is any solution to fix this problem?

[georgeretsi]

Hi there, Thanks for your interest!
Smirk generator is a "by-product" of the proposed methodology and we have not fully explored its capabilities for realistic reconstruction, albeit it is one of our potential future interests. It can be blurry if the encountered face is too "unknown". Training with more faces, or building a personalized generator can drastically reduce this blurriness. Moreover you can use more pixels inside the mask (mask_ratio parameter of the config file) to provide a better sampling of the initial face. Lastly, you can also use an extra GAN loss with a discriminator to provide more accurate reconstruction, if the goal is realism.

[airpdev]

Thanks for your message.
I understand that we need to train the model with more faces.
Btw is there any solution to improve image quality while using the current pre-trained model.
I have changed mask_ratio parameter but the generated image is almost same as before.
Looking forward to hearing from you.
Thanks

[georgeretsi]

Hi there! Sorry for the late reply. If you need to go towards more realistic generation you can always add a discriminator at the output to increase crispness of the images. This will most likely not affect 3D reconstruction at all, but it can give much more realistic generation.