E4S: Fine-Grained Face Swapping via Regional GAN Inversion (Under Review)

Maomao Li, Ge Yuan, Cairong Wang, Zhian Liu, Yong Zhang, Yongwei Nie, Jue Wang, Dong Xu

🦴 Abstract

TL;DR: A Face Swapping and Editing Framework Based on StyleGAN Latent Space

📋 Changelog

• 2023.10.21 Code and paper are released!

🏗️ Todo

• Release the image and video face swapping code
• Delete unuseful code
• Release the face editing code

🚀 Differences with E4S(CVPR2023)

1. Re-coloring modules producing more consistent facial color
2. Inpainting the mismatch face shape
3. Stable and temporal-consistent video face swapping results

🌏 Setup Environment

Our code is mainly based on python3.10+, pytorch2.0+, cuda12+, etc.

conda create -n e4s2023 python=3.10
conda activate e4s2023
pip install -r requirements.txt
export PYTHONPATH=$PWD

🍱 Pretrained Weights

All the weights (including our E4S weights and other third_party weights) can be downloaded from here. Please put all of them into ./pretrained like this:

pretrained
├── codeformer/
├── E4S/
├── face_blender/
├── faceseg/
├── faceVid2Vid/
├── GPEN/
├── inpainting/
├── pixel2style2pixel/
├── pose/
├── SwinIR/
└── zhian/

🍳 WebUI Inference Demo

Run face swapping gradio web-ui demo on your machine locally:

git clone https://github.com/e4s2023/E4S2023.git
python gradio_swap.py

💆‍♀️ Image Face Swapping

💃 Video Face Swapping

We follow STIT and AllInOneDeFliker to make the video face swapping results more stable, which is detailed in our paper. This repo only contains the PTI tuning step of STIT. We found that the PTI tuning step is sufficient to help the StyleGAN to generate stable video frames. This repo has not incorporated the AllInOneDeFliker code yet. You may visit their GitHub page for furthermore post-processing on PTI tuning frames.

Click https://e4s2023.github.io/ to see our video face swapping results.

📎 Citation

@misc{liE4S
    Author = {Maomao Li and Ge Yuan and Cairong Wang and Zhian Liu and Yong Zhang and Yongwei Nie and Jue Wang and Dong Xu},
    Title = {E4S: Fine-Grained Face Swapping via Regional GAN Inversion},
    Year = {2023},
    Eprint = {arXiv:xxxx},
}

💌 Acknowledgements

This repository borrows heavily from E4S(CVPR2023) and STIT. Thanks to the authors for sharing their code and models.

📣 Disclaimer

This is not an official product of Tencent. All the copyrights of the demo images and audio are from community users. Feel free to contact us if you would like remove them.