Shellbrady/LivePortrait5
0
1<h1 align="center">LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control</h1>2 3<div align='center'>4 <a href='https://github.com/cleardusk' target='_blank'><strong>Jianzhu Guo</strong></a><sup> 1†</sup> 5 <a href='https://github.com/KwaiVGI' target='_blank'><strong>Dingyun Zhang</strong></a><sup> 1,2</sup> 6 <a href='https://github.com/KwaiVGI' target='_blank'><strong>Xiaoqiang Liu</strong></a><sup> 1</sup> 7 <a href='https://github.com/KwaiVGI' target='_blank'><strong>Zhizhou Zhong</strong></a><sup> 1,3</sup> 8 <a href='https://scholar.google.com.hk/citations?user=_8k1ubAAAAAJ' target='_blank'><strong>Yuan Zhang</strong></a><sup> 1</sup> 9</div>10 11<div align='center'>12 <a href='https://scholar.google.com/citations?user=P6MraaYAAAAJ' target='_blank'><strong>Pengfei Wan</strong></a><sup> 1</sup> 13 <a href='https://openreview.net/profile?id=~Di_ZHANG3' target='_blank'><strong>Di Zhang</strong></a><sup> 1</sup> 14</div>15 16<div align='center'>17 <sup>1 </sup>Kuaishou Technology  <sup>2 </sup>University of Science and Technology of China  <sup>3 </sup>Fudan University 18</div>19 20<br>21<div align="center">22 <!-- <a href='LICENSE'><img src='https://img.shields.io/badge/license-MIT-yellow'></a> -->23 <a href='https://liveportrait.github.io'><img src='https://img.shields.io/badge/Project-Homepage-green'></a>24 <a href='https://arxiv.org/pdf/2407.03168'><img src='https://img.shields.io/badge/Paper-arXiv-red'></a>25</div>26<br>27 28<p align="center">29 <img src="./assets/docs/showcase2.gif" alt="showcase">30 <br>31 🔥 For more results, visit our <a href="https://liveportrait.github.io/"><strong>homepage</strong></a> 🔥32</p>33 34 35 36## 🔥 Updates37- **`2024/07/04`**: 🔥 We released the initial version of the inference code and models. Continuous updates, stay tuned!38- **`2024/07/04`**: 😊 We released the [homepage](https://liveportrait.github.io) and technical report on [arXiv](https://arxiv.org/pdf/2407.03168).39 40## Introduction41This repo, named **LivePortrait**, contains the official PyTorch implementation of our paper [LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control](https://arxiv.org/pdf/2407.03168).42We are actively updating and improving this repository. If you find any bugs or have suggestions, welcome to raise issues or submit pull requests (PR) 💖.43 44## 🔥 Getting Started45### 1. Clone the code and prepare the environment46```bash47git clone https://github.com/KwaiVGI/LivePortrait48cd LivePortrait49 50# create env using conda51conda create -n LivePortrait python==3.9.1852conda activate LivePortrait53# install dependencies with pip54pip install -r requirements.txt55```56 57### 2. Download pretrained weights58Download our pretrained LivePortrait weights and face detection models of InsightFace from [Google Drive](https://drive.google.com/drive/folders/1UtKgzKjFAOmZkhNK-OYT0caJ_w2XAnib) or [Baidu Yun](https://pan.baidu.com/s/1MGctWmNla_vZxDbEp2Dtzw?pwd=z5cn). We have packed all weights in one directory 😊. Unzip and place them in `./pretrained_weights` ensuring the directory structure is as follows:59```text60pretrained_weights61├── insightface62│ └── models63│ └── buffalo_l64│ ├── 2d106det.onnx65│ └── det_10g.onnx66└── liveportrait67 ├── base_models68 │ ├── appearance_feature_extractor.pth69 │ ├── motion_extractor.pth70 │ ├── spade_generator.pth71 │ └── warping_module.pth72 ├── landmark.onnx73 └── retargeting_models74 └── stitching_retargeting_module.pth75```76 77### 3. Inference 🚀78 79```bash80python inference.py81```82 83If the script runs successfully, you will get an output mp4 file named `animations/s6--d0_concat.mp4`. This file includes the following results: driving video, input image, and generated result.84 85<p align="center">86 <img src="./assets/docs/inference.gif" alt="image">87</p>88 89Or, you can change the input by specifying the `-s` and `-d` arguments:90 91```bash92python inference.py -s assets/examples/source/s9.jpg -d assets/examples/driving/d0.mp493 94# or disable pasting back95python inference.py -s assets/examples/source/s9.jpg -d assets/examples/driving/d0.mp4 --no_flag_pasteback96 97# more options to see98python inference.py -h99```100 101**More interesting results can be found in our [Homepage](https://liveportrait.github.io)** 😊102 103### 4. Gradio interface104 105We also provide a Gradio interface for a better experience, just run by:106 107```bash108python app.py109```110 111### 5. Inference speed evaluation 🚀🚀🚀112We have also provided a script to evaluate the inference speed of each module:113 114```bash115python speed.py116```117 118Below are the results of inferring one frame on an RTX 4090 GPU using the native PyTorch framework with `torch.compile`:119 120| Model | Parameters(M) | Model Size(MB) | Inference(ms) |121|-----------------------------------|:-------------:|:--------------:|:-------------:|122| Appearance Feature Extractor | 0.84 | 3.3 | 0.82 |123| Motion Extractor | 28.12 | 108 | 0.84 |124| Spade Generator | 55.37 | 212 | 7.59 |125| Warping Module | 45.53 | 174 | 5.21 |126| Stitching and Retargeting Modules| 0.23 | 2.3 | 0.31 |127 128*Note: the listed values of Stitching and Retargeting Modules represent the combined parameter counts and the total sequential inference time of three MLP networks.*129 130 131## Acknowledgements132We would like to thank the contributors of [FOMM](https://github.com/AliaksandrSiarohin/first-order-model), [Open Facevid2vid](https://github.com/zhanglonghao1992/One-Shot_Free-View_Neural_Talking_Head_Synthesis), [SPADE](https://github.com/NVlabs/SPADE), [InsightFace](https://github.com/deepinsight/insightface) repositories, for their open research and contributions.133 134## Citation 💖135If you find LivePortrait useful for your research, welcome to 🌟 this repo and cite our work using the following BibTeX:136```bibtex137@article{guo2024live,138 title = {LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control},139 author = {Jianzhu Guo and Dingyun Zhang and Xiaoqiang Liu and Zhizhou Zhong and Yuan Zhang and Pengfei Wan and Di Zhang},140 year = {2024},141 journal = {arXiv preprint:2407.03168},142}143```144