LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control
<|reference_start|>LivePortrait: Efficient Portrait Animation with Stitching and Retargeting Control: Portrait Animation aims to synthesize a lifelike video from a single source image, using it as an appearance reference, with motion (i.e., facial expressions and head pose) derived from a driving video, audio, text, or generation. Instead of following mainstream diffusion-based methods, we explore and extend the potential of the implicit-keypoint-based framework, which effectively balances computational efficiency and controllability. Building upon this, we develop a video-driven portrait animation framework named LivePortrait with a focus on better generalization, controllability, and efficiency for practical usage. To enhance the generation quality and generalization ability, we scale up the training data to about 69 million high-quality frames, adopt a mixed image-video training strategy, upgrade the network architecture, and design better motion transformation and optimization objectives. Additionally, we discover that compact implicit keypoints can effectively represent a kind of blendshapes and meticulously propose a stitching and two retargeting modules, which utilize a small MLP with negligible computational overhead, to enhance the controllability. Experimental results demonstrate the efficacy of our framework even compared to diffusion-based methods. The generation speed remarkably reaches 12.8ms on an RTX 4090 GPU with PyTorch. The inference code and models are available at https://github.com/KwaiVGI/LivePortrait<|reference_end|>
@article{guo2024liveportrait:,
    title={LivePortrait: Efficient Portrait Animation with Stitching and
  Retargeting Control},
    author={Jianzhu Guo, Dingyun Zhang, Xiaoqiang Liu, Zhizhou Zhong, Yuan Zhang,
  Pengfei Wan, Di Zhang},
    journal={arXiv preprint arXiv:2407.03168},
    year={2024},
    archivePrefix={arXiv},
    eprint={2407.03168},
    primaryClass={cs.CV}
}