UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animation
<|reference_start|>UniAnimate: Taming Unified Video Diffusion Models for Consistent Human Image Animation: Recent diffusion-based human image animation techniques have demonstrated impressive success in synthesizing videos that faithfully follow a given reference identity and a sequence of desired movement poses. Despite this, there are still two limitations: i) an extra reference model is required to align the identity image with the main video branch, which significantly increases the optimization burden and model parameters; ii) the generated video is usually short in time (e.g., 24 frames), hampering practical applications. To address these shortcomings, we present a UniAnimate framework to enable efficient and long-term human video generation. First, to reduce the optimization difficulty and ensure temporal coherence, we map the reference image along with the posture guidance and noise video into a common feature space by incorporating a unified video diffusion model. Second, we propose a unified noise input that supports random noised input as well as first frame conditioned input, which enhances the ability to generate long-term video. Finally, to further efficiently handle long sequences, we explore an alternative temporal modeling architecture based on state space model to replace the original computation-consuming temporal Transformer. Extensive experimental results indicate that UniAnimate achieves superior synthesis results over existing state-of-the-art counterparts in both quantitative and qualitative evaluations. Notably, UniAnimate can even generate highly consistent one-minute videos by iteratively employing the first frame conditioning strategy. Code and models will be publicly available. Project page: https://unianimate.github.io/.<|reference_end|>
@article{wang2024unianimate:,
    title={UniAnimate: Taming Unified Video Diffusion Models for Consistent Human
  Image Animation},
    author={Xiang Wang, Shiwei Zhang, Changxin Gao, Jiayu Wang, Xiaoqiang Zhou,
  Yingya Zhang, Luxin Yan, Nong Sang},
    journal={arXiv preprint arXiv:2406.01188},
    year={2024},
    archivePrefix={arXiv},
    eprint={2406.01188},
    primaryClass={cs.CV}
}