title: AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animation
summary: In this study, we propose AniPortrait, a novel framework for generating
  high-quality animation driven by audio and a reference portrait image. Our
  methodology is divided into two stages. Initially, we extract 3D intermediate
  representations from audio and project them into a sequence of 2D facial
  landmarks. Subsequently, we employ a robust diffusion model, coupled with a
  motion module, to convert the landmark sequence into photorealistic and
  temporally consistent portrait animation. Experimental results demonstrate the
  superiority of AniPortrait in terms of facial naturalness, pose diversity, and
  visual quality, thereby offering an enhanced perceptual experience. Moreover,
  our methodology exhibits considerable potential in terms of flexibility and
  controllability, which can be effectively applied in areas such as facial
  motion editing or face reenactment. We release code and model weights at
  https://github.com/scutzzj/AniPortrait
paper_authors._id: 6603ebf72a78438d282ebaca
paper_authors.name: Huawei Wei
paper_authors._id: 6603ebf72a78438d282ebacb
paper_authors.name: Zejun Yang
paper_authors._id: 6603ebf72a78438d282ebacc
paper_authors.name: Zhisheng Wang
paper_title: AniPortrait: Audio-Driven Synthesis of Photorealistic Portrait Animation
paper_summary: In this study, we propose AniPortrait, a novel framework for generating
  high-quality animation driven by audio and a reference portrait image. Our
  methodology is divided into two stages. Initially, we extract 3D intermediate
  representations from audio and project them into a sequence of 2D facial
  landmarks. Subsequently, we employ a robust diffusion model, coupled with a
  motion module, to convert the landmark sequence into photorealistic and
  temporally consistent portrait animation. Experimental results demonstrate the
  superiority of AniPortrait in terms of facial naturalness, pose diversity, and
  visual quality, thereby offering an enhanced perceptual experience. Moreover,
  our methodology exhibits considerable potential in terms of flexibility and
  controllability, which can be effectively applied in areas such as facial
  motion editing or face reenactment. We release code and model weights at
  https://github.com/scutzzj/AniPortrait
paper_ai_keywords: AniPortrait 3D intermediate representations 2D facial landmarks diffusion model motion module photorealistic temporally consistent facial naturalness pose diversity visual quality perceptual experience flexibility controllability facial motion editing face reenactment