title: Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion
    Dependency
summary: With the introduction of diffusion-based video generation techniques,
  audio-conditioned human video generation has recently achieved significant
  breakthroughs in both the naturalness of motion and the synthesis of portrait
  details. Due to the limited control of audio signals in driving human motion,
  existing methods often add auxiliary spatial signals to stabilize movements,
  which may compromise the naturalness and freedom of motion. In this paper, we
  propose an end-to-end audio-only conditioned video diffusion model named Loopy.
  Specifically, we designed an inter- and intra-clip temporal module and an
  audio-to-latents module, enabling the model to leverage long-term motion
  information from the data to learn natural motion patterns and improving
  audio-portrait movement correlation. This method removes the need for manually
  specified spatial motion templates used in existing methods to constrain motion
  during inference. Extensive experiments show that Loopy outperforms recent
  audio-driven portrait diffusion models, delivering more lifelike and
  high-quality results across various scenarios.
paper_authors._id: 66d9211e9e955e0e5d1d7040
paper_authors.name: Jianwen Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-05T10:16:32.675Z
paper_authors._id: 643410509bd5a84b5dca1345
paper_authors.avatarUrl: /avatars/597823ce61c7b1a5da77e178820824f6.svg
paper_authors.fullname: Jianwen Jiang
paper_authors.type: user
paper_authors.user: JianwenJ
paper_authors._id: 66d9211e9e955e0e5d1d7041
paper_authors.name: Chao Liang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-09T10:58:07.640Z
paper_authors._id: 66d7ca4c0429a62c38c46bf5
paper_authors.avatarUrl: /avatars/45c7859627ac4be70c40f5b1fd02b18a.svg
paper_authors.fullname: liangdebugger
paper_authors.type: user
paper_authors.user: chao0412
paper_authors._id: 66d9211e9e955e0e5d1d7042
paper_authors.name: Jiaqi Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-06T07:06:27.786Z
paper_authors._id: 6527603743db8c626c467726
paper_authors.avatarUrl: /avatars/216df0374b37355d57d2c76e6f08e4d6.svg
paper_authors.fullname: yang
paper_authors.type: user
paper_authors.user: jiaqi78
paper_authors._id: 66d9211e9e955e0e5d1d7043
paper_authors.name: Gaojie Lin
paper_authors._id: 66d9211e9e955e0e5d1d7044
paper_authors.name: Tianyun Zhong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-05T08:50:36.107Z
paper_authors._id: 6393200e7a88f87c10056c43
paper_authors.avatarUrl: /avatars/4ddde5cc4e5780e55d12ce0129234365.svg
paper_authors.fullname: zhongtianyun
paper_authors.type: user
paper_authors.user: tianyumyum
paper_authors._id: 66d9211e9e955e0e5d1d7045
paper_authors.name: Yanbo Zheng
paper_title: Loopy: Taming Audio-Driven Portrait Avatar with Long-Term Motion
    Dependency
paper_summary: With the introduction of diffusion-based video generation techniques,
  audio-conditioned human video generation has recently achieved significant
  breakthroughs in both the naturalness of motion and the synthesis of portrait
  details. Due to the limited control of audio signals in driving human motion,
  existing methods often add auxiliary spatial signals to stabilize movements,
  which may compromise the naturalness and freedom of motion. In this paper, we
  propose an end-to-end audio-only conditioned video diffusion model named Loopy.
  Specifically, we designed an inter- and intra-clip temporal module and an
  audio-to-latents module, enabling the model to leverage long-term motion
  information from the data to learn natural motion patterns and improving
  audio-portrait movement correlation. This method removes the need for manually
  specified spatial motion templates used in existing methods to constrain motion
  during inference. Extensive experiments show that Loopy outperforms recent
  audio-driven portrait diffusion models, delivering more lifelike and
  high-quality results across various scenarios.
paper_ai_summary: An audio-only conditioned video diffusion model, Loopy, improves natural and high-quality human video generation without auxiliary spatial signals.
paper_ai_keywords: diffusion-based video generation audio-conditioned human video generation motion portrait details audio signals inter- and intra-clip temporal module audio-to-latents module long-term motion information natural motion patterns audio-portrait movement correlation video diffusion model spatial motion templates lifelike results