EMO: Emote Portrait Alive - Generating Expressive Portrait Videos with
  Audio2Video Diffusion Model under Weak Conditions
In this work, we tackle the challenge of enhancing the realism and
expressiveness in talking head video generation by focusing on the dynamic and
nuanced relationship between audio cues and facial movements. We identify the
limitations of traditional techniques that often fail to capture the full
spectrum of human expressions and the uniqueness of individual facial styles.
To address these issues, we propose EMO, a novel framework that utilizes a
direct audio-to-video synthesis approach, bypassing the need for intermediate
3D models or facial landmarks. Our method ensures seamless frame transitions
and consistent identity preservation throughout the video, resulting in highly
expressive and lifelike animations. Experimental results demonsrate that EMO is
able to produce not only convincing speaking videos but also singing videos in
various styles, significantly outperforming existing state-of-the-art
methodologies in terms of expressiveness and realism.
65deacc617d90ff441877aaf
Linrui Tian
65deacc617d90ff441877ab0
Qi Wang
claimed_verified
2024-02-28T12:09:33.691Z
65df1f1ee98700500d4c289c
/avatars/be11bf61465df29ac997cc0fedad1cb9.svg
qi wang
user
lucaskingjade
65deacc617d90ff441877ab1
Bang Zhang
65deacc617d90ff441877ab2
Liefeng Bo
admin_assigned
2024-02-28T11:45:36.931Z
63d0cc736b985b0f25d0412c
/avatars/3eb8c79f9a7c4c819038ea7b04e323dd.svg
Bo
user
Liefeng
EMO: Emote Portrait Alive - Generating Expressive Portrait Videos with
  Audio2Video Diffusion Model under Weak Conditions
In this work, we tackle the challenge of enhancing the realism and
expressiveness in talking head video generation by focusing on the dynamic and
nuanced relationship between audio cues and facial movements. We identify the
limitations of traditional techniques that often fail to capture the full
spectrum of human expressions and the uniqueness of individual facial styles.
To address these issues, we propose EMO, a novel framework that utilizes a
direct audio-to-video synthesis approach, bypassing the need for intermediate
3D models or facial landmarks. Our method ensures seamless frame transitions
and consistent identity preservation throughout the video, resulting in highly
expressive and lifelike animations. Experimental results demonsrate that EMO is
able to produce not only convincing speaking videos but also singing videos in
various styles, significantly outperforming existing state-of-the-art
methodologies in terms of expressiveness and realism.
EMO, a direct audio-to-video synthesis framework, enhances realism and expressiveness in talking head video generation by capturing nuanced audio-facial relationships without intermediate models.
audio-to-video synthesis seamless frame transitions consistent identity preservation singing videos