EMO2: End-Effector Guided Audio-Driven Avatar Video Generation
In this paper, we propose a novel audio-driven talking head method capable of
simultaneously generating highly expressive facial expressions and hand
gestures. Unlike existing methods that focus on generating full-body or
half-body poses, we investigate the challenges of co-speech gesture generation
and identify the weak correspondence between audio features and full-body
gestures as a key limitation. To address this, we redefine the task as a
two-stage process. In the first stage, we generate hand poses directly from
audio input, leveraging the strong correlation between audio signals and hand
movements. In the second stage, we employ a diffusion model to synthesize video
frames, incorporating the hand poses generated in the first stage to produce
realistic facial expressions and body movements. Our experimental results
demonstrate that the proposed method outperforms state-of-the-art approaches,
such as CyberHost and Vlogger, in terms of both visual quality and
synchronization accuracy. This work provides a new perspective on audio-driven
gesture generation and a robust framework for creating expressive and natural
talking head animations.
6790856e3b0a6384a4117d0e
Linrui Tian
6790856e3b0a6384a4117d0f
Siqi Hu
admin_assigned
2025-01-22T16:01:58.376Z
66f9af43e45f81e73c0327ea
/avatars/30976181ab540fb42a3377a2734a5793.svg
Siqi Hu
user
Siqi-Hu
6790856e3b0a6384a4117d10
Qi Wang
6790856e3b0a6384a4117d11
Bang Zhang
6790856e3b0a6384a4117d12
Liefeng Bo
admin_assigned
2025-01-22T16:02:08.503Z
63d0cc736b985b0f25d0412c
/avatars/3eb8c79f9a7c4c819038ea7b04e323dd.svg
Bo
user
Liefeng
EMO2: End-Effector Guided Audio-Driven Avatar Video Generation
In this paper, we propose a novel audio-driven talking head method capable of
simultaneously generating highly expressive facial expressions and hand
gestures. Unlike existing methods that focus on generating full-body or
half-body poses, we investigate the challenges of co-speech gesture generation
and identify the weak correspondence between audio features and full-body
gestures as a key limitation. To address this, we redefine the task as a
two-stage process. In the first stage, we generate hand poses directly from
audio input, leveraging the strong correlation between audio signals and hand
movements. In the second stage, we employ a diffusion model to synthesize video
frames, incorporating the hand poses generated in the first stage to produce
realistic facial expressions and body movements. Our experimental results
demonstrate that the proposed method outperforms state-of-the-art approaches,
such as CyberHost and Vlogger, in terms of both visual quality and
synchronization accuracy. This work provides a new perspective on audio-driven
gesture generation and a robust framework for creating expressive and natural
talking head animations.
A novel audio-driven talking head method generates expressive facial expressions and hand gestures using a two-stage process, combining direct audio-to-hand-pose generation and a diffusion model for video synthesis.
audio-driven talking head facial expressions hand gestures co-speech gesture generation diffusion model video frames CyberHost Vlogger synchronization accuracy