Media2Face: Co-speech Facial Animation Generation With Multi-Modality
  Guidance
The synthesis of 3D facial animations from speech has garnered considerable
attention. Due to the scarcity of high-quality 4D facial data and
well-annotated abundant multi-modality labels, previous methods often suffer
from limited realism and a lack of lexible conditioning. We address this
challenge through a trilogy. We first introduce Generalized Neural Parametric
Facial Asset (GNPFA), an efficient variational auto-encoder mapping facial
geometry and images to a highly generalized expression latent space, decoupling
expressions and identities. Then, we utilize GNPFA to extract high-quality
expressions and accurate head poses from a large array of videos. This presents
the M2F-D dataset, a large, diverse, and scan-level co-speech 3D facial
animation dataset with well-annotated emotional and style labels. Finally, we
propose Media2Face, a diffusion model in GNPFA latent space for co-speech
facial animation generation, accepting rich multi-modality guidances from
audio, text, and image. Extensive experiments demonstrate that our model not
only achieves high fidelity in facial animation synthesis but also broadens the
scope of expressiveness and style adaptability in 3D facial animation.
65b886b2b456c2a99724648f
Qingcheng Zhao
admin_assigned
2024-01-30T09:54:15.303Z
64076bbc2e309e654523f0a0
https://cdn-avatars.huggingface.co/v1/production/uploads/1678207867710-noauth.jpeg
Qingcheng Zhao
user
Clarive
65b886b2b456c2a997246490
Pengyu Long
claimed_verified
2024-01-31T08:58:57.365Z
65b9e5060659ad04d010b95a
/avatars/203b14d7e47d468a4a4286fe151c58fe.svg
Pengyu Long
user
Lonng1
65b886b2b456c2a997246491
Qixuan Zhang
admin_assigned
2024-01-30T09:54:37.955Z
64c0d0b30a4d02f37aa0bf56
/avatars/35f13dff3a65e16a0c9f3c039f4ce847.svg
Zhang Qixuan
user
oakori
65b886b2b456c2a997246492
Dafei Qin
claimed_verified
2024-01-30T12:43:49.582Z
65b8c8b3fc836d9cd5b8a6ec
/avatars/9c2490aebcca7b16396302f4f96acdfd.svg
Dafei Qin
user
dafeiqin
65b886b2b456c2a997246493
Han Liang
65b886b2b456c2a997246494
Longwen Zhang
claimed_verified
2024-07-26T07:41:22.134Z
636d12455aaed143cd665607
https://cdn-avatars.huggingface.co/v1/production/uploads/1679399015950-636d12455aaed143cd665607.png
ZLW
user
ZarkLngeW
65b886b2b456c2a997246495
Yingliang Zhang
65b886b2b456c2a997246496
Jingyi Yu
admin_assigned
2024-01-30T09:55:25.091Z
63ffcec6e7767a89533b8f80
/avatars/b9e8673943741c2e7b80b15a061bb951.svg
JINGYI Yu
user
Daluuu
65b886b2b456c2a997246497
Lan Xu
Media2Face: Co-speech Facial Animation Generation With Multi-Modality
  Guidance
The synthesis of 3D facial animations from speech has garnered considerable
attention. Due to the scarcity of high-quality 4D facial data and
well-annotated abundant multi-modality labels, previous methods often suffer
from limited realism and a lack of lexible conditioning. We address this
challenge through a trilogy. We first introduce Generalized Neural Parametric
Facial Asset (GNPFA), an efficient variational auto-encoder mapping facial
geometry and images to a highly generalized expression latent space, decoupling
expressions and identities. Then, we utilize GNPFA to extract high-quality
expressions and accurate head poses from a large array of videos. This presents
the M2F-D dataset, a large, diverse, and scan-level co-speech 3D facial
animation dataset with well-annotated emotional and style labels. Finally, we
propose Media2Face, a diffusion model in GNPFA latent space for co-speech
facial animation generation, accepting rich multi-modality guidances from
audio, text, and image. Extensive experiments demonstrate that our model not
only achieves high fidelity in facial animation synthesis but also broadens the
scope of expressiveness and style adaptability in 3D facial animation.
The proposed Generalized Neural Parametric Facial Asset and Media2Face diffusion model generate high-fidelity and expressive 3D facial animations from speech with rich multi-modality guidance.
variational auto-encoder latent space M2F-D dataset diffusion model 3D facial animation co-speech emotional labels style labels Media2Face