title: Media2Face: Co-speech Facial Animation Generation With Multi-Modality
    Guidance
summary: The synthesis of 3D facial animations from speech has garnered considerable
  attention. Due to the scarcity of high-quality 4D facial data and
  well-annotated abundant multi-modality labels, previous methods often suffer
  from limited realism and a lack of lexible conditioning. We address this
  challenge through a trilogy. We first introduce Generalized Neural Parametric
  Facial Asset (GNPFA), an efficient variational auto-encoder mapping facial
  geometry and images to a highly generalized expression latent space, decoupling
  expressions and identities. Then, we utilize GNPFA to extract high-quality
  expressions and accurate head poses from a large array of videos. This presents
  the M2F-D dataset, a large, diverse, and scan-level co-speech 3D facial
  animation dataset with well-annotated emotional and style labels. Finally, we
  propose Media2Face, a diffusion model in GNPFA latent space for co-speech
  facial animation generation, accepting rich multi-modality guidances from
  audio, text, and image. Extensive experiments demonstrate that our model not
  only achieves high fidelity in facial animation synthesis but also broadens the
  scope of expressiveness and style adaptability in 3D facial animation.
paper_authors._id: 65b886b2b456c2a99724648f
paper_authors.name: Qingcheng Zhao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-30T09:54:15.303Z
paper_authors._id: 64076bbc2e309e654523f0a0
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1678207867710-noauth.jpeg
paper_authors.fullname: Qingcheng Zhao
paper_authors.type: user
paper_authors.user: Clarive
paper_authors._id: 65b886b2b456c2a997246490
paper_authors.name: Pengyu Long
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-31T08:58:57.365Z
paper_authors._id: 65b9e5060659ad04d010b95a
paper_authors.avatarUrl: /avatars/203b14d7e47d468a4a4286fe151c58fe.svg
paper_authors.fullname: Pengyu Long
paper_authors.type: user
paper_authors.user: Lonng1
paper_authors._id: 65b886b2b456c2a997246491
paper_authors.name: Qixuan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-30T09:54:37.955Z
paper_authors._id: 64c0d0b30a4d02f37aa0bf56
paper_authors.avatarUrl: /avatars/35f13dff3a65e16a0c9f3c039f4ce847.svg
paper_authors.fullname: Zhang Qixuan
paper_authors.type: user
paper_authors.user: oakori
paper_authors._id: 65b886b2b456c2a997246492
paper_authors.name: Dafei Qin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-30T12:43:49.582Z
paper_authors._id: 65b8c8b3fc836d9cd5b8a6ec
paper_authors.avatarUrl: /avatars/9c2490aebcca7b16396302f4f96acdfd.svg
paper_authors.fullname: Dafei Qin
paper_authors.type: user
paper_authors.user: dafeiqin
paper_authors._id: 65b886b2b456c2a997246493
paper_authors.name: Han Liang
paper_authors._id: 65b886b2b456c2a997246494
paper_authors.name: Longwen Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-26T07:41:22.134Z
paper_authors._id: 636d12455aaed143cd665607
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1679399015950-636d12455aaed143cd665607.png
paper_authors.fullname: ZLW
paper_authors.type: user
paper_authors.user: ZarkLngeW
paper_authors._id: 65b886b2b456c2a997246495
paper_authors.name: Yingliang Zhang
paper_authors._id: 65b886b2b456c2a997246496
paper_authors.name: Jingyi Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-30T09:55:25.091Z
paper_authors._id: 63ffcec6e7767a89533b8f80
paper_authors.avatarUrl: /avatars/b9e8673943741c2e7b80b15a061bb951.svg
paper_authors.fullname: JINGYI Yu
paper_authors.type: user
paper_authors.user: Daluuu
paper_authors._id: 65b886b2b456c2a997246497
paper_authors.name: Lan Xu
paper_title: Media2Face: Co-speech Facial Animation Generation With Multi-Modality
    Guidance
paper_summary: The synthesis of 3D facial animations from speech has garnered considerable
  attention. Due to the scarcity of high-quality 4D facial data and
  well-annotated abundant multi-modality labels, previous methods often suffer
  from limited realism and a lack of lexible conditioning. We address this
  challenge through a trilogy. We first introduce Generalized Neural Parametric
  Facial Asset (GNPFA), an efficient variational auto-encoder mapping facial
  geometry and images to a highly generalized expression latent space, decoupling
  expressions and identities. Then, we utilize GNPFA to extract high-quality
  expressions and accurate head poses from a large array of videos. This presents
  the M2F-D dataset, a large, diverse, and scan-level co-speech 3D facial
  animation dataset with well-annotated emotional and style labels. Finally, we
  propose Media2Face, a diffusion model in GNPFA latent space for co-speech
  facial animation generation, accepting rich multi-modality guidances from
  audio, text, and image. Extensive experiments demonstrate that our model not
  only achieves high fidelity in facial animation synthesis but also broadens the
  scope of expressiveness and style adaptability in 3D facial animation.
paper_ai_keywords: variational auto-encoder expression latent space diffusion model M2F-D dataset Media2Face co-speech facial animation generation high fidelity expressiveness style adaptability