title: GenCA: A Text-conditioned Generative Model for Realistic and Drivable
    Codec Avatars
summary: Photo-realistic and controllable 3D avatars are crucial for various
  applications such as virtual and mixed reality (VR/MR), telepresence, gaming,
  and film production. Traditional methods for avatar creation often involve
  time-consuming scanning and reconstruction processes for each avatar, which
  limits their scalability. Furthermore, these methods do not offer the
  flexibility to sample new identities or modify existing ones. On the other
  hand, by learning a strong prior from data, generative models provide a
  promising alternative to traditional reconstruction methods, easing the time
  constraints for both data capture and processing. Additionally, generative
  methods enable downstream applications beyond reconstruction, such as editing
  and stylization. Nonetheless, the research on generative 3D avatars is still in
  its infancy, and therefore current methods still have limitations such as
  creating static avatars, lacking photo-realism, having incomplete facial
  details, or having limited drivability. To address this, we propose a
  text-conditioned generative model that can generate photo-realistic facial
  avatars of diverse identities, with more complete details like hair, eyes and
  mouth interior, and which can be driven through a powerful non-parametric
  latent expression space. Specifically, we integrate the generative and editing
  capabilities of latent diffusion models with a strong prior model for avatar
  expression driving.
    Our model can generate and control high-fidelity avatars, even those
  out-of-distribution. We also highlight its potential for downstream
  applications, including avatar editing and single-shot avatar reconstruction.
paper_authors._id: 66ce87b28ec9fda2cf61078d
paper_authors.name: Keqiang Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-28T08:16:15.959Z
paper_authors._id: 645ca9f3a19f3e64bbed7534
paper_authors.avatarUrl: /avatars/37fde5fd978bc03a3a02c30102557432.svg
paper_authors.fullname: Keqiang Sun
paper_authors.type: user
paper_authors.user: kqsun
paper_authors._id: 66ce87b28ec9fda2cf61078e
paper_authors.name: Amin Jourabloo
paper_authors._id: 66ce87b28ec9fda2cf61078f
paper_authors.name: Riddhish Bhalodia
paper_authors._id: 66ce87b28ec9fda2cf610790
paper_authors.name: Moustafa Meshry
paper_authors._id: 66ce87b28ec9fda2cf610791
paper_authors.name: Yu Rong
paper_authors._id: 66ce87b28ec9fda2cf610792
paper_authors.name: Zhengyu Yang
paper_authors._id: 66ce87b28ec9fda2cf610793
paper_authors.name: Thu Nguyen-Phuoc
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-28T08:17:42.507Z
paper_authors._id: 6304b663c2f4f2d4929ad200
paper_authors.avatarUrl: /avatars/805d4dab512fc53d8bd6bcb60db3821b.svg
paper_authors.fullname: Thu Nguyen-Phuoc
paper_authors.type: user
paper_authors.user: TNP23
paper_authors._id: 66ce87b28ec9fda2cf610794
paper_authors.name: Christian Haene
paper_authors._id: 66ce87b28ec9fda2cf610795
paper_authors.name: Jiu Xu
paper_authors._id: 66ce87b28ec9fda2cf610796
paper_authors.name: Sam Johnson
paper_authors._id: 66ce87b28ec9fda2cf610797
paper_authors.name: Hongsheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-28T08:18:33.911Z
paper_authors._id: 65c04e9c27a5fdca81abcbd9
paper_authors.avatarUrl: /avatars/12a155683c824fa23da4a9e2bed4f64e.svg
paper_authors.fullname: Hongsheng LI
paper_authors.type: user
paper_authors.user: hsli-cuhk
paper_authors._id: 66ce87b28ec9fda2cf610798
paper_authors.name: Sofien Bouaziz
paper_title: GenCA: A Text-conditioned Generative Model for Realistic and Drivable
    Codec Avatars
paper_summary: Photo-realistic and controllable 3D avatars are crucial for various
  applications such as virtual and mixed reality (VR/MR), telepresence, gaming,
  and film production. Traditional methods for avatar creation often involve
  time-consuming scanning and reconstruction processes for each avatar, which
  limits their scalability. Furthermore, these methods do not offer the
  flexibility to sample new identities or modify existing ones. On the other
  hand, by learning a strong prior from data, generative models provide a
  promising alternative to traditional reconstruction methods, easing the time
  constraints for both data capture and processing. Additionally, generative
  methods enable downstream applications beyond reconstruction, such as editing
  and stylization. Nonetheless, the research on generative 3D avatars is still in
  its infancy, and therefore current methods still have limitations such as
  creating static avatars, lacking photo-realism, having incomplete facial
  details, or having limited drivability. To address this, we propose a
  text-conditioned generative model that can generate photo-realistic facial
  avatars of diverse identities, with more complete details like hair, eyes and
  mouth interior, and which can be driven through a powerful non-parametric
  latent expression space. Specifically, we integrate the generative and editing
  capabilities of latent diffusion models with a strong prior model for avatar
  expression driving.
    Our model can generate and control high-fidelity avatars, even those
  out-of-distribution. We also highlight its potential for downstream
  applications, including avatar editing and single-shot avatar reconstruction.
paper_ai_keywords: generative models latent diffusion models non-parametric latent expression space high-fidelity avatars out-of-distribution avatar editing single-shot avatar reconstruction