title: Learning Disentangled Avatars with Hybrid 3D Representations
summary: Tremendous efforts have been made to learn animatable and photorealistic
  human avatars. Towards this end, both explicit and implicit 3D representations
  are heavily studied for a holistic modeling and capture of the whole human
  (e.g., body, clothing, face and hair), but neither representation is an optimal
  choice in terms of representation efficacy since different parts of the human
  avatar have different modeling desiderata. For example, meshes are generally
  not suitable for modeling clothing and hair. Motivated by this, we present
  Disentangled Avatars~(DELTA), which models humans with hybrid explicit-implicit
  3D representations. DELTA takes a monocular RGB video as input, and produces a
  human avatar with separate body and clothing/hair layers. Specifically, we
  demonstrate two important applications for DELTA. For the first one, we
  consider the disentanglement of the human body and clothing and in the second,
  we disentangle the face and hair. To do so, DELTA represents the body or face
  with an explicit mesh-based parametric 3D model and the clothing or hair with
  an implicit neural radiance field. To make this possible, we design an
  end-to-end differentiable renderer that integrates meshes into volumetric
  rendering, enabling DELTA to learn directly from monocular videos without any
  3D supervision. Finally, we show that how these two applications can be easily
  combined to model full-body avatars, such that the hair, face, body and
  clothing can be fully disentangled yet jointly rendered. Such a disentanglement
  enables hair and clothing transfer to arbitrary body shapes. We empirically
  validate the effectiveness of DELTA's disentanglement by demonstrating its
  promising performance on disentangled reconstruction, virtual clothing try-on
  and hairstyle transfer. To facilitate future research, we also release an
  open-sourced pipeline for the study of hybrid human avatar modeling.
paper_authors._id: 6501391635ec971762ad7d76
paper_authors.name: Yao Feng
paper_authors._id: 6501391635ec971762ad7d77
paper_authors.name: Weiyang Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-14T08:23:03.768Z
paper_authors._id: 648905d1a15c43c791d4381f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648905d1a15c43c791d4381f/GpqGBzsLiMHX0gWZEz3qn.jpeg
paper_authors.fullname: Weiyang Liu
paper_authors.type: user
paper_authors.user: wy1iu
paper_authors._id: 6501391635ec971762ad7d78
paper_authors.name: Timo Bolkart
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-14T08:23:11.707Z
paper_authors._id: 648b73eb1c3e38f18eb39988
paper_authors.avatarUrl: /avatars/ac7b858f504648c1f5d1c3e3034c7996.svg
paper_authors.fullname: Timo Bolkart
paper_authors.type: user
paper_authors.user: tbolkart
paper_authors._id: 6501391635ec971762ad7d79
paper_authors.name: Jinlong Yang
paper_authors._id: 6501391635ec971762ad7d7a
paper_authors.name: Marc Pollefeys
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-25T09:42:59.411Z
paper_authors._id: 67b5fa179782a5e2fd2cb26a
paper_authors.avatarUrl: /avatars/62c38f29ec641e001eeddf840bea21a0.svg
paper_authors.fullname: Marc Pollefeys
paper_authors.type: user
paper_authors.user: mapo1
paper_authors._id: 6501391635ec971762ad7d7b
paper_authors.name: Michael J. Black
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-14T08:24:01.868Z
paper_authors._id: 649fbababa7a05e3626546c8
paper_authors.avatarUrl: /avatars/a9aa4b4914649527094fc1b12779e014.svg
paper_authors.fullname: Michael J. Black
paper_authors.type: user
paper_authors.user: MichaelJBlack
paper_title: Learning Disentangled Avatars with Hybrid 3D Representations
paper_summary: Tremendous efforts have been made to learn animatable and photorealistic
  human avatars. Towards this end, both explicit and implicit 3D representations
  are heavily studied for a holistic modeling and capture of the whole human
  (e.g., body, clothing, face and hair), but neither representation is an optimal
  choice in terms of representation efficacy since different parts of the human
  avatar have different modeling desiderata. For example, meshes are generally
  not suitable for modeling clothing and hair. Motivated by this, we present
  Disentangled Avatars~(DELTA), which models humans with hybrid explicit-implicit
  3D representations. DELTA takes a monocular RGB video as input, and produces a
  human avatar with separate body and clothing/hair layers. Specifically, we
  demonstrate two important applications for DELTA. For the first one, we
  consider the disentanglement of the human body and clothing and in the second,
  we disentangle the face and hair. To do so, DELTA represents the body or face
  with an explicit mesh-based parametric 3D model and the clothing or hair with
  an implicit neural radiance field. To make this possible, we design an
  end-to-end differentiable renderer that integrates meshes into volumetric
  rendering, enabling DELTA to learn directly from monocular videos without any
  3D supervision. Finally, we show that how these two applications can be easily
  combined to model full-body avatars, such that the hair, face, body and
  clothing can be fully disentangled yet jointly rendered. Such a disentanglement
  enables hair and clothing transfer to arbitrary body shapes. We empirically
  validate the effectiveness of DELTA's disentanglement by demonstrating its
  promising performance on disentangled reconstruction, virtual clothing try-on
  and hairstyle transfer. To facilitate future research, we also release an
  open-sourced pipeline for the study of hybrid human avatar modeling.
paper_ai_keywords: animatable and photorealistic human avatars explicit 3D representations implicit 3D representations disentangled avatars DELTA mon RGB video hybrid explicit-implicit 3D representations body and clothing/hair layers neural radiance field end-to-end differentiable renderer volumetric rendering monocular videos disentangled reconstruction virtual clothing try-on hairstyle transfer