title: HeadSculpt: Crafting 3D Head Avatars with Text
summary: Recently, text-guided 3D generative methods have made remarkable advancements
  in producing high-quality textures and geometry, capitalizing on the
  proliferation of large vision-language and image diffusion models. However,
  existing methods still struggle to create high-fidelity 3D head avatars in two
  aspects: (1) They rely mostly on a pre-trained text-to-image diffusion model
  whilst missing the necessary 3D awareness and head priors. This makes them
  prone to inconsistency and geometric distortions in the generated avatars. (2)
  They fall short in fine-grained editing. This is primarily due to the inherited
  limitations from the pre-trained 2D image diffusion models, which become more
  pronounced when it comes to 3D head avatars. In this work, we address these
  challenges by introducing a versatile coarse-to-fine pipeline dubbed HeadSculpt
  for crafting (i.e., generating and editing) 3D head avatars from textual
  prompts. Specifically, we first equip the diffusion model with 3D awareness by
  leveraging landmark-based control and a learned textual embedding representing
  the back view appearance of heads, enabling 3D-consistent head avatar
  generations. We further propose a novel identity-aware editing score
  distillation strategy to optimize a textured mesh with a high-resolution
  differentiable rendering technique. This enables identity preservation while
  following the editing instruction. We showcase HeadSculpt's superior fidelity
  and editing capabilities through comprehensive experiments and comparisons with
  existing methods.
paper_authors._id: 647eae12a49bffab5d7503c7
paper_authors.name: Xiao Han
paper_authors._id: 647eae12a49bffab5d7503c8
paper_authors.name: Yukang Cao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-21T18:04:39.516Z
paper_authors._id: 63a07c3ab5515dccd40fdb71
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63a07c3ab5515dccd40fdb71/ly3pwhjWVge25LAeVgriV.png
paper_authors.fullname: Yukang Cao
paper_authors.type: user
paper_authors.user: yukangcao
paper_authors._id: 647eae12a49bffab5d7503c9
paper_authors.name: Kai Han
paper_authors._id: 647eae12a49bffab5d7503ca
paper_authors.name: Xiatian Zhu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-07T10:15:46.790Z
paper_authors._id: 647ef9e7aa8c04bbf9362a8a
paper_authors.avatarUrl: /avatars/dc23e15acf7ee7f3b9f1a68c2716a66d.svg
paper_authors.fullname: Xiatian Zhu
paper_authors.type: user
paper_authors.user: Xiatian-Zhu
paper_authors._id: 647eae12a49bffab5d7503cb
paper_authors.name: Jiankang Deng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-07T10:17:24.390Z
paper_authors._id: 62cc7a38376917c0223dd24b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1657566065867-noauth.png
paper_authors.fullname: JiankangDeng
paper_authors.type: user
paper_authors.user: JiankangDeng
paper_authors._id: 647eae12a49bffab5d7503cc
paper_authors.name: Yi-Zhe Song
paper_authors._id: 647eae12a49bffab5d7503cd
paper_authors.name: Tao Xiang
paper_authors._id: 647eae12a49bffab5d7503ce
paper_authors.name: Kwan-Yee K. Wong
paper_title: HeadSculpt: Crafting 3D Head Avatars with Text
paper_summary: Recently, text-guided 3D generative methods have made remarkable advancements
  in producing high-quality textures and geometry, capitalizing on the
  proliferation of large vision-language and image diffusion models. However,
  existing methods still struggle to create high-fidelity 3D head avatars in two
  aspects: (1) They rely mostly on a pre-trained text-to-image diffusion model
  whilst missing the necessary 3D awareness and head priors. This makes them
  prone to inconsistency and geometric distortions in the generated avatars. (2)
  They fall short in fine-grained editing. This is primarily due to the inherited
  limitations from the pre-trained 2D image diffusion models, which become more
  pronounced when it comes to 3D head avatars. In this work, we address these
  challenges by introducing a versatile coarse-to-fine pipeline dubbed HeadSculpt
  for crafting (i.e., generating and editing) 3D head avatars from textual
  prompts. Specifically, we first equip the diffusion model with 3D awareness by
  leveraging landmark-based control and a learned textual embedding representing
  the back view appearance of heads, enabling 3D-consistent head avatar
  generations. We further propose a novel identity-aware editing score
  distillation strategy to optimize a textured mesh with a high-resolution
  differentiable rendering technique. This enables identity preservation while
  following the editing instruction. We showcase HeadSculpt's superior fidelity
  and editing capabilities through comprehensive experiments and comparisons with
  existing methods.
paper_ai_summary: HeadSculpt introduces a coarse-to-fine pipeline for generating and editing 3D head avatars with high fidelity and identity preservation using a diffusion model enhanced with 3D awareness and a novel editing score distillation strategy.
paper_ai_keywords: text-guided 3D generative methods text-to-image diffusion model 3D awareness head priors landmark-based control textual embedding identity-aware editing score distillation differentiable rendering technique textured mesh