HeadSculpt: Crafting 3D Head Avatars with Text
Recently, text-guided 3D generative methods have made remarkable advancements
in producing high-quality textures and geometry, capitalizing on the
proliferation of large vision-language and image diffusion models. However,
existing methods still struggle to create high-fidelity 3D head avatars in two
aspects: (1) They rely mostly on a pre-trained text-to-image diffusion model
whilst missing the necessary 3D awareness and head priors. This makes them
prone to inconsistency and geometric distortions in the generated avatars. (2)
They fall short in fine-grained editing. This is primarily due to the inherited
limitations from the pre-trained 2D image diffusion models, which become more
pronounced when it comes to 3D head avatars. In this work, we address these
challenges by introducing a versatile coarse-to-fine pipeline dubbed HeadSculpt
for crafting (i.e., generating and editing) 3D head avatars from textual
prompts. Specifically, we first equip the diffusion model with 3D awareness by
leveraging landmark-based control and a learned textual embedding representing
the back view appearance of heads, enabling 3D-consistent head avatar
generations. We further propose a novel identity-aware editing score
distillation strategy to optimize a textured mesh with a high-resolution
differentiable rendering technique. This enables identity preservation while
following the editing instruction. We showcase HeadSculpt's superior fidelity
and editing capabilities through comprehensive experiments and comparisons with
existing methods.
647eae12a49bffab5d7503c7
Xiao Han
647eae12a49bffab5d7503c8
Yukang Cao
claimed_verified
2023-08-21T18:04:39.516Z
63a07c3ab5515dccd40fdb71
https://cdn-avatars.huggingface.co/v1/production/uploads/63a07c3ab5515dccd40fdb71/ly3pwhjWVge25LAeVgriV.png
Yukang Cao
user
yukangcao
647eae12a49bffab5d7503c9
Kai Han
647eae12a49bffab5d7503ca
Xiatian Zhu
admin_assigned
2023-06-07T10:15:46.790Z
647ef9e7aa8c04bbf9362a8a
/avatars/dc23e15acf7ee7f3b9f1a68c2716a66d.svg
Xiatian Zhu
user
Xiatian-Zhu
647eae12a49bffab5d7503cb
Jiankang Deng
admin_assigned
2023-06-07T10:17:24.390Z
62cc7a38376917c0223dd24b
https://cdn-avatars.huggingface.co/v1/production/uploads/1657566065867-noauth.png
JiankangDeng
user
JiankangDeng
647eae12a49bffab5d7503cc
Yi-Zhe Song
647eae12a49bffab5d7503cd
Tao Xiang
647eae12a49bffab5d7503ce
Kwan-Yee K. Wong
HeadSculpt: Crafting 3D Head Avatars with Text
Recently, text-guided 3D generative methods have made remarkable advancements
in producing high-quality textures and geometry, capitalizing on the
proliferation of large vision-language and image diffusion models. However,
existing methods still struggle to create high-fidelity 3D head avatars in two
aspects: (1) They rely mostly on a pre-trained text-to-image diffusion model
whilst missing the necessary 3D awareness and head priors. This makes them
prone to inconsistency and geometric distortions in the generated avatars. (2)
They fall short in fine-grained editing. This is primarily due to the inherited
limitations from the pre-trained 2D image diffusion models, which become more
pronounced when it comes to 3D head avatars. In this work, we address these
challenges by introducing a versatile coarse-to-fine pipeline dubbed HeadSculpt
for crafting (i.e., generating and editing) 3D head avatars from textual
prompts. Specifically, we first equip the diffusion model with 3D awareness by
leveraging landmark-based control and a learned textual embedding representing
the back view appearance of heads, enabling 3D-consistent head avatar
generations. We further propose a novel identity-aware editing score
distillation strategy to optimize a textured mesh with a high-resolution
differentiable rendering technique. This enables identity preservation while
following the editing instruction. We showcase HeadSculpt's superior fidelity
and editing capabilities through comprehensive experiments and comparisons with
existing methods.
HeadSculpt introduces a coarse-to-fine pipeline for generating and editing 3D head avatars with high fidelity and identity preservation using a diffusion model enhanced with 3D awareness and a novel editing score distillation strategy.
text-guided 3D generative methods text-to-image diffusion model 3D awareness head priors landmark-based control textual embedding identity-aware editing score distillation differentiable rendering technique textured mesh