title: Sherpa3D: Boosting High-Fidelity Text-to-3D Generation via Coarse 3D
    Prior
summary: Recently, 3D content creation from text prompts has demonstrated remarkable
  progress by utilizing 2D and 3D diffusion models. While 3D diffusion models
  ensure great multi-view consistency, their ability to generate high-quality and
  diverse 3D assets is hindered by the limited 3D data. In contrast, 2D diffusion
  models find a distillation approach that achieves excellent generalization and
  rich details without any 3D data. However, 2D lifting methods suffer from
  inherent view-agnostic ambiguity thereby leading to serious multi-face Janus
  issues, where text prompts fail to provide sufficient guidance to learn
  coherent 3D results. Instead of retraining a costly viewpoint-aware model, we
  study how to fully exploit easily accessible coarse 3D knowledge to enhance the
  prompts and guide 2D lifting optimization for refinement. In this paper, we
  propose Sherpa3D, a new text-to-3D framework that achieves high-fidelity,
  generalizability, and geometric consistency simultaneously. Specifically, we
  design a pair of guiding strategies derived from the coarse 3D prior generated
  by the 3D diffusion model: a structural guidance for geometric fidelity and a
  semantic guidance for 3D coherence. Employing the two types of guidance, the 2D
  diffusion model enriches the 3D content with diversified and high-quality
  results. Extensive experiments show the superiority of our Sherpa3D over the
  state-of-the-art text-to-3D methods in terms of quality and 3D consistency.
paper_authors._id: 6577f42b1e4ac8627f96fe50
paper_authors.name: Fangfu Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:14:52.412Z
paper_authors._id: 6505a02f9310ce8c400edc63
paper_authors.avatarUrl: /avatars/bbf781594fc8c812316711aa8e2797aa.svg
paper_authors.fullname: Fangfu Liu
paper_authors.type: user
paper_authors.user: Liuff23
paper_authors._id: 6577f42b1e4ac8627f96fe51
paper_authors.name: Diankun Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:15:01.479Z
paper_authors._id: 6577fba2eb02736add6377f5
paper_authors.avatarUrl: /avatars/3e486dd36021feaa4a0259cd89c1eee9.svg
paper_authors.fullname: Wu
paper_authors.type: user
paper_authors.user: Diankun
paper_authors._id: 6577f42b1e4ac8627f96fe52
paper_authors.name: Yi Wei
paper_authors._id: 6577f42b1e4ac8627f96fe53
paper_authors.name: Yongming Rao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-12T08:15:19.905Z
paper_authors._id: 63e4865354f51ea342d45d78
paper_authors.avatarUrl: /avatars/2e7eccc878751331ca8b282f53e38899.svg
paper_authors.fullname: Yongming Rao
paper_authors.type: user
paper_authors.user: raoyongming
paper_authors._id: 6577f42b1e4ac8627f96fe54
paper_authors.name: Yueqi Duan
paper_title: Sherpa3D: Boosting High-Fidelity Text-to-3D Generation via Coarse 3D
    Prior
paper_summary: Recently, 3D content creation from text prompts has demonstrated remarkable
  progress by utilizing 2D and 3D diffusion models. While 3D diffusion models
  ensure great multi-view consistency, their ability to generate high-quality and
  diverse 3D assets is hindered by the limited 3D data. In contrast, 2D diffusion
  models find a distillation approach that achieves excellent generalization and
  rich details without any 3D data. However, 2D lifting methods suffer from
  inherent view-agnostic ambiguity thereby leading to serious multi-face Janus
  issues, where text prompts fail to provide sufficient guidance to learn
  coherent 3D results. Instead of retraining a costly viewpoint-aware model, we
  study how to fully exploit easily accessible coarse 3D knowledge to enhance the
  prompts and guide 2D lifting optimization for refinement. In this paper, we
  propose Sherpa3D, a new text-to-3D framework that achieves high-fidelity,
  generalizability, and geometric consistency simultaneously. Specifically, we
  design a pair of guiding strategies derived from the coarse 3D prior generated
  by the 3D diffusion model: a structural guidance for geometric fidelity and a
  semantic guidance for 3D coherence. Employing the two types of guidance, the 2D
  diffusion model enriches the 3D content with diversified and high-quality
  results. Extensive experiments show the superiority of our Sherpa3D over the
  state-of-the-art text-to-3D methods in terms of quality and 3D consistency.
paper_ai_summary: Sherpa3D leverages 2D diffusion models and coarse 3D priors to generate high-fidelity, geometrically consistent, and diverse 3D content from text prompts.
paper_ai_keywords: 3D diffusion models 2D diffusion models multi-view consistency view-agnostic ambiguity multi-face Janus issues coarse 3D knowledge text-to-3D framework structural guidance semantic guidance geometric fidelity 3D coherence