title: Text-to-3D Generation with Bidirectional Diffusion using both 2D and 3D
    priors
summary: Most 3D generation research focuses on up-projecting 2D foundation models
  into the 3D space, either by minimizing 2D Score Distillation Sampling (SDS)
  loss or fine-tuning on multi-view datasets. Without explicit 3D priors, these
  methods often lead to geometric anomalies and multi-view inconsistency.
  Recently, researchers have attempted to improve the genuineness of 3D objects
  by directly training on 3D datasets, albeit at the cost of low-quality texture
  generation due to the limited texture diversity in 3D datasets. To harness the
  advantages of both approaches, we propose Bidirectional Diffusion(BiDiff), a
  unified framework that incorporates both a 3D and a 2D diffusion process, to
  preserve both 3D fidelity and 2D texture richness, respectively. Moreover, as a
  simple combination may yield inconsistent generation results, we further bridge
  them with novel bidirectional guidance. In addition, our method can be used as
  an initialization of optimization-based models to further improve the quality
  of 3D model and efficiency of optimization, reducing the generation process
  from 3.4 hours to 20 minutes. Experimental results have shown that our model
  achieves high-quality, diverse, and scalable 3D generation. Project website:
  https://bidiff.github.io/.
paper_authors._id: 657681d9cbd807a02e03b656
paper_authors.name: Lihe Ding
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-11T08:48:34.674Z
paper_authors._id: 64190a6c0956be7233a5f317
paper_authors.avatarUrl: /avatars/8590515089bc31a168bc2da2c7516dfb.svg
paper_authors.fullname: Lihe Ding
paper_authors.type: user
paper_authors.user: Lihe
paper_authors._id: 657681d9cbd807a02e03b657
paper_authors.name: Shaocong Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-11T08:48:40.926Z
paper_authors._id: 63ae91af2314b93f9e6dde42
paper_authors.avatarUrl: /avatars/792ce138cbee85b8754fdcec7fb1ff52.svg
paper_authors.fullname: Shaocong Dong
paper_authors.type: user
paper_authors.user: dscdyc
paper_authors._id: 657681d9cbd807a02e03b658
paper_authors.name: Zhanpeng Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-11T08:48:53.713Z
paper_authors._id: 6351129285bdb764f6aa4c8a
paper_authors.avatarUrl: /avatars/cfc32d19e038caedd65180eb07f87881.svg
paper_authors.fullname: HuangZhanpeng
paper_authors.type: user
paper_authors.user: RyanHZP
paper_authors._id: 657681d9cbd807a02e03b659
paper_authors.name: Zibin Wang
paper_authors._id: 657681d9cbd807a02e03b65a
paper_authors.name: Yiyuan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-29T08:28:52.326Z
paper_authors._id: 63176933b58b0184630d2c74
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63176933b58b0184630d2c74/53b5EASwW76zeyyqeJA3O.jpeg
paper_authors.fullname: Yiyuan Zhang
paper_authors.type: user
paper_authors.user: Yiyuan
paper_authors._id: 657681d9cbd807a02e03b65b
paper_authors.name: Kaixiong Gong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-11T08:49:18.537Z
paper_authors._id: 642e427f6748dd4f8eeb2f38
paper_authors.avatarUrl: /avatars/07158ff6aa1803c846403594c5d55a34.svg
paper_authors.fullname: Kaixiong Gong
paper_authors.type: user
paper_authors.user: kxgong
paper_authors._id: 657681d9cbd807a02e03b65c
paper_authors.name: Dan Xu
paper_authors._id: 657681d9cbd807a02e03b65d
paper_authors.name: Tianfan Xue
paper_title: Text-to-3D Generation with Bidirectional Diffusion using both 2D and 3D
    priors
paper_summary: Most 3D generation research focuses on up-projecting 2D foundation models
  into the 3D space, either by minimizing 2D Score Distillation Sampling (SDS)
  loss or fine-tuning on multi-view datasets. Without explicit 3D priors, these
  methods often lead to geometric anomalies and multi-view inconsistency.
  Recently, researchers have attempted to improve the genuineness of 3D objects
  by directly training on 3D datasets, albeit at the cost of low-quality texture
  generation due to the limited texture diversity in 3D datasets. To harness the
  advantages of both approaches, we propose Bidirectional Diffusion(BiDiff), a
  unified framework that incorporates both a 3D and a 2D diffusion process, to
  preserve both 3D fidelity and 2D texture richness, respectively. Moreover, as a
  simple combination may yield inconsistent generation results, we further bridge
  them with novel bidirectional guidance. In addition, our method can be used as
  an initialization of optimization-based models to further improve the quality
  of 3D model and efficiency of optimization, reducing the generation process
  from 3.4 hours to 20 minutes. Experimental results have shown that our model
  achieves high-quality, diverse, and scalable 3D generation. Project website:
  https://bidiff.github.io/.
paper_ai_keywords: Score Distillation Sampling (SDS) fine-tuning multi-view datasets 3D priors geometric anomalies multi-view inconsistency 3D datasets Bidirectional Diffusion (BiDiff) diffusion process 3D fidelity 2D texture richness bidirectional guidance optimization-based models