title: PlacidDreamer: Advancing Harmony in Text-to-3D Generation
summary: Recently, text-to-3D generation has attracted significant attention,
  resulting in notable performance enhancements. Previous methods utilize
  end-to-end 3D generation models to initialize 3D Gaussians, multi-view
  diffusion models to enforce multi-view consistency, and text-to-image diffusion
  models to refine details with score distillation algorithms. However, these
  methods exhibit two limitations. Firstly, they encounter conflicts in
  generation directions since different models aim to produce diverse 3D assets.
  Secondly, the issue of over-saturation in score distillation has not been
  thoroughly investigated and solved. To address these limitations, we propose
  PlacidDreamer, a text-to-3D framework that harmonizes initialization,
  multi-view generation, and text-conditioned generation with a single multi-view
  diffusion model, while simultaneously employing a novel score distillation
  algorithm to achieve balanced saturation. To unify the generation direction, we
  introduce the Latent-Plane module, a training-friendly plug-in extension that
  enables multi-view diffusion models to provide fast geometry reconstruction for
  initialization and enhanced multi-view images to personalize the text-to-image
  diffusion model. To address the over-saturation problem, we propose to view
  score distillation as a multi-objective optimization problem and introduce the
  Balanced Score Distillation algorithm, which offers a Pareto Optimal solution
  that achieves both rich details and balanced saturation. Extensive experiments
  validate the outstanding capabilities of our PlacidDreamer. The code is
  available at https://github.com/HansenHuang0823/PlacidDreamer.
paper_authors._id: 669db9dea188ffb7e47129a7
paper_authors.name: Shuo Huang
paper_authors._id: 669db9dea188ffb7e47129a8
paper_authors.name: Shikun Sun
paper_authors._id: 669db9dea188ffb7e47129a9
paper_authors.name: Zixuan Wang
paper_authors._id: 669db9dea188ffb7e47129aa
paper_authors.name: Xiaoyu Qin
paper_authors._id: 669db9dea188ffb7e47129ab
paper_authors.name: Yanmin Xiong
paper_authors._id: 669db9dea188ffb7e47129ac
paper_authors.name: Yuan Zhang
paper_authors._id: 669db9dea188ffb7e47129ad
paper_authors.name: Pengfei Wan
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-23T07:48:17.307Z
paper_authors._id: 662f93942510ef5735d7ad00
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/662f93942510ef5735d7ad00/ZIDIPm63sncIHFTT5b0uR.png
paper_authors.fullname: magicwpf
paper_authors.type: user
paper_authors.user: magicwpf
paper_authors._id: 669db9dea188ffb7e47129ae
paper_authors.name: Di Zhang
paper_authors._id: 669db9dea188ffb7e47129af
paper_authors.name: Jia Jia
paper_title: PlacidDreamer: Advancing Harmony in Text-to-3D Generation
paper_summary: Recently, text-to-3D generation has attracted significant attention,
  resulting in notable performance enhancements. Previous methods utilize
  end-to-end 3D generation models to initialize 3D Gaussians, multi-view
  diffusion models to enforce multi-view consistency, and text-to-image diffusion
  models to refine details with score distillation algorithms. However, these
  methods exhibit two limitations. Firstly, they encounter conflicts in
  generation directions since different models aim to produce diverse 3D assets.
  Secondly, the issue of over-saturation in score distillation has not been
  thoroughly investigated and solved. To address these limitations, we propose
  PlacidDreamer, a text-to-3D framework that harmonizes initialization,
  multi-view generation, and text-conditioned generation with a single multi-view
  diffusion model, while simultaneously employing a novel score distillation
  algorithm to achieve balanced saturation. To unify the generation direction, we
  introduce the Latent-Plane module, a training-friendly plug-in extension that
  enables multi-view diffusion models to provide fast geometry reconstruction for
  initialization and enhanced multi-view images to personalize the text-to-image
  diffusion model. To address the over-saturation problem, we propose to view
  score distillation as a multi-objective optimization problem and introduce the
  Balanced Score Distillation algorithm, which offers a Pareto Optimal solution
  that achieves both rich details and balanced saturation. Extensive experiments
  validate the outstanding capabilities of our PlacidDreamer. The code is
  available at https://github.com/HansenHuang0823/PlacidDreamer.
paper_ai_summary: PlacidDreamer is a unified text-to-3D generation framework that uses a single multi-view diffusion model for initialization, multi-view consistency, and text-conditioned generation, while addressing over-saturation with a Balanced Score Distillation algorithm.
paper_ai_keywords: text-to-3D generation end-to-end 3D generation models multi-view diffusion models text-to-image diffusion models score distillation Latent-Plane module geometry reconstruction multi-objective optimization Balanced Score Distillation Pareto Optimal solution