PlacidDreamer: Advancing Harmony in Text-to-3D Generation
Recently, text-to-3D generation has attracted significant attention,
resulting in notable performance enhancements. Previous methods utilize
end-to-end 3D generation models to initialize 3D Gaussians, multi-view
diffusion models to enforce multi-view consistency, and text-to-image diffusion
models to refine details with score distillation algorithms. However, these
methods exhibit two limitations. Firstly, they encounter conflicts in
generation directions since different models aim to produce diverse 3D assets.
Secondly, the issue of over-saturation in score distillation has not been
thoroughly investigated and solved. To address these limitations, we propose
PlacidDreamer, a text-to-3D framework that harmonizes initialization,
multi-view generation, and text-conditioned generation with a single multi-view
diffusion model, while simultaneously employing a novel score distillation
algorithm to achieve balanced saturation. To unify the generation direction, we
introduce the Latent-Plane module, a training-friendly plug-in extension that
enables multi-view diffusion models to provide fast geometry reconstruction for
initialization and enhanced multi-view images to personalize the text-to-image
diffusion model. To address the over-saturation problem, we propose to view
score distillation as a multi-objective optimization problem and introduce the
Balanced Score Distillation algorithm, which offers a Pareto Optimal solution
that achieves both rich details and balanced saturation. Extensive experiments
validate the outstanding capabilities of our PlacidDreamer. The code is
available at https://github.com/HansenHuang0823/PlacidDreamer.
669db9dea188ffb7e47129a7
Shuo Huang
669db9dea188ffb7e47129a8
Shikun Sun
669db9dea188ffb7e47129a9
Zixuan Wang
669db9dea188ffb7e47129aa
Xiaoyu Qin
669db9dea188ffb7e47129ab
Yanmin Xiong
669db9dea188ffb7e47129ac
Yuan Zhang
669db9dea188ffb7e47129ad
Pengfei Wan
claimed_verified
2024-07-23T07:48:17.307Z
662f93942510ef5735d7ad00
https://cdn-avatars.huggingface.co/v1/production/uploads/662f93942510ef5735d7ad00/ZIDIPm63sncIHFTT5b0uR.png
magicwpf
user
magicwpf
669db9dea188ffb7e47129ae
Di Zhang
669db9dea188ffb7e47129af
Jia Jia
PlacidDreamer: Advancing Harmony in Text-to-3D Generation
Recently, text-to-3D generation has attracted significant attention,
resulting in notable performance enhancements. Previous methods utilize
end-to-end 3D generation models to initialize 3D Gaussians, multi-view
diffusion models to enforce multi-view consistency, and text-to-image diffusion
models to refine details with score distillation algorithms. However, these
methods exhibit two limitations. Firstly, they encounter conflicts in
generation directions since different models aim to produce diverse 3D assets.
Secondly, the issue of over-saturation in score distillation has not been
thoroughly investigated and solved. To address these limitations, we propose
PlacidDreamer, a text-to-3D framework that harmonizes initialization,
multi-view generation, and text-conditioned generation with a single multi-view
diffusion model, while simultaneously employing a novel score distillation
algorithm to achieve balanced saturation. To unify the generation direction, we
introduce the Latent-Plane module, a training-friendly plug-in extension that
enables multi-view diffusion models to provide fast geometry reconstruction for
initialization and enhanced multi-view images to personalize the text-to-image
diffusion model. To address the over-saturation problem, we propose to view
score distillation as a multi-objective optimization problem and introduce the
Balanced Score Distillation algorithm, which offers a Pareto Optimal solution
that achieves both rich details and balanced saturation. Extensive experiments
validate the outstanding capabilities of our PlacidDreamer. The code is
available at https://github.com/HansenHuang0823/PlacidDreamer.
PlacidDreamer is a unified text-to-3D generation framework that uses a single multi-view diffusion model for initialization, multi-view consistency, and text-conditioned generation, while addressing over-saturation with a Balanced Score Distillation algorithm.
text-to-3D generation end-to-end 3D generation models multi-view diffusion models text-to-image diffusion models score distillation Latent-Plane module geometry reconstruction multi-objective optimization Balanced Score Distillation Pareto Optimal solution