DiffSplat: Repurposing Image Diffusion Models for Scalable Gaussian
  Splat Generation
Recent advancements in 3D content generation from text or a single image
struggle with limited high-quality 3D datasets and inconsistency from 2D
multi-view generation. We introduce DiffSplat, a novel 3D generative framework
that natively generates 3D Gaussian splats by taming large-scale text-to-image
diffusion models. It differs from previous 3D generative models by effectively
utilizing web-scale 2D priors while maintaining 3D consistency in a unified
model. To bootstrap the training, a lightweight reconstruction model is
proposed to instantly produce multi-view Gaussian splat grids for scalable
dataset curation. In conjunction with the regular diffusion loss on these
grids, a 3D rendering loss is introduced to facilitate 3D coherence across
arbitrary views. The compatibility with image diffusion models enables seamless
adaptions of numerous techniques for image generation to the 3D realm.
Extensive experiments reveal the superiority of DiffSplat in text- and
image-conditioned generation tasks and downstream applications. Thorough
ablation studies validate the efficacy of each critical design choice and
provide insights into the underlying mechanism.
6799aa5a311dbfe3c96724cd
Chenguo Lin
claimed_verified
2025-01-29T17:16:14.090Z
62e18206926f4892a4c782bd
https://cdn-avatars.huggingface.co/v1/production/uploads/62e18206926f4892a4c782bd/m_OAeRb2a7a5PlnK6d4Dg.jpeg
Chenguo Lin
user
chenguolin
6799aa5a311dbfe3c96724ce
Panwang Pan
admin_assigned
2025-01-29T17:26:28.217Z
654866e8cd0a5621395f8287
https://cdn-avatars.huggingface.co/v1/production/uploads/654866e8cd0a5621395f8287/4Bccwd1ehn-Ee4T1rId5S.jpeg
Panwang Pan
user
paulpanwang
6799aa5a311dbfe3c96724cf
Bangbang Yang
claimed_verified
2025-04-22T10:16:33.342Z
63d748ff6f49aa82306b7e48
/avatars/9f0b8b8a09b14d76e52ed1bd312e6b63.svg
BB Yang
user
ybbbbt
6799aa5a311dbfe3c96724d0
Zeming Li
admin_assigned
2025-01-29T17:26:44.370Z
65c04adbc63d6a8d7f217c4d
/avatars/3407dd73d1f5cf41c56cee7542858f93.svg
Zeming Li
user
ZemingLi
6799aa5a311dbfe3c96724d1
Yadong Mu
DiffSplat: Repurposing Image Diffusion Models for Scalable Gaussian
  Splat Generation
Recent advancements in 3D content generation from text or a single image
struggle with limited high-quality 3D datasets and inconsistency from 2D
multi-view generation. We introduce DiffSplat, a novel 3D generative framework
that natively generates 3D Gaussian splats by taming large-scale text-to-image
diffusion models. It differs from previous 3D generative models by effectively
utilizing web-scale 2D priors while maintaining 3D consistency in a unified
model. To bootstrap the training, a lightweight reconstruction model is
proposed to instantly produce multi-view Gaussian splat grids for scalable
dataset curation. In conjunction with the regular diffusion loss on these
grids, a 3D rendering loss is introduced to facilitate 3D coherence across
arbitrary views. The compatibility with image diffusion models enables seamless
adaptions of numerous techniques for image generation to the 3D realm.
Extensive experiments reveal the superiority of DiffSplat in text- and
image-conditioned generation tasks and downstream applications. Thorough
ablation studies validate the efficacy of each critical design choice and
provide insights into the underlying mechanism.
DiffSplat is a 3D generative model that uses text-to-image diffusion models to generate consistent 3D Gaussian splats from text or images, leveraging web-scale 2D priors and introducing a 3D rendering loss for coherence.
DiffSplat 3D Gaussian splats text-to-image diffusion models web-scale 2D priors 3D rendering loss Gaussian splat grids diffusion loss