title: DiffSplat: Repurposing Image Diffusion Models for Scalable Gaussian
    Splat Generation
summary: Recent advancements in 3D content generation from text or a single image
  struggle with limited high-quality 3D datasets and inconsistency from 2D
  multi-view generation. We introduce DiffSplat, a novel 3D generative framework
  that natively generates 3D Gaussian splats by taming large-scale text-to-image
  diffusion models. It differs from previous 3D generative models by effectively
  utilizing web-scale 2D priors while maintaining 3D consistency in a unified
  model. To bootstrap the training, a lightweight reconstruction model is
  proposed to instantly produce multi-view Gaussian splat grids for scalable
  dataset curation. In conjunction with the regular diffusion loss on these
  grids, a 3D rendering loss is introduced to facilitate 3D coherence across
  arbitrary views. The compatibility with image diffusion models enables seamless
  adaptions of numerous techniques for image generation to the 3D realm.
  Extensive experiments reveal the superiority of DiffSplat in text- and
  image-conditioned generation tasks and downstream applications. Thorough
  ablation studies validate the efficacy of each critical design choice and
  provide insights into the underlying mechanism.
paper_authors._id: 6799aa5a311dbfe3c96724cd
paper_authors.name: Chenguo Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-29T17:16:14.090Z
paper_authors._id: 62e18206926f4892a4c782bd
paper_authors.avatarUrl: /avatars/0f89091a5eb72165d2e860d15b339539.svg
paper_authors.fullname: Chenguo Lin
paper_authors.type: user
paper_authors.user: chenguolin
paper_authors._id: 6799aa5a311dbfe3c96724ce
paper_authors.name: Panwang Pan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-29T17:26:28.217Z
paper_authors._id: 654866e8cd0a5621395f8287
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/654866e8cd0a5621395f8287/4Bccwd1ehn-Ee4T1rId5S.jpeg
paper_authors.fullname: Panwang Pan
paper_authors.type: user
paper_authors.user: paulpanwang
paper_authors._id: 6799aa5a311dbfe3c96724cf
paper_authors.name: Bangbang Yang
paper_authors._id: 6799aa5a311dbfe3c96724d0
paper_authors.name: Zeming Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-29T17:26:44.370Z
paper_authors._id: 65c04adbc63d6a8d7f217c4d
paper_authors.avatarUrl: /avatars/3407dd73d1f5cf41c56cee7542858f93.svg
paper_authors.fullname: Zeming Li
paper_authors.type: user
paper_authors.user: ZemingLi
paper_authors._id: 6799aa5a311dbfe3c96724d1
paper_authors.name: Yadong Mu
paper_title: DiffSplat: Repurposing Image Diffusion Models for Scalable Gaussian
    Splat Generation
paper_summary: Recent advancements in 3D content generation from text or a single image
  struggle with limited high-quality 3D datasets and inconsistency from 2D
  multi-view generation. We introduce DiffSplat, a novel 3D generative framework
  that natively generates 3D Gaussian splats by taming large-scale text-to-image
  diffusion models. It differs from previous 3D generative models by effectively
  utilizing web-scale 2D priors while maintaining 3D consistency in a unified
  model. To bootstrap the training, a lightweight reconstruction model is
  proposed to instantly produce multi-view Gaussian splat grids for scalable
  dataset curation. In conjunction with the regular diffusion loss on these
  grids, a 3D rendering loss is introduced to facilitate 3D coherence across
  arbitrary views. The compatibility with image diffusion models enables seamless
  adaptions of numerous techniques for image generation to the 3D realm.
  Extensive experiments reveal the superiority of DiffSplat in text- and
  image-conditioned generation tasks and downstream applications. Thorough
  ablation studies validate the efficacy of each critical design choice and
  provide insights into the underlying mechanism.
paper_ai_keywords: Diffusion models 3D Gaussian splats web-scale 2D priors gaussian splat grids diffusion loss 3D rendering loss 3D coherence image diffusion models text- and image-conditioned generation tasks