title: Baking Gaussian Splatting into Diffusion Denoiser for Fast and Scalable
    Single-stage Image-to-3D Generation
summary: Existing feed-forward image-to-3D methods mainly rely on 2D multi-view
  diffusion models that cannot guarantee 3D consistency. These methods easily
  collapse when changing the prompt view direction and mainly handle
  object-centric prompt images. In this paper, we propose a novel single-stage 3D
  diffusion model, DiffusionGS, for object and scene generation from a single
  view. DiffusionGS directly outputs 3D Gaussian point clouds at each timestep to
  enforce view consistency and allow the model to generate robustly given prompt
  views of any directions, beyond object-centric inputs. Plus, to improve the
  capability and generalization ability of DiffusionGS, we scale up 3D training
  data by developing a scene-object mixed training strategy. Experiments show
  that our method enjoys better generation quality (2.20 dB higher in PSNR and
  23.25 lower in FID) and over 5x faster speed (~6s on an A100 GPU) than SOTA
  methods. The user study and text-to-3D applications also reveals the practical
  values of our method. Our Project page at
  https://caiyuanhao1998.github.io/project/DiffusionGS/ shows the video and
  interactive generation results.
paper_authors._id: 6740a8d5bc8c37ec679fd369
paper_authors.name: Yuanhao Cai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T16:28:35.610Z
paper_authors._id: 673969726c12c4b98b6ab29f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/C2elfn7L68jAt4dtHzDAW.png
paper_authors.fullname: Yuanhao Cai
paper_authors.type: user
paper_authors.user: CaiYuanhao
paper_authors._id: 6740a8d5bc8c37ec679fd36a
paper_authors.name: He Zhang
paper_authors._id: 6740a8d5bc8c37ec679fd36b
paper_authors.name: Kai Zhang
paper_authors._id: 6740a8d5bc8c37ec679fd36c
paper_authors.name: Yixun Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T16:29:02.377Z
paper_authors._id: 647d8f65becb41a272907e7a
paper_authors.avatarUrl: /avatars/f891a1986a9361b23b7c83c23031cf26.svg
paper_authors.fullname: yixunliang
paper_authors.type: user
paper_authors.user: lyxun
paper_authors._id: 6740a8d5bc8c37ec679fd36d
paper_authors.name: Mengwei Ren
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T16:29:09.110Z
paper_authors._id: 63b48ed5a50cfcefda9dbe67
paper_authors.avatarUrl: /avatars/98a2f07ea6a7ce3792f250cf9fecf402.svg
paper_authors.fullname: Mengwei Ren
paper_authors.type: user
paper_authors.user: mengweir
paper_authors._id: 6740a8d5bc8c37ec679fd36e
paper_authors.name: Fujun Luan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T16:29:14.570Z
paper_authors._id: 662ae0aa4f711ee4e1e17ffc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/662ae0aa4f711ee4e1e17ffc/nEi6tv8f6DIhTauwHpw_C.jpeg
paper_authors.fullname: Fujun Luan
paper_authors.type: user
paper_authors.user: luanfujun
paper_authors._id: 6740a8d5bc8c37ec679fd36f
paper_authors.name: Qing Liu
paper_authors._id: 6740a8d5bc8c37ec679fd370
paper_authors.name: Soo Ye Kim
paper_authors._id: 6740a8d5bc8c37ec679fd371
paper_authors.name: Jianming Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-11-22T16:29:28.371Z
paper_authors._id: 632d52047bad38a82f75daee
paper_authors.avatarUrl: /avatars/56cc9b5dea95408a83cb80ee8c5565bf.svg
paper_authors.fullname: Jianming Zhang
paper_authors.type: user
paper_authors.user: jimmie33
paper_authors._id: 6740a8d5bc8c37ec679fd372
paper_authors.name: Zhifei Zhang
paper_authors._id: 6740a8d5bc8c37ec679fd373
paper_authors.name: Yuqian Zhou
paper_authors._id: 6740a8d5bc8c37ec679fd374
paper_authors.name: Zhe Lin
paper_authors._id: 6740a8d5bc8c37ec679fd375
paper_authors.name: Alan Yuille
paper_title: Baking Gaussian Splatting into Diffusion Denoiser for Fast and Scalable
    Single-stage Image-to-3D Generation
paper_summary: Existing feed-forward image-to-3D methods mainly rely on 2D multi-view
  diffusion models that cannot guarantee 3D consistency. These methods easily
  collapse when changing the prompt view direction and mainly handle
  object-centric prompt images. In this paper, we propose a novel single-stage 3D
  diffusion model, DiffusionGS, for object and scene generation from a single
  view. DiffusionGS directly outputs 3D Gaussian point clouds at each timestep to
  enforce view consistency and allow the model to generate robustly given prompt
  views of any directions, beyond object-centric inputs. Plus, to improve the
  capability and generalization ability of DiffusionGS, we scale up 3D training
  data by developing a scene-object mixed training strategy. Experiments show
  that our method enjoys better generation quality (2.20 dB higher in PSNR and
  23.25 lower in FID) and over 5x faster speed (~6s on an A100 GPU) than SOTA
  methods. The user study and text-to-3D applications also reveals the practical
  values of our method. Our Project page at
  https://caiyuanhao1998.github.io/project/DiffusionGS/ shows the video and
  interactive generation results.
paper_ai_summary: A novel single-stage 3D diffusion model, DiffusionGS, is proposed for generating 3D point clouds from a single view, offering better view consistency, quality, and speed compared to state-of-the-art methods.
paper_ai_keywords: feed-forward image-to-3D methods 2D multi-view diffusion models 3D diffusion models DiffusionGS 3D Gaussian point clouds view consistency scene-object mixed training strategy PSNR FID text-to-3D applications