Baking Gaussian Splatting into Diffusion Denoiser for Fast and Scalable
  Single-stage Image-to-3D Generation
Existing feed-forward image-to-3D methods mainly rely on 2D multi-view
diffusion models that cannot guarantee 3D consistency. These methods easily
collapse when changing the prompt view direction and mainly handle
object-centric prompt images. In this paper, we propose a novel single-stage 3D
diffusion model, DiffusionGS, for object and scene generation from a single
view. DiffusionGS directly outputs 3D Gaussian point clouds at each timestep to
enforce view consistency and allow the model to generate robustly given prompt
views of any directions, beyond object-centric inputs. Plus, to improve the
capability and generalization ability of DiffusionGS, we scale up 3D training
data by developing a scene-object mixed training strategy. Experiments show
that our method enjoys better generation quality (2.20 dB higher in PSNR and
23.25 lower in FID) and over 5x faster speed (~6s on an A100 GPU) than SOTA
methods. The user study and text-to-3D applications also reveals the practical
values of our method. Our Project page at
https://caiyuanhao1998.github.io/project/DiffusionGS/ shows the video and
interactive generation results.
6740a8d5bc8c37ec679fd369
Yuanhao Cai
admin_assigned
2024-11-22T16:28:35.610Z
673969726c12c4b98b6ab29f
https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/C2elfn7L68jAt4dtHzDAW.png
Yuanhao Cai
user
CaiYuanhao
6740a8d5bc8c37ec679fd36a
He Zhang
6740a8d5bc8c37ec679fd36b
Kai Zhang
6740a8d5bc8c37ec679fd36c
Yixun Liang
admin_assigned
2024-11-22T16:29:02.377Z
647d8f65becb41a272907e7a
/avatars/f891a1986a9361b23b7c83c23031cf26.svg
yixunliang
user
lyxun
6740a8d5bc8c37ec679fd36d
Mengwei Ren
admin_assigned
2024-11-22T16:29:09.110Z
63b48ed5a50cfcefda9dbe67
/avatars/98a2f07ea6a7ce3792f250cf9fecf402.svg
Mengwei Ren
user
mengweir
6740a8d5bc8c37ec679fd36e
Fujun Luan
admin_assigned
2024-11-22T16:29:14.570Z
662ae0aa4f711ee4e1e17ffc
https://cdn-avatars.huggingface.co/v1/production/uploads/662ae0aa4f711ee4e1e17ffc/nEi6tv8f6DIhTauwHpw_C.jpeg
Fujun Luan
user
luanfujun
6740a8d5bc8c37ec679fd36f
Qing Liu
6740a8d5bc8c37ec679fd370
Soo Ye Kim
6740a8d5bc8c37ec679fd371
Jianming Zhang
admin_assigned
2024-11-22T16:29:28.371Z
632d52047bad38a82f75daee
/avatars/56cc9b5dea95408a83cb80ee8c5565bf.svg
Jianming Zhang
user
jimmie33
6740a8d5bc8c37ec679fd372
Zhifei Zhang
6740a8d5bc8c37ec679fd373
Yuqian Zhou
6740a8d5bc8c37ec679fd374
Zhe Lin
6740a8d5bc8c37ec679fd375
Alan Yuille
Baking Gaussian Splatting into Diffusion Denoiser for Fast and Scalable
  Single-stage Image-to-3D Generation
Existing feed-forward image-to-3D methods mainly rely on 2D multi-view
diffusion models that cannot guarantee 3D consistency. These methods easily
collapse when changing the prompt view direction and mainly handle
object-centric prompt images. In this paper, we propose a novel single-stage 3D
diffusion model, DiffusionGS, for object and scene generation from a single
view. DiffusionGS directly outputs 3D Gaussian point clouds at each timestep to
enforce view consistency and allow the model to generate robustly given prompt
views of any directions, beyond object-centric inputs. Plus, to improve the
capability and generalization ability of DiffusionGS, we scale up 3D training
data by developing a scene-object mixed training strategy. Experiments show
that our method enjoys better generation quality (2.20 dB higher in PSNR and
23.25 lower in FID) and over 5x faster speed (~6s on an A100 GPU) than SOTA
methods. The user study and text-to-3D applications also reveals the practical
values of our method. Our Project page at
https://caiyuanhao1998.github.io/project/DiffusionGS/ shows the video and
interactive generation results.
A novel single-stage 3D diffusion model, DiffusionGS, is proposed for generating 3D point clouds from a single view, offering better view consistency, quality, and speed compared to state-of-the-art methods.
feed-forward image-to-3D methods 2D multi-view diffusion models 3D diffusion models DiffusionGS 3D Gaussian point clouds view consistency scene-object mixed training strategy PSNR FID text-to-3D applications