LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes
With the widespread usage of VR devices and contents, demands for 3D scene
generation techniques become more popular. Existing 3D scene generation models,
however, limit the target scene to specific domain, primarily due to their
training strategies using 3D scan dataset that is far from the real-world. To
address such limitation, we propose LucidDreamer, a domain-free scene
generation pipeline by fully leveraging the power of existing large-scale
diffusion-based generative model. Our LucidDreamer has two alternate steps:
Dreaming and Alignment. First, to generate multi-view consistent images from
inputs, we set the point cloud as a geometrical guideline for each image
generation. Specifically, we project a portion of point cloud to the desired
view and provide the projection as a guidance for inpainting using the
generative model. The inpainted images are lifted to 3D space with estimated
depth maps, composing a new points. Second, to aggregate the new points into
the 3D scene, we propose an aligning algorithm which harmoniously integrates
the portions of newly generated 3D scenes. The finally obtained 3D scene serves
as initial points for optimizing Gaussian splats. LucidDreamer produces
Gaussian splats that are highly-detailed compared to the previous 3D scene
generation methods, with no constraint on domain of the target scene.
655eec9332537bcc8d7482d5
Jaeyoung Chung
admin_assigned
2023-11-23T10:59:35.357Z
60d3f9b507da9c17c7270921
/avatars/3bf882523ea6ed538cce9b08983ecb32.svg
Jaeyoung Chung
user
JYChung
655eec9332537bcc8d7482d6
Suyoung Lee
claimed_verified
2023-11-23T15:47:04.720Z
655f3c854f2c90ed741673a2
/avatars/a519ce8f165318bf1934824c759a7e91.svg
Suyoung Lee
user
esw0116
655eec9332537bcc8d7482d7
Hyeongjin Nam
admin_assigned
2023-11-23T10:59:48.726Z
64d904ef8711e883ca9f8764
/avatars/0a9bd57c3431fc353b4b863971410c6f.svg
Hyeongjin Nam
user
hygenie1228
655eec9332537bcc8d7482d8
Jaerin Lee
admin_assigned
2023-11-23T10:59:55.427Z
643bcf4786ab6dbe34ecae59
/avatars/4fab7726c0eb5f26b2e741d1d6bf2edb.svg
Jaerin Lee
user
ironjr
655eec9332537bcc8d7482d9
Kyoung Mu Lee
claimed_verified
2023-11-24T08:45:08.580Z
656056b21392aa3beb5de0bd
/avatars/07f25b750ef308d65f2e6c82506e7816.svg
Kyoung Mu  Lee 
user
kyoungmu
LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes
With the widespread usage of VR devices and contents, demands for 3D scene
generation techniques become more popular. Existing 3D scene generation models,
however, limit the target scene to specific domain, primarily due to their
training strategies using 3D scan dataset that is far from the real-world. To
address such limitation, we propose LucidDreamer, a domain-free scene
generation pipeline by fully leveraging the power of existing large-scale
diffusion-based generative model. Our LucidDreamer has two alternate steps:
Dreaming and Alignment. First, to generate multi-view consistent images from
inputs, we set the point cloud as a geometrical guideline for each image
generation. Specifically, we project a portion of point cloud to the desired
view and provide the projection as a guidance for inpainting using the
generative model. The inpainted images are lifted to 3D space with estimated
depth maps, composing a new points. Second, to aggregate the new points into
the 3D scene, we propose an aligning algorithm which harmoniously integrates
the portions of newly generated 3D scenes. The finally obtained 3D scene serves
as initial points for optimizing Gaussian splats. LucidDreamer produces
Gaussian splats that are highly-detailed compared to the previous 3D scene
generation methods, with no constraint on domain of the target scene.
LucidDreamer generates domain-free 3D scenes using diffusion-based generative models and Gaussian splats, producing highly detailed results.
diffusion-based generative model point cloud inpainting depth maps 3D scene generation Gaussian splats