title: LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes
summary: With the widespread usage of VR devices and contents, demands for 3D scene
  generation techniques become more popular. Existing 3D scene generation models,
  however, limit the target scene to specific domain, primarily due to their
  training strategies using 3D scan dataset that is far from the real-world. To
  address such limitation, we propose LucidDreamer, a domain-free scene
  generation pipeline by fully leveraging the power of existing large-scale
  diffusion-based generative model. Our LucidDreamer has two alternate steps:
  Dreaming and Alignment. First, to generate multi-view consistent images from
  inputs, we set the point cloud as a geometrical guideline for each image
  generation. Specifically, we project a portion of point cloud to the desired
  view and provide the projection as a guidance for inpainting using the
  generative model. The inpainted images are lifted to 3D space with estimated
  depth maps, composing a new points. Second, to aggregate the new points into
  the 3D scene, we propose an aligning algorithm which harmoniously integrates
  the portions of newly generated 3D scenes. The finally obtained 3D scene serves
  as initial points for optimizing Gaussian splats. LucidDreamer produces
  Gaussian splats that are highly-detailed compared to the previous 3D scene
  generation methods, with no constraint on domain of the target scene.
paper_authors._id: 655eec9332537bcc8d7482d5
paper_authors.name: Jaeyoung Chung
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T10:59:35.357Z
paper_authors._id: 60d3f9b507da9c17c7270921
paper_authors.avatarUrl: /avatars/3bf882523ea6ed538cce9b08983ecb32.svg
paper_authors.fullname: Jaeyoung Chung
paper_authors.type: user
paper_authors.user: JYChung
paper_authors._id: 655eec9332537bcc8d7482d6
paper_authors.name: Suyoung Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-23T15:47:04.720Z
paper_authors._id: 655f3c854f2c90ed741673a2
paper_authors.avatarUrl: /avatars/a519ce8f165318bf1934824c759a7e91.svg
paper_authors.fullname: Suyoung Lee
paper_authors.type: user
paper_authors.user: esw0116
paper_authors._id: 655eec9332537bcc8d7482d7
paper_authors.name: Hyeongjin Nam
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T10:59:48.726Z
paper_authors._id: 64d904ef8711e883ca9f8764
paper_authors.avatarUrl: /avatars/0a9bd57c3431fc353b4b863971410c6f.svg
paper_authors.fullname: Hyeongjin Nam
paper_authors.type: user
paper_authors.user: hygenie1228
paper_authors._id: 655eec9332537bcc8d7482d8
paper_authors.name: Jaerin Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-23T10:59:55.427Z
paper_authors._id: 643bcf4786ab6dbe34ecae59
paper_authors.avatarUrl: /avatars/4fab7726c0eb5f26b2e741d1d6bf2edb.svg
paper_authors.fullname: Jaerin Lee
paper_authors.type: user
paper_authors.user: ironjr
paper_authors._id: 655eec9332537bcc8d7482d9
paper_authors.name: Kyoung Mu Lee
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-11-24T08:45:08.580Z
paper_authors._id: 656056b21392aa3beb5de0bd
paper_authors.avatarUrl: /avatars/07f25b750ef308d65f2e6c82506e7816.svg
paper_authors.fullname: Kyoung Mu  Lee 
paper_authors.type: user
paper_authors.user: kyoungmu
paper_title: LucidDreamer: Domain-free Generation of 3D Gaussian Splatting Scenes
paper_summary: With the widespread usage of VR devices and contents, demands for 3D scene
  generation techniques become more popular. Existing 3D scene generation models,
  however, limit the target scene to specific domain, primarily due to their
  training strategies using 3D scan dataset that is far from the real-world. To
  address such limitation, we propose LucidDreamer, a domain-free scene
  generation pipeline by fully leveraging the power of existing large-scale
  diffusion-based generative model. Our LucidDreamer has two alternate steps:
  Dreaming and Alignment. First, to generate multi-view consistent images from
  inputs, we set the point cloud as a geometrical guideline for each image
  generation. Specifically, we project a portion of point cloud to the desired
  view and provide the projection as a guidance for inpainting using the
  generative model. The inpainted images are lifted to 3D space with estimated
  depth maps, composing a new points. Second, to aggregate the new points into
  the 3D scene, we propose an aligning algorithm which harmoniously integrates
  the portions of newly generated 3D scenes. The finally obtained 3D scene serves
  as initial points for optimizing Gaussian splats. LucidDreamer produces
  Gaussian splats that are highly-detailed compared to the previous 3D scene
  generation methods, with no constraint on domain of the target scene.
paper_ai_keywords: diffusion-based generative model multi-view consistent images point cloud inpainting depth maps aligning algorithm Gaussian splats