title: RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth
    Diffusion
summary: We introduce RealmDreamer, a technique for generation of general
  forward-facing 3D scenes from text descriptions. Our technique optimizes a 3D
  Gaussian Splatting representation to match complex text prompts. We initialize
  these splats by utilizing the state-of-the-art text-to-image generators,
  lifting their samples into 3D, and computing the occlusion volume. We then
  optimize this representation across multiple views as a 3D inpainting task with
  image-conditional diffusion models. To learn correct geometric structure, we
  incorporate a depth diffusion model by conditioning on the samples from the
  inpainting model, giving rich geometric structure. Finally, we finetune the
  model using sharpened samples from image generators. Notably, our technique
  does not require video or multi-view data and can synthesize a variety of
  high-quality 3D scenes in different styles, consisting of multiple objects. Its
  generality additionally allows 3D synthesis from a single image.
paper_authors._id: 66173de961c1ebcda969238b
paper_authors.name: Jaidev Shriram
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T13:15:53.741Z
paper_authors._id: 6109b9266bb9e045866e3329
paper_authors.avatarUrl: /avatars/caade782fd0ecc5a0f3b0bbbc7d16ccb.svg
paper_authors.fullname: Jaidev Shriram
paper_authors.type: user
paper_authors.user: JaidevShriram
paper_authors._id: 66173de961c1ebcda969238c
paper_authors.name: Alex Trevithick
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T13:15:59.848Z
paper_authors._id: 656a76839dcedd16d5d4ed49
paper_authors.avatarUrl: /avatars/81e4a17f9af8c990dc13bf1a7bb4efbd.svg
paper_authors.fullname: Alex Trevithick
paper_authors.type: user
paper_authors.user: atrevithick
paper_authors._id: 66173de961c1ebcda969238d
paper_authors.name: Lingjie Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-04-18T13:16:14.360Z
paper_authors._id: 65ef493cd08d92badad38e54
paper_authors.avatarUrl: /avatars/734fde502646760eab2c82ea87bfdff0.svg
paper_authors.fullname: liulingie
paper_authors.type: user
paper_authors.user: liulingjie
paper_authors._id: 66173de961c1ebcda969238e
paper_authors.name: Ravi Ramamoorthi
paper_title: RealmDreamer: Text-Driven 3D Scene Generation with Inpainting and Depth
    Diffusion
paper_summary: We introduce RealmDreamer, a technique for generation of general
  forward-facing 3D scenes from text descriptions. Our technique optimizes a 3D
  Gaussian Splatting representation to match complex text prompts. We initialize
  these splats by utilizing the state-of-the-art text-to-image generators,
  lifting their samples into 3D, and computing the occlusion volume. We then
  optimize this representation across multiple views as a 3D inpainting task with
  image-conditional diffusion models. To learn correct geometric structure, we
  incorporate a depth diffusion model by conditioning on the samples from the
  inpainting model, giving rich geometric structure. Finally, we finetune the
  model using sharpened samples from image generators. Notably, our technique
  does not require video or multi-view data and can synthesize a variety of
  high-quality 3D scenes in different styles, consisting of multiple objects. Its
  generality additionally allows 3D synthesis from a single image.
paper_ai_summary: RealmDreamer generates 3D scenes from text descriptions using 3D Gaussian Splatting, image-conditional diffusion models, and depth diffusion models, achieving high-quality and diverse scene synthesis without requiring video or multi-view data.
paper_ai_keywords: 3D Gaussian Splatting text-to-image generators image-conditional diffusion models depth diffusion models 3D inpainting task sharpened samples