title: GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphs
summary: As pretrained text-to-image diffusion models become increasingly powerful,
  recent efforts have been made to distill knowledge from these text-to-image
  pretrained models for optimizing a text-guided 3D model. Most of the existing
  methods generate a holistic 3D model from a plain text input. This can be
  problematic when the text describes a complex scene with multiple objects,
  because the vectorized text embeddings are inherently unable to capture a
  complex description with multiple entities and relationships. Holistic 3D
  modeling of the entire scene further prevents accurate grounding of text
  entities and concepts. To address this limitation, we propose GraphDreamer, a
  novel framework to generate compositional 3D scenes from scene graphs, where
  objects are represented as nodes and their interactions as edges. By exploiting
  node and edge information in scene graphs, our method makes better use of the
  pretrained text-to-image diffusion model and is able to fully disentangle
  different objects without image-level supervision. To facilitate modeling of
  object-wise relationships, we use signed distance fields as representation and
  impose a constraint to avoid inter-penetration of objects. To avoid manual
  scene graph creation, we design a text prompt for ChatGPT to generate scene
  graphs based on text inputs. We conduct both qualitative and quantitative
  experiments to validate the effectiveness of GraphDreamer in generating
  high-fidelity compositional 3D scenes with disentangled object entities.
paper_authors._id: 656d30f1f7be0986b4b5a351
paper_authors.name: Gege Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-04T09:46:08.169Z
paper_authors._id: 6476473442e5f5297459e8f1
paper_authors.avatarUrl: /avatars/3fe8549dfd05a167d5b7b0287d5a7d46.svg
paper_authors.fullname: Gege Gao
paper_authors.type: user
paper_authors.user: GGGHSL
paper_authors._id: 656d30f1f7be0986b4b5a352
paper_authors.name: Weiyang Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-04T09:39:15.183Z
paper_authors._id: 648905d1a15c43c791d4381f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/648905d1a15c43c791d4381f/GpqGBzsLiMHX0gWZEz3qn.jpeg
paper_authors.fullname: Weiyang Liu
paper_authors.type: user
paper_authors.user: wy1iu
paper_authors._id: 656d30f1f7be0986b4b5a353
paper_authors.name: Anpei Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-04T09:39:30.232Z
paper_authors._id: 63ea4b4be7f148f46015099f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63ea4b4be7f148f46015099f/tTuZYvtmM2IIGBH8BpKA1.jpeg
paper_authors.fullname: ANPEI CHEN
paper_authors.type: user
paper_authors.user: apchen
paper_authors._id: 656d30f1f7be0986b4b5a354
paper_authors.name: Andreas Geiger
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-04T09:39:37.165Z
paper_authors._id: 620cae049086f3c07f01e3d5
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1644998139538-noauth.jpeg
paper_authors.fullname: Andreas Geiger
paper_authors.type: user
paper_authors.user: andreas-geiger
paper_authors._id: 656d30f1f7be0986b4b5a355
paper_authors.name: Bernhard Schölkopf
paper_title: GraphDreamer: Compositional 3D Scene Synthesis from Scene Graphs
paper_summary: As pretrained text-to-image diffusion models become increasingly powerful,
  recent efforts have been made to distill knowledge from these text-to-image
  pretrained models for optimizing a text-guided 3D model. Most of the existing
  methods generate a holistic 3D model from a plain text input. This can be
  problematic when the text describes a complex scene with multiple objects,
  because the vectorized text embeddings are inherently unable to capture a
  complex description with multiple entities and relationships. Holistic 3D
  modeling of the entire scene further prevents accurate grounding of text
  entities and concepts. To address this limitation, we propose GraphDreamer, a
  novel framework to generate compositional 3D scenes from scene graphs, where
  objects are represented as nodes and their interactions as edges. By exploiting
  node and edge information in scene graphs, our method makes better use of the
  pretrained text-to-image diffusion model and is able to fully disentangle
  different objects without image-level supervision. To facilitate modeling of
  object-wise relationships, we use signed distance fields as representation and
  impose a constraint to avoid inter-penetration of objects. To avoid manual
  scene graph creation, we design a text prompt for ChatGPT to generate scene
  graphs based on text inputs. We conduct both qualitative and quantitative
  experiments to validate the effectiveness of GraphDreamer in generating
  high-fidelity compositional 3D scenes with disentangled object entities.
paper_ai_summary: GraphDreamer generates complex 3D scenes from text inputs by using scene graphs to disentangle object entities and relationships, leveraging pretrained text-to-image diffusion models.
paper_ai_keywords: pretrained text-to-image diffusion models scene graphs nodes edges text-to-image diffusion model signed distance fields inter-penetration ChatGPT