title: Holodeck: Language Guided Generation of 3D Embodied AI Environments
summary: 3D simulated environments play a critical role in Embodied AI, but their
  creation requires expertise and extensive manual effort, restricting their
  diversity and scope. To mitigate this limitation, we present Holodeck, a system
  that generates 3D environments to match a user-supplied prompt fully
  automatedly. Holodeck can generate diverse scenes, e.g., arcades, spas, and
  museums, adjust the designs for styles, and can capture the semantics of
  complex queries such as "apartment for a researcher with a cat" and "office of
  a professor who is a fan of Star Wars". Holodeck leverages a large language
  model (GPT-4) for common sense knowledge about what the scene might look like
  and uses a large collection of 3D assets from Objaverse to populate the scene
  with diverse objects. To address the challenge of positioning objects
  correctly, we prompt GPT-4 to generate spatial relational constraints between
  objects and then optimize the layout to satisfy those constraints. Our
  large-scale human evaluation shows that annotators prefer Holodeck over
  manually designed procedural baselines in residential scenes and that Holodeck
  can produce high-quality outputs for diverse scene types. We also demonstrate
  an exciting application of Holodeck in Embodied AI, training agents to navigate
  in novel scenes like music rooms and daycares without human-constructed data,
  which is a significant step forward in developing general-purpose embodied
  agents.
paper_authors._id: 657bb7555c6f0b1f36f47613
paper_authors.name: Yue Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-27T11:06:07.639Z
paper_authors._id: 62f6c68904e5e02f82b04690
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62f6c68904e5e02f82b04690/kK2-PkeAGzAOLhkfajswf.jpeg
paper_authors.fullname: Yue Yang
paper_authors.type: user
paper_authors.user: yyupenn
paper_authors._id: 657bb7555c6f0b1f36f47614
paper_authors.name: Fan-Yun Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-19T07:40:26.886Z
paper_authors._id: 66287db3fe7ace12c40026b6
paper_authors.avatarUrl: /avatars/ee86cddd6c7f6f456f7643ee0ddb084a.svg
paper_authors.fullname: Fan-Yun Sun
paper_authors.type: user
paper_authors.user: sunfanyun
paper_authors._id: 657bb7555c6f0b1f36f47615
paper_authors.name: Luca Weihs
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T13:09:33.767Z
paper_authors._id: 620acbdb3c0931626a7c9297
paper_authors.avatarUrl: /avatars/f63b1d225ed81e223d3e8876a5c708c4.svg
paper_authors.fullname: Luca Weihs
paper_authors.type: user
paper_authors.user: lucaweihs
paper_authors._id: 657bb7555c6f0b1f36f47616
paper_authors.name: Eli VanderBilt
paper_authors._id: 657bb7555c6f0b1f36f47617
paper_authors.name: Alvaro Herrasti
paper_authors._id: 657bb7555c6f0b1f36f47618
paper_authors.name: Winson Han
paper_authors._id: 657bb7555c6f0b1f36f47619
paper_authors.name: Jiajun Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T13:09:17.412Z
paper_authors._id: 647b7b8345616c3de3627863
paper_authors.avatarUrl: /avatars/8196364bffed71086ba9a0a4a62027d3.svg
paper_authors.fullname: Jiajun Wu
paper_authors.type: user
paper_authors.user: jiajunwu
paper_authors._id: 657bb7555c6f0b1f36f4761a
paper_authors.name: Nick Haber
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-12-18T08:48:19.263Z
paper_authors._id: 657d3b7d69cfe9b4cd0bfa04
paper_authors.avatarUrl: /avatars/a97b0df7187d08ccca8a6a597cba7f67.svg
paper_authors.fullname: Nick Haber
paper_authors.type: user
paper_authors.user: nhaber
paper_authors._id: 657bb7555c6f0b1f36f4761b
paper_authors.name: Ranjay Krishna
paper_authors._id: 657bb7555c6f0b1f36f4761c
paper_authors.name: Lingjie Liu
paper_authors._id: 657bb7555c6f0b1f36f4761d
paper_authors.name: Chris Callison-Burch
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T13:08:47.889Z
paper_authors._id: 6303ce25fc783bfc744216af
paper_authors.avatarUrl: /avatars/09f5e87c1f56a1b7f6ef9c5037682285.svg
paper_authors.fullname: Chris Callison-Burch
paper_authors.type: user
paper_authors.user: CCB
paper_authors._id: 657bb7555c6f0b1f36f4761e
paper_authors.name: Mark Yatskar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T13:08:41.546Z
paper_authors._id: 631f42e1b6628770f6efd87a
paper_authors.avatarUrl: /avatars/0fa93b3513ebca737cce26dfa5611cf1.svg
paper_authors.fullname: Mark Yatskar
paper_authors.type: user
paper_authors.user: myatskar
paper_authors._id: 657bb7555c6f0b1f36f4761f
paper_authors.name: Aniruddha Kembhavi
paper_authors._id: 657bb7555c6f0b1f36f47620
paper_authors.name: Christopher Clark
paper_title: Holodeck: Language Guided Generation of 3D Embodied AI Environments
paper_summary: 3D simulated environments play a critical role in Embodied AI, but their
  creation requires expertise and extensive manual effort, restricting their
  diversity and scope. To mitigate this limitation, we present Holodeck, a system
  that generates 3D environments to match a user-supplied prompt fully
  automatedly. Holodeck can generate diverse scenes, e.g., arcades, spas, and
  museums, adjust the designs for styles, and can capture the semantics of
  complex queries such as "apartment for a researcher with a cat" and "office of
  a professor who is a fan of Star Wars". Holodeck leverages a large language
  model (GPT-4) for common sense knowledge about what the scene might look like
  and uses a large collection of 3D assets from Objaverse to populate the scene
  with diverse objects. To address the challenge of positioning objects
  correctly, we prompt GPT-4 to generate spatial relational constraints between
  objects and then optimize the layout to satisfy those constraints. Our
  large-scale human evaluation shows that annotators prefer Holodeck over
  manually designed procedural baselines in residential scenes and that Holodeck
  can produce high-quality outputs for diverse scene types. We also demonstrate
  an exciting application of Holodeck in Embodied AI, training agents to navigate
  in novel scenes like music rooms and daycares without human-constructed data,
  which is a significant step forward in developing general-purpose embodied
  agents.
paper_ai_keywords: Embodied AI 3D simulated environments Holodeck user-supplied prompt automated generation 3D environments diverse scenes styles semantic queries GPT-4 common sense knowledge Objaverse 3D assets spatial relational constraints layout optimization residential scenes human evaluation procedural baselines high-quality outputs scene types embodied agents navigating novel scenes human-constructed data general-purpose agents