title: Aladdin: Zero-Shot Hallucination of Stylized 3D Assets from Abstract
    Scene Descriptions
summary: What constitutes the "vibe" of a particular scene? What should one find in "a
  busy, dirty city street", "an idyllic countryside", or "a crime scene in an
  abandoned living room"? The translation from abstract scene descriptions to
  stylized scene elements cannot be done with any generality by extant systems
  trained on rigid and limited indoor datasets. In this paper, we propose to
  leverage the knowledge captured by foundation models to accomplish this
  translation. We present a system that can serve as a tool to generate stylized
  assets for 3D scenes described by a short phrase, without the need to enumerate
  the objects to be found within the scene or give instructions on their
  appearance. Additionally, it is robust to open-world concepts in a way that
  traditional methods trained on limited data are not, affording more creative
  freedom to the 3D artist. Our system demonstrates this using a foundation model
  "team" composed of a large language model, a vision-language model and several
  image diffusion models, which communicate using an interpretable and
  user-editable intermediate representation, thus allowing for more versatile and
  controllable stylized asset generation for 3D artists. We introduce novel
  metrics for this task, and show through human evaluations that in 91% of the
  cases, our system outputs are judged more faithful to the semantics of the
  input scene description than the baseline, thus highlighting the potential of
  this approach to radically accelerate the 3D content creation process for 3D
  artists.
paper_authors._id: 6487c5eb214e9598c04fc488
paper_authors.name: Ian Huang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-13T13:41:37.927Z
paper_authors._id: 639ce51c66106be14370f510
paper_authors.avatarUrl: /avatars/74f68f5951fff9c02c7607ff197812aa.svg
paper_authors.fullname: Ian Huang
paper_authors.type: user
paper_authors.user: ianhuang
paper_authors._id: 6487c5eb214e9598c04fc489
paper_authors.name: Vrishab Krishna
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-13T13:41:58.965Z
paper_authors._id: 634f8d2407e669188d3ff1e5
paper_authors.avatarUrl: /avatars/1aff23ce965caf7fcfeb6c38b06c5cd7.svg
paper_authors.fullname: Vrishab Krishna
paper_authors.type: user
paper_authors.user: vrishk
paper_authors._id: 6487c5eb214e9598c04fc48a
paper_authors.name: Omoruyi Atekha
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-13T13:42:42.728Z
paper_authors._id: 63c1e851c58fcfeac18b30e2
paper_authors.avatarUrl: /avatars/1080237eef17f0e3476cfd2738602959.svg
paper_authors.fullname: Atekha
paper_authors.type: user
paper_authors.user: Omoruyi
paper_authors._id: 6487c5eb214e9598c04fc48b
paper_authors.name: Leonidas Guibas
paper_title: Aladdin: Zero-Shot Hallucination of Stylized 3D Assets from Abstract
    Scene Descriptions
paper_summary: What constitutes the "vibe" of a particular scene? What should one find in "a
  busy, dirty city street", "an idyllic countryside", or "a crime scene in an
  abandoned living room"? The translation from abstract scene descriptions to
  stylized scene elements cannot be done with any generality by extant systems
  trained on rigid and limited indoor datasets. In this paper, we propose to
  leverage the knowledge captured by foundation models to accomplish this
  translation. We present a system that can serve as a tool to generate stylized
  assets for 3D scenes described by a short phrase, without the need to enumerate
  the objects to be found within the scene or give instructions on their
  appearance. Additionally, it is robust to open-world concepts in a way that
  traditional methods trained on limited data are not, affording more creative
  freedom to the 3D artist. Our system demonstrates this using a foundation model
  "team" composed of a large language model, a vision-language model and several
  image diffusion models, which communicate using an interpretable and
  user-editable intermediate representation, thus allowing for more versatile and
  controllable stylized asset generation for 3D artists. We introduce novel
  metrics for this task, and show through human evaluations that in 91% of the
  cases, our system outputs are judged more faithful to the semantics of the
  input scene description than the baseline, thus highlighting the potential of
  this approach to radically accelerate the 3D content creation process for 3D
  artists.
paper_ai_summary: A system leverages foundation models to generate stylized 3D scene assets from descriptions, demonstrating higher fidelity than baselines through human evaluations.
paper_ai_keywords: foundation models large language model vision-language model image diffusion models intermediate representation stylized asset generation 3D content creation