Aladdin: Zero-Shot Hallucination of Stylized 3D Assets from Abstract
  Scene Descriptions
What constitutes the "vibe" of a particular scene? What should one find in "a
busy, dirty city street", "an idyllic countryside", or "a crime scene in an
abandoned living room"? The translation from abstract scene descriptions to
stylized scene elements cannot be done with any generality by extant systems
trained on rigid and limited indoor datasets. In this paper, we propose to
leverage the knowledge captured by foundation models to accomplish this
translation. We present a system that can serve as a tool to generate stylized
assets for 3D scenes described by a short phrase, without the need to enumerate
the objects to be found within the scene or give instructions on their
appearance. Additionally, it is robust to open-world concepts in a way that
traditional methods trained on limited data are not, affording more creative
freedom to the 3D artist. Our system demonstrates this using a foundation model
"team" composed of a large language model, a vision-language model and several
image diffusion models, which communicate using an interpretable and
user-editable intermediate representation, thus allowing for more versatile and
controllable stylized asset generation for 3D artists. We introduce novel
metrics for this task, and show through human evaluations that in 91% of the
cases, our system outputs are judged more faithful to the semantics of the
input scene description than the baseline, thus highlighting the potential of
this approach to radically accelerate the 3D content creation process for 3D
artists.
6487c5eb214e9598c04fc488
Ian Huang
admin_assigned
2023-06-13T13:41:37.927Z
639ce51c66106be14370f510
/avatars/74f68f5951fff9c02c7607ff197812aa.svg
Ian Huang
user
ianhuang
6487c5eb214e9598c04fc489
Vrishab Krishna
admin_assigned
2023-06-13T13:41:58.965Z
634f8d2407e669188d3ff1e5
/avatars/1aff23ce965caf7fcfeb6c38b06c5cd7.svg
Vrishab Krishna
user
vrishk
6487c5eb214e9598c04fc48a
Omoruyi Atekha
admin_assigned
2023-06-13T13:42:42.728Z
63c1e851c58fcfeac18b30e2
/avatars/1080237eef17f0e3476cfd2738602959.svg
Atekha
user
Omoruyi
6487c5eb214e9598c04fc48b
Leonidas Guibas
Aladdin: Zero-Shot Hallucination of Stylized 3D Assets from Abstract
  Scene Descriptions
What constitutes the "vibe" of a particular scene? What should one find in "a
busy, dirty city street", "an idyllic countryside", or "a crime scene in an
abandoned living room"? The translation from abstract scene descriptions to
stylized scene elements cannot be done with any generality by extant systems
trained on rigid and limited indoor datasets. In this paper, we propose to
leverage the knowledge captured by foundation models to accomplish this
translation. We present a system that can serve as a tool to generate stylized
assets for 3D scenes described by a short phrase, without the need to enumerate
the objects to be found within the scene or give instructions on their
appearance. Additionally, it is robust to open-world concepts in a way that
traditional methods trained on limited data are not, affording more creative
freedom to the 3D artist. Our system demonstrates this using a foundation model
"team" composed of a large language model, a vision-language model and several
image diffusion models, which communicate using an interpretable and
user-editable intermediate representation, thus allowing for more versatile and
controllable stylized asset generation for 3D artists. We introduce novel
metrics for this task, and show through human evaluations that in 91% of the
cases, our system outputs are judged more faithful to the semantics of the
input scene description than the baseline, thus highlighting the potential of
this approach to radically accelerate the 3D content creation process for 3D
artists.
A system leverages foundation models to generate stylized 3D scene assets from descriptions, demonstrating higher fidelity than baselines through human evaluations.
foundation models large language model vision-language model image diffusion models intermediate representation stylized asset generation 3D content creation