L3GO: Language Agents with Chain-of-3D-Thoughts for Generating
  Unconventional Objects
Diffusion-based image generation models such as DALL-E 3 and Stable
Diffusion-XL demonstrate remarkable capabilities in generating images with
realistic and unique compositions. Yet, these models are not robust in
precisely reasoning about physical and spatial configurations of objects,
especially when instructed with unconventional, thereby out-of-distribution
descriptions, such as "a chair with five legs". In this paper, we propose a
language agent with chain-of-3D-thoughts (L3GO), an inference-time approach
that can reason about part-based 3D mesh generation of unconventional objects
that current data-driven diffusion models struggle with. More concretely, we
use large language models as agents to compose a desired object via
trial-and-error within the 3D simulation environment. To facilitate our
investigation, we develop a new benchmark, Unconventionally Feasible Objects
(UFO), as well as SimpleBlenv, a wrapper environment built on top of Blender
where language agents can build and compose atomic building blocks via API
calls. Human and automatic GPT-4V evaluations show that our approach surpasses
the standard GPT-4 and other language agents (e.g., ReAct and Reflexion) for 3D
mesh generation on ShapeNet. Moreover, when tested on our UFO benchmark, our
approach outperforms other state-of-the-art text-to-2D image and text-to-3D
models based on human evaluation.
65cd7b626f58553a7451821f
Yutaro Yamada
extracted_confirmed
2024-02-15T04:08:48.923Z
63142272e29fb2e86d63db0a
/avatars/5726b5c40cea190a6cd9907fa596b66b.svg
Yutaro
user
yyamada
65cd7b626f58553a74518220
Khyathi Chandu
admin_assigned
2024-02-15T09:52:34.372Z
62ebe6472f8999993a847bb6
/avatars/29fb590f8116bdc9c55f7a333e2c4280.svg
Khyathi Raghavi Chandu
user
khyathi
65cd7b626f58553a74518221
Yuchen Lin
claimed_verified
2024-02-15T08:26:20.257Z
607f666a4ad99100d63ce35c
https://cdn-avatars.huggingface.co/v1/production/uploads/607f666a4ad99100d63ce35c/QxhxnvfeV6efkxwUFHwjI.png
Bill Yuchen Lin
user
yuchenlin
65cd7b626f58553a74518222
Jack Hessel
admin_assigned
2024-02-15T09:48:44.757Z
625498644f4edf771516b2cb
https://cdn-avatars.huggingface.co/v1/production/uploads/1664497251391-625498644f4edf771516b2cb.jpeg
Jack Hessel
user
jmhessel
65cd7b626f58553a74518223
Ilker Yildirim
65cd7b626f58553a74518224
Yejin Choi
admin_assigned
2024-02-15T09:49:03.220Z
64d42729f63b01b7f676b176
/avatars/52e54bdd6a1fb6c774a40cd70f3d7925.svg
Yejin Choi
user
yejinchoinka
L3GO: Language Agents with Chain-of-3D-Thoughts for Generating
  Unconventional Objects
Diffusion-based image generation models such as DALL-E 3 and Stable
Diffusion-XL demonstrate remarkable capabilities in generating images with
realistic and unique compositions. Yet, these models are not robust in
precisely reasoning about physical and spatial configurations of objects,
especially when instructed with unconventional, thereby out-of-distribution
descriptions, such as "a chair with five legs". In this paper, we propose a
language agent with chain-of-3D-thoughts (L3GO), an inference-time approach
that can reason about part-based 3D mesh generation of unconventional objects
that current data-driven diffusion models struggle with. More concretely, we
use large language models as agents to compose a desired object via
trial-and-error within the 3D simulation environment. To facilitate our
investigation, we develop a new benchmark, Unconventionally Feasible Objects
(UFO), as well as SimpleBlenv, a wrapper environment built on top of Blender
where language agents can build and compose atomic building blocks via API
calls. Human and automatic GPT-4V evaluations show that our approach surpasses
the standard GPT-4 and other language agents (e.g., ReAct and Reflexion) for 3D
mesh generation on ShapeNet. Moreover, when tested on our UFO benchmark, our
approach outperforms other state-of-the-art text-to-2D image and text-to-3D
models based on human evaluation.
A language agent with chain-of-3D-thoughts (L3GO) enhances 3D object generation for unconventional descriptions by using large language models to compose objects in a 3D simulation environment, outperforming GPT-4 and other models in the UFO benchmark.
diffusion-based image generation models DALL-E 3 Stable Diffusion-XL chain-of-3D-thoughts (L3GO) part-based 3D mesh generation 3D simulation environment Unconventionally Feasible Objects (UFO) SimpleBlenv GPT-4V ReAct Reflexion ShapeNet text-to-2D image text-to-3D models