title: L3GO: Language Agents with Chain-of-3D-Thoughts for Generating
    Unconventional Objects
summary: Diffusion-based image generation models such as DALL-E 3 and Stable
  Diffusion-XL demonstrate remarkable capabilities in generating images with
  realistic and unique compositions. Yet, these models are not robust in
  precisely reasoning about physical and spatial configurations of objects,
  especially when instructed with unconventional, thereby out-of-distribution
  descriptions, such as "a chair with five legs". In this paper, we propose a
  language agent with chain-of-3D-thoughts (L3GO), an inference-time approach
  that can reason about part-based 3D mesh generation of unconventional objects
  that current data-driven diffusion models struggle with. More concretely, we
  use large language models as agents to compose a desired object via
  trial-and-error within the 3D simulation environment. To facilitate our
  investigation, we develop a new benchmark, Unconventionally Feasible Objects
  (UFO), as well as SimpleBlenv, a wrapper environment built on top of Blender
  where language agents can build and compose atomic building blocks via API
  calls. Human and automatic GPT-4V evaluations show that our approach surpasses
  the standard GPT-4 and other language agents (e.g., ReAct and Reflexion) for 3D
  mesh generation on ShapeNet. Moreover, when tested on our UFO benchmark, our
  approach outperforms other state-of-the-art text-to-2D image and text-to-3D
  models based on human evaluation.
paper_authors._id: 65cd7b626f58553a7451821f
paper_authors.name: Yutaro Yamada
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-02-15T04:08:48.923Z
paper_authors._id: 63142272e29fb2e86d63db0a
paper_authors.avatarUrl: /avatars/95eb031a7fe5dc9e17566f9d60263a05.svg
paper_authors.fullname: Yutaro
paper_authors.type: user
paper_authors.user: yyamada
paper_authors._id: 65cd7b626f58553a74518220
paper_authors.name: Khyathi Chandu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-15T09:52:34.372Z
paper_authors._id: 62ebe6472f8999993a847bb6
paper_authors.avatarUrl: /avatars/29fb590f8116bdc9c55f7a333e2c4280.svg
paper_authors.fullname: Khyathi Raghavi Chandu
paper_authors.type: user
paper_authors.user: khyathi
paper_authors._id: 65cd7b626f58553a74518221
paper_authors.name: Yuchen Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-15T08:26:20.257Z
paper_authors._id: 607f666a4ad99100d63ce35c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/607f666a4ad99100d63ce35c/QxhxnvfeV6efkxwUFHwjI.png
paper_authors.fullname: Bill Yuchen Lin
paper_authors.type: user
paper_authors.user: yuchenlin
paper_authors._id: 65cd7b626f58553a74518222
paper_authors.name: Jack Hessel
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-15T09:48:44.757Z
paper_authors._id: 625498644f4edf771516b2cb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1664497251391-625498644f4edf771516b2cb.jpeg
paper_authors.fullname: Jack Hessel
paper_authors.type: user
paper_authors.user: jmhessel
paper_authors._id: 65cd7b626f58553a74518223
paper_authors.name: Ilker Yildirim
paper_authors._id: 65cd7b626f58553a74518224
paper_authors.name: Yejin Choi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-15T09:49:03.220Z
paper_authors._id: 64d42729f63b01b7f676b176
paper_authors.avatarUrl: /avatars/52e54bdd6a1fb6c774a40cd70f3d7925.svg
paper_authors.fullname: Yejin Choi
paper_authors.type: user
paper_authors.user: yejinchoinka
paper_title: L3GO: Language Agents with Chain-of-3D-Thoughts for Generating
    Unconventional Objects
paper_summary: Diffusion-based image generation models such as DALL-E 3 and Stable
  Diffusion-XL demonstrate remarkable capabilities in generating images with
  realistic and unique compositions. Yet, these models are not robust in
  precisely reasoning about physical and spatial configurations of objects,
  especially when instructed with unconventional, thereby out-of-distribution
  descriptions, such as "a chair with five legs". In this paper, we propose a
  language agent with chain-of-3D-thoughts (L3GO), an inference-time approach
  that can reason about part-based 3D mesh generation of unconventional objects
  that current data-driven diffusion models struggle with. More concretely, we
  use large language models as agents to compose a desired object via
  trial-and-error within the 3D simulation environment. To facilitate our
  investigation, we develop a new benchmark, Unconventionally Feasible Objects
  (UFO), as well as SimpleBlenv, a wrapper environment built on top of Blender
  where language agents can build and compose atomic building blocks via API
  calls. Human and automatic GPT-4V evaluations show that our approach surpasses
  the standard GPT-4 and other language agents (e.g., ReAct and Reflexion) for 3D
  mesh generation on ShapeNet. Moreover, when tested on our UFO benchmark, our
  approach outperforms other state-of-the-art text-to-2D image and text-to-3D
  models based on human evaluation.
paper_ai_keywords: diffusion-based image generation models DALL-E 3 Stable Diffusion-XL language agent chain-of-3D-thoughts (L3GO) inference-time approach part-based 3D mesh generation out-of-distribution descriptions large language models 3D simulation environment atomic building blocks API calls Unconventionally Feasible Objects (UFO) SimpleBlenv GPT-4V ReAct Reflexion ShapeNet text-to-2D image text-to-3D models