title: Divide and Conquer: Language Models can Plan and Self-Correct for
    Compositional Text-to-Image Generation
summary: Despite significant advancements in text-to-image models for generating
  high-quality images, these methods still struggle to ensure the controllability
  of text prompts over images in the context of complex text prompts, especially
  when it comes to retaining object attributes and relationships. In this paper,
  we propose CompAgent, a training-free approach for compositional text-to-image
  generation, with a large language model (LLM) agent as its core. The
  fundamental idea underlying CompAgent is premised on a divide-and-conquer
  methodology. Given a complex text prompt containing multiple concepts including
  objects, attributes, and relationships, the LLM agent initially decomposes it,
  which entails the extraction of individual objects, their associated
  attributes, and the prediction of a coherent scene layout. These individual
  objects can then be independently conquered. Subsequently, the agent performs
  reasoning by analyzing the text, plans and employs the tools to compose these
  isolated objects. The verification and human feedback mechanism is finally
  incorporated into our agent to further correct the potential attribute errors
  and refine the generated images. Guided by the LLM agent, we propose a
  tuning-free multi-concept customization model and a layout-to-image generation
  model as the tools for concept composition, and a local image editing method as
  the tool to interact with the agent for verification. The scene layout controls
  the image generation process among these tools to prevent confusion among
  multiple objects. Extensive experiments demonstrate the superiority of our
  approach for compositional text-to-image generation: CompAgent achieves more
  than 10\% improvement on T2I-CompBench, a comprehensive benchmark for
  open-world compositional T2I generation. The extension to various related tasks
  also illustrates the flexibility of our CompAgent for potential applications.
paper_authors._id: 65b878a914125834a9d7b71a
paper_authors.name: Zhenyu Wang
paper_authors._id: 65b878a914125834a9d7b71b
paper_authors.name: Enze Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-30T10:02:42.836Z
paper_authors._id: 616961bdd3f656f79ad18ec1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/616961bdd3f656f79ad18ec1/NYpxas378lDevRuUDepJJ.jpeg
paper_authors.fullname: xieenze
paper_authors.type: user
paper_authors.user: xieenze
paper_authors._id: 65b878a914125834a9d7b71c
paper_authors.name: Aoxue Li
paper_authors._id: 65b878a914125834a9d7b71d
paper_authors.name: Zhongdao Wang
paper_authors._id: 65b878a914125834a9d7b71e
paper_authors.name: Xihui Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-03T18:29:15.933Z
paper_authors._id: 65d5ec74cd05bc1eaa125040
paper_authors.avatarUrl: /avatars/2de1b1539a86452c2c89570eeb02f5ab.svg
paper_authors.fullname: Xihui Liu
paper_authors.type: user
paper_authors.user: XihuiLiu
paper_authors._id: 65b878a914125834a9d7b71f
paper_authors.name: Zhenguo Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-30T10:03:45.174Z
paper_authors._id: 639944bef7c4648254d124f0
paper_authors.avatarUrl: /avatars/c5cbb87cd51cc1341844f67d42c55151.svg
paper_authors.fullname: lzg
paper_authors.type: user
paper_authors.user: lizhenguo
paper_title: Divide and Conquer: Language Models can Plan and Self-Correct for
    Compositional Text-to-Image Generation
paper_summary: Despite significant advancements in text-to-image models for generating
  high-quality images, these methods still struggle to ensure the controllability
  of text prompts over images in the context of complex text prompts, especially
  when it comes to retaining object attributes and relationships. In this paper,
  we propose CompAgent, a training-free approach for compositional text-to-image
  generation, with a large language model (LLM) agent as its core. The
  fundamental idea underlying CompAgent is premised on a divide-and-conquer
  methodology. Given a complex text prompt containing multiple concepts including
  objects, attributes, and relationships, the LLM agent initially decomposes it,
  which entails the extraction of individual objects, their associated
  attributes, and the prediction of a coherent scene layout. These individual
  objects can then be independently conquered. Subsequently, the agent performs
  reasoning by analyzing the text, plans and employs the tools to compose these
  isolated objects. The verification and human feedback mechanism is finally
  incorporated into our agent to further correct the potential attribute errors
  and refine the generated images. Guided by the LLM agent, we propose a
  tuning-free multi-concept customization model and a layout-to-image generation
  model as the tools for concept composition, and a local image editing method as
  the tool to interact with the agent for verification. The scene layout controls
  the image generation process among these tools to prevent confusion among
  multiple objects. Extensive experiments demonstrate the superiority of our
  approach for compositional text-to-image generation: CompAgent achieves more
  than 10\% improvement on T2I-CompBench, a comprehensive benchmark for
  open-world compositional T2I generation. The extension to various related tasks
  also illustrates the flexibility of our CompAgent for potential applications.
paper_ai_summary: CompAgent, a training-free method using an LLM agent, improves compositional text-to-image generation by decomposing complex text prompts, reasoning, and integrating multiple tools to enhance image quality and controllability.
paper_ai_keywords: CompAgent large language model (LLM) compositional text-to-image generation divide-and-conquer scene layout concept composition layout-to-image generation model local image editing method T2I-CompBench