title: Mastering Text-to-Image Diffusion: Recaptioning, Planning, and
    Generating with Multimodal LLMs
summary: Diffusion models have exhibit exceptional performance in text-to-image
  generation and editing. However, existing methods often face challenges when
  handling complex text prompts that involve multiple objects with multiple
  attributes and relationships. In this paper, we propose a brand new
  training-free text-to-image generation/editing framework, namely Recaption,
  Plan and Generate (RPG), harnessing the powerful chain-of-thought reasoning
  ability of multimodal LLMs to enhance the compositionality of text-to-image
  diffusion models. Our approach employs the MLLM as a global planner to
  decompose the process of generating complex images into multiple simpler
  generation tasks within subregions. We propose complementary regional diffusion
  to enable region-wise compositional generation. Furthermore, we integrate
  text-guided image generation and editing within the proposed RPG in a
  closed-loop fashion, thereby enhancing generalization ability. Extensive
  experiments demonstrate our RPG outperforms state-of-the-art text-to-image
  diffusion models, including DALL-E 3 and SDXL, particularly in multi-category
  object composition and text-image semantic alignment. Notably, our RPG
  framework exhibits wide compatibility with various MLLM architectures (e.g.,
  MiniGPT-4) and diffusion backbones (e.g., ControlNet). Our code is available
  at: https://github.com/YangLing0818/RPG-DiffusionMaster
paper_authors._id: 65af1fe0101482afcc5e0c19
paper_authors.name: Ling Yang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-23T13:49:03.172Z
paper_authors._id: 64fde4e252e82dd432b74ce9
paper_authors.avatarUrl: /avatars/061a69d858b86d1600be916122cae7fc.svg
paper_authors.fullname: Ling Yang
paper_authors.type: user
paper_authors.user: Lingaaaaaaa
paper_authors._id: 65af1fe0101482afcc5e0c1a
paper_authors.name: Zhaochen Yu
paper_authors._id: 65af1fe0101482afcc5e0c1b
paper_authors.name: Chenlin Meng
paper_authors._id: 65af1fe0101482afcc5e0c1c
paper_authors.name: Minkai Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-23T09:48:26.326Z
paper_authors._id: 64c0e950aa57599de1c75dad
paper_authors.avatarUrl: /avatars/374d53317cbccc30fae70e5152ca13e0.svg
paper_authors.fullname: Minkai Xu
paper_authors.type: user
paper_authors.user: mkxu
paper_authors._id: 65af1fe0101482afcc5e0c1d
paper_authors.name: Stefano Ermon
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-01-23T09:48:14.407Z
paper_authors._id: 62f6e244329d4d014d1f4ac5
paper_authors.avatarUrl: /avatars/5a8b2bb063c2ebc340504b22530f6811.svg
paper_authors.fullname: Stefano Ermon
paper_authors.type: user
paper_authors.user: ermonste
paper_authors._id: 65af1fe0101482afcc5e0c1e
paper_authors.name: Bin Cui
paper_title: Mastering Text-to-Image Diffusion: Recaptioning, Planning, and
    Generating with Multimodal LLMs
paper_summary: Diffusion models have exhibit exceptional performance in text-to-image
  generation and editing. However, existing methods often face challenges when
  handling complex text prompts that involve multiple objects with multiple
  attributes and relationships. In this paper, we propose a brand new
  training-free text-to-image generation/editing framework, namely Recaption,
  Plan and Generate (RPG), harnessing the powerful chain-of-thought reasoning
  ability of multimodal LLMs to enhance the compositionality of text-to-image
  diffusion models. Our approach employs the MLLM as a global planner to
  decompose the process of generating complex images into multiple simpler
  generation tasks within subregions. We propose complementary regional diffusion
  to enable region-wise compositional generation. Furthermore, we integrate
  text-guided image generation and editing within the proposed RPG in a
  closed-loop fashion, thereby enhancing generalization ability. Extensive
  experiments demonstrate our RPG outperforms state-of-the-art text-to-image
  diffusion models, including DALL-E 3 and SDXL, particularly in multi-category
  object composition and text-image semantic alignment. Notably, our RPG
  framework exhibits wide compatibility with various MLLM architectures (e.g.,
  MiniGPT-4) and diffusion backbones (e.g., ControlNet). Our code is available
  at: https://github.com/YangLing0818/RPG-DiffusionMaster
paper_ai_summary: A text-to-image generation/editing framework, RPG, uses multimodal LLMs for chain-of-thought reasoning to enhance the compositionality and performance of diffusion models, particularly in handling complex text prompts.
paper_ai_keywords: diffusion models text-to-image generation editing chain-of-thought reasoning multimodal LLMs complementary regional diffusion closed-loop fashion DALL-E 3 SDXL MiniGPT-4 ControlNet