Mastering Text-to-Image Diffusion: Recaptioning, Planning, and
  Generating with Multimodal LLMs
Diffusion models have exhibit exceptional performance in text-to-image
generation and editing. However, existing methods often face challenges when
handling complex text prompts that involve multiple objects with multiple
attributes and relationships. In this paper, we propose a brand new
training-free text-to-image generation/editing framework, namely Recaption,
Plan and Generate (RPG), harnessing the powerful chain-of-thought reasoning
ability of multimodal LLMs to enhance the compositionality of text-to-image
diffusion models. Our approach employs the MLLM as a global planner to
decompose the process of generating complex images into multiple simpler
generation tasks within subregions. We propose complementary regional diffusion
to enable region-wise compositional generation. Furthermore, we integrate
text-guided image generation and editing within the proposed RPG in a
closed-loop fashion, thereby enhancing generalization ability. Extensive
experiments demonstrate our RPG outperforms state-of-the-art text-to-image
diffusion models, including DALL-E 3 and SDXL, particularly in multi-category
object composition and text-image semantic alignment. Notably, our RPG
framework exhibits wide compatibility with various MLLM architectures (e.g.,
MiniGPT-4) and diffusion backbones (e.g., ControlNet). Our code is available
at: https://github.com/YangLing0818/RPG-DiffusionMaster
65af1fe0101482afcc5e0c19
Ling Yang
claimed_verified
2024-01-23T13:49:03.172Z
64fde4e252e82dd432b74ce9
https://cdn-avatars.huggingface.co/v1/production/uploads/64fde4e252e82dd432b74ce9/-CQZbBP7FsPPyawYrsi4z.jpeg
Ling Yang
user
Lingaaaaaaa
65af1fe0101482afcc5e0c1a
Zhaochen Yu
65af1fe0101482afcc5e0c1b
Chenlin Meng
65af1fe0101482afcc5e0c1c
Minkai Xu
admin_assigned
2024-01-23T09:48:26.326Z
64c0e950aa57599de1c75dad
/avatars/374d53317cbccc30fae70e5152ca13e0.svg
Minkai Xu
user
mkxu
65af1fe0101482afcc5e0c1d
Stefano Ermon
admin_assigned
2024-01-23T09:48:14.407Z
62f6e244329d4d014d1f4ac5
/avatars/5a8b2bb063c2ebc340504b22530f6811.svg
Stefano Ermon
user
ermonste
65af1fe0101482afcc5e0c1e
Bin Cui
Mastering Text-to-Image Diffusion: Recaptioning, Planning, and
  Generating with Multimodal LLMs
Diffusion models have exhibit exceptional performance in text-to-image
generation and editing. However, existing methods often face challenges when
handling complex text prompts that involve multiple objects with multiple
attributes and relationships. In this paper, we propose a brand new
training-free text-to-image generation/editing framework, namely Recaption,
Plan and Generate (RPG), harnessing the powerful chain-of-thought reasoning
ability of multimodal LLMs to enhance the compositionality of text-to-image
diffusion models. Our approach employs the MLLM as a global planner to
decompose the process of generating complex images into multiple simpler
generation tasks within subregions. We propose complementary regional diffusion
to enable region-wise compositional generation. Furthermore, we integrate
text-guided image generation and editing within the proposed RPG in a
closed-loop fashion, thereby enhancing generalization ability. Extensive
experiments demonstrate our RPG outperforms state-of-the-art text-to-image
diffusion models, including DALL-E 3 and SDXL, particularly in multi-category
object composition and text-image semantic alignment. Notably, our RPG
framework exhibits wide compatibility with various MLLM architectures (e.g.,
MiniGPT-4) and diffusion backbones (e.g., ControlNet). Our code is available
at: https://github.com/YangLing0818/RPG-DiffusionMaster
A text-to-image generation/editing framework, RPG, uses multimodal LLMs for chain-of-thought reasoning to enhance the compositionality and performance of diffusion models, particularly in handling complex text prompts.
diffusion models text-to-image generation editing chain-of-thought reasoning multimodal LLMs complementary regional diffusion closed-loop fashion DALL-E 3 SDXL MiniGPT-4 ControlNet