title: ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment
summary: Diffusion models have demonstrated remarkable performance in the domain of
  text-to-image generation. However, most widely used models still employ CLIP as
  their text encoder, which constrains their ability to comprehend dense prompts,
  encompassing multiple objects, detailed attributes, complex relationships,
  long-text alignment, etc. In this paper, we introduce an Efficient Large
  Language Model Adapter, termed ELLA, which equips text-to-image diffusion
  models with powerful Large Language Models (LLM) to enhance text alignment
  without training of either U-Net or LLM. To seamlessly bridge two pre-trained
  models, we investigate a range of semantic alignment connector designs and
  propose a novel module, the Timestep-Aware Semantic Connector (TSC), which
  dynamically extracts timestep-dependent conditions from LLM. Our approach
  adapts semantic features at different stages of the denoising process,
  assisting diffusion models in interpreting lengthy and intricate prompts over
  sampling timesteps. Additionally, ELLA can be readily incorporated with
  community models and tools to improve their prompt-following capabilities. To
  assess text-to-image models in dense prompt following, we introduce Dense
  Prompt Graph Benchmark (DPG-Bench), a challenging benchmark consisting of 1K
  dense prompts. Extensive experiments demonstrate the superiority of ELLA in
  dense prompt following compared to state-of-the-art methods, particularly in
  multiple object compositions involving diverse attributes and relationships.
paper_authors._id: 65ee79a2e28439e21ba8b1d8
paper_authors.name: Xiwei Hu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-12T12:06:20.855Z
paper_authors._id: 6436766d0b2c0d86d0b11d68
paper_authors.avatarUrl: /avatars/1157cdaf47c313d1e879049a987ed0a6.svg
paper_authors.fullname: Melodie Hu
paper_authors.type: user
paper_authors.user: melohux
paper_authors._id: 65ee79a2e28439e21ba8b1d9
paper_authors.name: Rui Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-11T08:20:03.461Z
paper_authors._id: 630443483926de1f7ec6bda0
paper_authors.avatarUrl: /avatars/e8e95ab0d03f58c61846ccc1318502fe.svg
paper_authors.fullname: Rui Wang
paper_authors.type: user
paper_authors.user: budui
paper_authors._id: 65ee79a2e28439e21ba8b1da
paper_authors.name: Yixiao Fang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-11T08:36:54.000Z
paper_authors._id: 647469b9a51711a3b58bda2b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/647469b9a51711a3b58bda2b/yeDf8Sa8IDEQyney1dGC9.jpeg
paper_authors.fullname: Yixiao Fang
paper_authors.type: user
paper_authors.user: fangyixiao
paper_authors._id: 65ee79a2e28439e21ba8b1db
paper_authors.name: Bin Fu
paper_authors._id: 65ee79a2e28439e21ba8b1dc
paper_authors.name: Pei Cheng
paper_authors._id: 65ee79a2e28439e21ba8b1dd
paper_authors.name: Gang Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-11T08:27:59.050Z
paper_authors._id: 63417332c5565a4b8d43a0d8
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63417332c5565a4b8d43a0d8/MmnYG7Wu2Z_lqCBvTfJmy.png
paper_authors.fullname: Gang Yu
paper_authors.type: user
paper_authors.user: skicy
paper_title: ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment
paper_summary: Diffusion models have demonstrated remarkable performance in the domain of
  text-to-image generation. However, most widely used models still employ CLIP as
  their text encoder, which constrains their ability to comprehend dense prompts,
  encompassing multiple objects, detailed attributes, complex relationships,
  long-text alignment, etc. In this paper, we introduce an Efficient Large
  Language Model Adapter, termed ELLA, which equips text-to-image diffusion
  models with powerful Large Language Models (LLM) to enhance text alignment
  without training of either U-Net or LLM. To seamlessly bridge two pre-trained
  models, we investigate a range of semantic alignment connector designs and
  propose a novel module, the Timestep-Aware Semantic Connector (TSC), which
  dynamically extracts timestep-dependent conditions from LLM. Our approach
  adapts semantic features at different stages of the denoising process,
  assisting diffusion models in interpreting lengthy and intricate prompts over
  sampling timesteps. Additionally, ELLA can be readily incorporated with
  community models and tools to improve their prompt-following capabilities. To
  assess text-to-image models in dense prompt following, we introduce Dense
  Prompt Graph Benchmark (DPG-Bench), a challenging benchmark consisting of 1K
  dense prompts. Extensive experiments demonstrate the superiority of ELLA in
  dense prompt following compared to state-of-the-art methods, particularly in
  multiple object compositions involving diverse attributes and relationships.
paper_ai_keywords: diffusion models text-to-image generation CLIP text encoder dense prompts Efficient Large Language Model Adapter ELLA Large Language Models U-Net semantic alignment connector Timestep-Aware Semantic Connector TSC semantic features denoising process dense prompt following Dense Prompt Graph Benchmark DPG-Bench multiple object compositions diverse attributes relationships