title: Exploring the Role of Large Language Models in Prompt Encoding for
    Diffusion Models
summary: Large language models (LLMs) based on decoder-only transformers have
  demonstrated superior text understanding capabilities compared to CLIP and
  T5-series models. However, the paradigm for utilizing current advanced LLMs in
  text-to-image diffusion models remains to be explored. We observed an unusual
  phenomenon: directly using a large language model as the prompt encoder
  significantly degrades the prompt-following ability in image generation. We
  identified two main obstacles behind this issue. One is the misalignment
  between the next token prediction training in LLM and the requirement for
  discriminative prompt features in diffusion models. The other is the intrinsic
  positional bias introduced by the decoder-only architecture. To deal with this
  issue, we propose a novel framework to fully harness the capabilities of LLMs.
  Through the carefully designed usage guidance, we effectively enhance the text
  representation capability for prompt encoding and eliminate its inherent
  positional bias. This allows us to integrate state-of-the-art LLMs into the
  text-to-image generation model flexibly. Furthermore, we also provide an
  effective manner to fuse multiple LLMs into our framework. Considering the
  excellent performance and scaling capabilities demonstrated by the transformer
  architecture, we further design an LLM-Infused Diffusion Transformer (LI-DiT)
  based on the framework. We conduct extensive experiments to validate LI-DiT
  across model size and data size. Benefiting from the inherent ability of the
  LLMs and our innovative designs, the prompt understanding performance of LI-DiT
  easily surpasses state-of-the-art open-source models as well as mainstream
  closed-source commercial models including Stable Diffusion 3, DALL-E 3, and
  Midjourney V6. The powerful LI-DiT-10B will be available after further
  optimization and security checks.
paper_authors._id: 66718799d059806ae6ee5b6b
paper_authors.name: Bingqi Ma
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T15:33:46.646Z
paper_authors._id: 64424d1d5933b4560698ff86
paper_authors.avatarUrl: /avatars/f44d73744796151eda0ef0035ad10a2b.svg
paper_authors.fullname: bingqi ma
paper_authors.type: user
paper_authors.user: A157801
paper_authors._id: 66718799d059806ae6ee5b6c
paper_authors.name: Zhuofan Zong
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T13:44:18.963Z
paper_authors._id: 64db05f23725f8d9a9f566c4
paper_authors.avatarUrl: /avatars/345b6d2dfadb91ac1f02be023dd27c8f.svg
paper_authors.fullname: Zhuofan Zong
paper_authors.type: user
paper_authors.user: zongzhuofan
paper_authors._id: 66718799d059806ae6ee5b6d
paper_authors.name: Guanglu Song
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T15:34:04.069Z
paper_authors._id: 636a2efffbc31ee68fee5c1c
paper_authors.avatarUrl: /avatars/21e119f23e1e6fe926567409e74d1a89.svg
paper_authors.fullname: Song
paper_authors.type: user
paper_authors.user: Guanglu
paper_authors._id: 66718799d059806ae6ee5b6e
paper_authors.name: Hongsheng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T15:34:25.224Z
paper_authors._id: 65c04e9c27a5fdca81abcbd9
paper_authors.avatarUrl: /avatars/12a155683c824fa23da4a9e2bed4f64e.svg
paper_authors.fullname: Hongsheng LI
paper_authors.type: user
paper_authors.user: hsli-cuhk
paper_authors._id: 66718799d059806ae6ee5b6f
paper_authors.name: Yu Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-18T15:34:39.370Z
paper_authors._id: 636de85cc4a7a729c164d2b5
paper_authors.avatarUrl: /avatars/3e281e547e1697e1c06805e7e63f3918.svg
paper_authors.fullname: Yu Liu
paper_authors.type: user
paper_authors.user: YuLiu
paper_title: Exploring the Role of Large Language Models in Prompt Encoding for
    Diffusion Models
paper_summary: Large language models (LLMs) based on decoder-only transformers have
  demonstrated superior text understanding capabilities compared to CLIP and
  T5-series models. However, the paradigm for utilizing current advanced LLMs in
  text-to-image diffusion models remains to be explored. We observed an unusual
  phenomenon: directly using a large language model as the prompt encoder
  significantly degrades the prompt-following ability in image generation. We
  identified two main obstacles behind this issue. One is the misalignment
  between the next token prediction training in LLM and the requirement for
  discriminative prompt features in diffusion models. The other is the intrinsic
  positional bias introduced by the decoder-only architecture. To deal with this
  issue, we propose a novel framework to fully harness the capabilities of LLMs.
  Through the carefully designed usage guidance, we effectively enhance the text
  representation capability for prompt encoding and eliminate its inherent
  positional bias. This allows us to integrate state-of-the-art LLMs into the
  text-to-image generation model flexibly. Furthermore, we also provide an
  effective manner to fuse multiple LLMs into our framework. Considering the
  excellent performance and scaling capabilities demonstrated by the transformer
  architecture, we further design an LLM-Infused Diffusion Transformer (LI-DiT)
  based on the framework. We conduct extensive experiments to validate LI-DiT
  across model size and data size. Benefiting from the inherent ability of the
  LLMs and our innovative designs, the prompt understanding performance of LI-DiT
  easily surpasses state-of-the-art open-source models as well as mainstream
  closed-source commercial models including Stable Diffusion 3, DALL-E 3, and
  Midjourney V6. The powerful LI-DiT-10B will be available after further
  optimization and security checks.
paper_ai_summary: A novel framework and LLM-Infused Diffusion Transformer (LI-DiT) improve text-to-image generation by effectively integrating state-of-the-art large language models and overcoming prompt encoding limitations.
paper_ai_keywords: decoder-only transformers CLIP T5-series models text-to-image diffusion models prompt encoder next token prediction discriminative prompt features positional bias usage guidance text representation capability LLM-Infused Diffusion Transformer LI-DiT State-of-the-art LLMs Stable Diffusion 3 DALL-E 3 Midjourney V6