Exploring the Role of Large Language Models in Prompt Encoding for
  Diffusion Models
Large language models (LLMs) based on decoder-only transformers have
demonstrated superior text understanding capabilities compared to CLIP and
T5-series models. However, the paradigm for utilizing current advanced LLMs in
text-to-image diffusion models remains to be explored. We observed an unusual
phenomenon: directly using a large language model as the prompt encoder
significantly degrades the prompt-following ability in image generation. We
identified two main obstacles behind this issue. One is the misalignment
between the next token prediction training in LLM and the requirement for
discriminative prompt features in diffusion models. The other is the intrinsic
positional bias introduced by the decoder-only architecture. To deal with this
issue, we propose a novel framework to fully harness the capabilities of LLMs.
Through the carefully designed usage guidance, we effectively enhance the text
representation capability for prompt encoding and eliminate its inherent
positional bias. This allows us to integrate state-of-the-art LLMs into the
text-to-image generation model flexibly. Furthermore, we also provide an
effective manner to fuse multiple LLMs into our framework. Considering the
excellent performance and scaling capabilities demonstrated by the transformer
architecture, we further design an LLM-Infused Diffusion Transformer (LI-DiT)
based on the framework. We conduct extensive experiments to validate LI-DiT
across model size and data size. Benefiting from the inherent ability of the
LLMs and our innovative designs, the prompt understanding performance of LI-DiT
easily surpasses state-of-the-art open-source models as well as mainstream
closed-source commercial models including Stable Diffusion 3, DALL-E 3, and
Midjourney V6. The powerful LI-DiT-10B will be available after further
optimization and security checks.
66718799d059806ae6ee5b6b
Bingqi Ma
claimed_verified
2024-06-18T15:33:46.646Z
64424d1d5933b4560698ff86
/avatars/f44d73744796151eda0ef0035ad10a2b.svg
bingqi ma
user
A157801
66718799d059806ae6ee5b6c
Zhuofan Zong
claimed_verified
2024-06-18T13:44:18.963Z
64db05f23725f8d9a9f566c4
/avatars/345b6d2dfadb91ac1f02be023dd27c8f.svg
Zhuofan Zong
user
zongzhuofan
66718799d059806ae6ee5b6d
Guanglu Song
admin_assigned
2024-06-18T15:34:04.069Z
636a2efffbc31ee68fee5c1c
/avatars/21e119f23e1e6fe926567409e74d1a89.svg
Song
user
Guanglu
66718799d059806ae6ee5b6e
Hongsheng Li
admin_assigned
2024-06-18T15:34:25.224Z
65c04e9c27a5fdca81abcbd9
/avatars/12a155683c824fa23da4a9e2bed4f64e.svg
Hongsheng LI
user
hsli-cuhk
66718799d059806ae6ee5b6f
Yu Liu
admin_assigned
2024-06-18T15:34:39.370Z
636de85cc4a7a729c164d2b5
/avatars/3e281e547e1697e1c06805e7e63f3918.svg
Yu Liu
user
YuLiu
Exploring the Role of Large Language Models in Prompt Encoding for
  Diffusion Models
Large language models (LLMs) based on decoder-only transformers have
demonstrated superior text understanding capabilities compared to CLIP and
T5-series models. However, the paradigm for utilizing current advanced LLMs in
text-to-image diffusion models remains to be explored. We observed an unusual
phenomenon: directly using a large language model as the prompt encoder
significantly degrades the prompt-following ability in image generation. We
identified two main obstacles behind this issue. One is the misalignment
between the next token prediction training in LLM and the requirement for
discriminative prompt features in diffusion models. The other is the intrinsic
positional bias introduced by the decoder-only architecture. To deal with this
issue, we propose a novel framework to fully harness the capabilities of LLMs.
Through the carefully designed usage guidance, we effectively enhance the text
representation capability for prompt encoding and eliminate its inherent
positional bias. This allows us to integrate state-of-the-art LLMs into the
text-to-image generation model flexibly. Furthermore, we also provide an
effective manner to fuse multiple LLMs into our framework. Considering the
excellent performance and scaling capabilities demonstrated by the transformer
architecture, we further design an LLM-Infused Diffusion Transformer (LI-DiT)
based on the framework. We conduct extensive experiments to validate LI-DiT
across model size and data size. Benefiting from the inherent ability of the
LLMs and our innovative designs, the prompt understanding performance of LI-DiT
easily surpasses state-of-the-art open-source models as well as mainstream
closed-source commercial models including Stable Diffusion 3, DALL-E 3, and
Midjourney V6. The powerful LI-DiT-10B will be available after further
optimization and security checks.
A novel framework and LLM-Infused Diffusion Transformer (LI-DiT) improve text-to-image generation by effectively integrating state-of-the-art large language models and overcoming prompt encoding limitations.
decoder-only transformers CLIP T5-series models text-to-image diffusion models prompt encoder next token prediction discriminative prompt features positional bias usage guidance text representation capability LLM-Infused Diffusion Transformer LI-DiT State-of-the-art LLMs Stable Diffusion 3 DALL-E 3 Midjourney V6