Exploring the Deep Fusion of Large Language Models and Diffusion
  Transformers for Text-to-Image Synthesis
This paper does not describe a new method; instead, it provides a thorough
exploration of an important yet understudied design space related to recent
advances in text-to-image synthesis -- specifically, the deep fusion of large
language models (LLMs) and diffusion transformers (DiTs) for multi-modal
generation. Previous studies mainly focused on overall system performance
rather than detailed comparisons with alternative methods, and key design
details and training recipes were often left undisclosed. These gaps create
uncertainty about the real potential of this approach. To fill these gaps, we
conduct an empirical study on text-to-image generation, performing controlled
comparisons with established baselines, analyzing important design choices, and
providing a clear, reproducible recipe for training at scale. We hope this work
offers meaningful data points and practical guidelines for future research in
multi-modal generation.
6826b00ed4c8b864e5ed1c0f
Bingda Tang
admin_assigned
2025-05-16T14:21:21.171Z
666de415ea11ab8f28560962
/avatars/c910b35e3128cf17ba1e45afff551820.svg
Bingda Tang
user
ooutlierr
6826b00ed4c8b864e5ed1c10
Boyang Zheng
admin_assigned
2025-05-16T14:21:27.353Z
6434226da4c9c55871a78052
/avatars/3309832b3115bc6ad08ae1d10f43118b.svg
BoYang Zheng
user
bytetriper
6826b00ed4c8b864e5ed1c11
Xichen Pan
admin_assigned
2025-05-16T14:21:34.655Z
63172831c92fd6fee3181f50
/avatars/0f57068a138cb181e9451bfc1ed3d1c0.svg
Xichen Pan
user
xcpan
6826b00ed4c8b864e5ed1c12
Sayak Paul
claimed_verified
2025-05-16T13:23:23.708Z
5f7fbd813e94f16a85448745
https://cdn-avatars.huggingface.co/v1/production/uploads/1649681653581-5f7fbd813e94f16a85448745.jpeg
Sayak Paul
user
sayakpaul
6826b00ed4c8b864e5ed1c13
Saining Xie
admin_assigned
2025-05-16T14:21:41.504Z
6596422646624a86ff3b3bda
/avatars/216e12b77e45ac5f1fa20932f5745411.svg
Saining Xie
user
sainx
Exploring the Deep Fusion of Large Language Models and Diffusion
  Transformers for Text-to-Image Synthesis
This paper does not describe a new method; instead, it provides a thorough
exploration of an important yet understudied design space related to recent
advances in text-to-image synthesis -- specifically, the deep fusion of large
language models (LLMs) and diffusion transformers (DiTs) for multi-modal
generation. Previous studies mainly focused on overall system performance
rather than detailed comparisons with alternative methods, and key design
details and training recipes were often left undisclosed. These gaps create
uncertainty about the real potential of this approach. To fill these gaps, we
conduct an empirical study on text-to-image generation, performing controlled
comparisons with established baselines, analyzing important design choices, and
providing a clear, reproducible recipe for training at scale. We hope this work
offers meaningful data points and practical guidelines for future research in
multi-modal generation.
Empirical exploration of text-to-image synthesis focuses on the deep fusion of large language models and diffusion transformers, providing reproducible guidelines and insights.
large language models diffusion transformers multi-modal generation empirical study controlled comparisons baseline design choices reproducible recipe training at scale