Unified Multimodal Understanding and Generation Models: Advances,
  Challenges, and Opportunities
Recent years have seen remarkable progress in both multimodal understanding
models and image generation models. Despite their respective successes, these
two domains have evolved independently, leading to distinct architectural
paradigms: While autoregressive-based architectures have dominated multimodal
understanding, diffusion-based models have become the cornerstone of image
generation. Recently, there has been growing interest in developing unified
frameworks that integrate these tasks. The emergence of GPT-4o's new
capabilities exemplifies this trend, highlighting the potential for
unification. However, the architectural differences between the two domains
pose significant challenges. To provide a clear overview of current efforts
toward unification, we present a comprehensive survey aimed at guiding future
research. First, we introduce the foundational concepts and recent advancements
in multimodal understanding and text-to-image generation models. Next, we
review existing unified models, categorizing them into three main architectural
paradigms: diffusion-based, autoregressive-based, and hybrid approaches that
fuse autoregressive and diffusion mechanisms. For each category, we analyze the
structural designs and innovations introduced by related works. Additionally,
we compile datasets and benchmarks tailored for unified models, offering
resources for future exploration. Finally, we discuss the key challenges facing
this nascent field, including tokenization strategy, cross-modal attention, and
data. As this area is still in its early stages, we anticipate rapid
advancements and will regularly update this survey. Our goal is to inspire
further research and provide a valuable reference for the community. The
references associated with this survey are available on GitHub
(https://github.com/AIDC-AI/Awesome-Unified-Multimodal-Models).
681c7895c7211b7efbc49f17
Xinjie Zhang
681c7895c7211b7efbc49f18
Jintao Guo
681c7895c7211b7efbc49f19
Shanshan Zhao
claimed_verified
2025-05-08T10:07:03.107Z
66ab4c8a1703f12f49583c6d
/avatars/59c77d4556edc049bb410e180813d5e3.svg
zss
user
Suikong
681c7895c7211b7efbc49f1a
Minghao Fu
681c7895c7211b7efbc49f1b
Lunhao Duan
681c7895c7211b7efbc49f1c
Guo-Hua Wang
claimed_verified
2025-05-08T09:58:04.757Z
636f4c6b5d2050767e4a1491
https://cdn-avatars.huggingface.co/v1/production/uploads/636f4c6b5d2050767e4a1491/qErBLQnYzL35EkbDAhvqN.jpeg
Guo-Hua Wang
user
Flourish
681c7895c7211b7efbc49f1d
Qing-Guo Chen
681c7895c7211b7efbc49f1e
Zhao Xu
681c7895c7211b7efbc49f1f
Weihua Luo
681c7895c7211b7efbc49f20
Kaifu Zhang
Unified Multimodal Understanding and Generation Models: Advances,
  Challenges, and Opportunities
Recent years have seen remarkable progress in both multimodal understanding
models and image generation models. Despite their respective successes, these
two domains have evolved independently, leading to distinct architectural
paradigms: While autoregressive-based architectures have dominated multimodal
understanding, diffusion-based models have become the cornerstone of image
generation. Recently, there has been growing interest in developing unified
frameworks that integrate these tasks. The emergence of GPT-4o's new
capabilities exemplifies this trend, highlighting the potential for
unification. However, the architectural differences between the two domains
pose significant challenges. To provide a clear overview of current efforts
toward unification, we present a comprehensive survey aimed at guiding future
research. First, we introduce the foundational concepts and recent advancements
in multimodal understanding and text-to-image generation models. Next, we
review existing unified models, categorizing them into three main architectural
paradigms: diffusion-based, autoregressive-based, and hybrid approaches that
fuse autoregressive and diffusion mechanisms. For each category, we analyze the
structural designs and innovations introduced by related works. Additionally,
we compile datasets and benchmarks tailored for unified models, offering
resources for future exploration. Finally, we discuss the key challenges facing
this nascent field, including tokenization strategy, cross-modal attention, and
data. As this area is still in its early stages, we anticipate rapid
advancements and will regularly update this survey. Our goal is to inspire
further research and provide a valuable reference for the community. The
references associated with this survey are available on GitHub
(https://github.com/AIDC-AI/Awesome-Unified-Multimodal-Models).
A survey examines the integration of multimodal understanding and image generation models, categorizing them into diffusion-based, autoregressive-based, and hybrid approaches, and discussing challenges and future research directions.
multimodal understanding diffusion-based models autoregressive-based models hybrid approaches cross-modal attention