title: Unified Multimodal Understanding and Generation Models: Advances,
    Challenges, and Opportunities
summary: Recent years have seen remarkable progress in both multimodal understanding
  models and image generation models. Despite their respective successes, these
  two domains have evolved independently, leading to distinct architectural
  paradigms: While autoregressive-based architectures have dominated multimodal
  understanding, diffusion-based models have become the cornerstone of image
  generation. Recently, there has been growing interest in developing unified
  frameworks that integrate these tasks. The emergence of GPT-4o's new
  capabilities exemplifies this trend, highlighting the potential for
  unification. However, the architectural differences between the two domains
  pose significant challenges. To provide a clear overview of current efforts
  toward unification, we present a comprehensive survey aimed at guiding future
  research. First, we introduce the foundational concepts and recent advancements
  in multimodal understanding and text-to-image generation models. Next, we
  review existing unified models, categorizing them into three main architectural
  paradigms: diffusion-based, autoregressive-based, and hybrid approaches that
  fuse autoregressive and diffusion mechanisms. For each category, we analyze the
  structural designs and innovations introduced by related works. Additionally,
  we compile datasets and benchmarks tailored for unified models, offering
  resources for future exploration. Finally, we discuss the key challenges facing
  this nascent field, including tokenization strategy, cross-modal attention, and
  data. As this area is still in its early stages, we anticipate rapid
  advancements and will regularly update this survey. Our goal is to inspire
  further research and provide a valuable reference for the community. The
  references associated with this survey are available on GitHub
  (https://github.com/AIDC-AI/Awesome-Unified-Multimodal-Models).
paper_authors._id: 681c7895c7211b7efbc49f17
paper_authors.name: Xinjie Zhang
paper_authors._id: 681c7895c7211b7efbc49f18
paper_authors.name: Jintao Guo
paper_authors._id: 681c7895c7211b7efbc49f19
paper_authors.name: Shanshan Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-08T10:07:03.107Z
paper_authors._id: 66ab4c8a1703f12f49583c6d
paper_authors.avatarUrl: /avatars/59c77d4556edc049bb410e180813d5e3.svg
paper_authors.fullname: zss
paper_authors.type: user
paper_authors.user: Suikong
paper_authors._id: 681c7895c7211b7efbc49f1a
paper_authors.name: Minghao Fu
paper_authors._id: 681c7895c7211b7efbc49f1b
paper_authors.name: Lunhao Duan
paper_authors._id: 681c7895c7211b7efbc49f1c
paper_authors.name: Guo-Hua Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-08T09:58:04.757Z
paper_authors._id: 636f4c6b5d2050767e4a1491
paper_authors.avatarUrl: /avatars/630c2ae12937fdb16ccd3280bc05729d.svg
paper_authors.fullname: Guo-Hua Wang
paper_authors.type: user
paper_authors.user: Flourish
paper_authors._id: 681c7895c7211b7efbc49f1d
paper_authors.name: Qing-Guo Chen
paper_authors._id: 681c7895c7211b7efbc49f1e
paper_authors.name: Zhao Xu
paper_authors._id: 681c7895c7211b7efbc49f1f
paper_authors.name: Weihua Luo
paper_authors._id: 681c7895c7211b7efbc49f20
paper_authors.name: Kaifu Zhang
paper_title: Unified Multimodal Understanding and Generation Models: Advances,
    Challenges, and Opportunities
paper_summary: Recent years have seen remarkable progress in both multimodal understanding
  models and image generation models. Despite their respective successes, these
  two domains have evolved independently, leading to distinct architectural
  paradigms: While autoregressive-based architectures have dominated multimodal
  understanding, diffusion-based models have become the cornerstone of image
  generation. Recently, there has been growing interest in developing unified
  frameworks that integrate these tasks. The emergence of GPT-4o's new
  capabilities exemplifies this trend, highlighting the potential for
  unification. However, the architectural differences between the two domains
  pose significant challenges. To provide a clear overview of current efforts
  toward unification, we present a comprehensive survey aimed at guiding future
  research. First, we introduce the foundational concepts and recent advancements
  in multimodal understanding and text-to-image generation models. Next, we
  review existing unified models, categorizing them into three main architectural
  paradigms: diffusion-based, autoregressive-based, and hybrid approaches that
  fuse autoregressive and diffusion mechanisms. For each category, we analyze the
  structural designs and innovations introduced by related works. Additionally,
  we compile datasets and benchmarks tailored for unified models, offering
  resources for future exploration. Finally, we discuss the key challenges facing
  this nascent field, including tokenization strategy, cross-modal attention, and
  data. As this area is still in its early stages, we anticipate rapid
  advancements and will regularly update this survey. Our goal is to inspire
  further research and provide a valuable reference for the community. The
  references associated with this survey are available on GitHub
  (https://github.com/AIDC-AI/Awesome-Unified-Multimodal-Models).
paper_ai_summary: A survey examines the integration of multimodal understanding and image generation models, categorizing them into diffusion-based, autoregressive-based, and hybrid approaches, and discussing challenges and future research directions.
paper_ai_keywords: multimodal understanding diffusion-based models autoregressive-based models hybrid approaches cross-modal attention