title: MM-LLMs: Recent Advances in MultiModal Large Language Models
summary: In the past year, MultiModal Large Language Models (MM-LLMs) have undergone
  substantial advancements, augmenting off-the-shelf LLMs to support MM inputs or
  outputs via cost-effective training strategies. The resulting models not only
  preserve the inherent reasoning and decision-making capabilities of LLMs but
  also empower a diverse range of MM tasks. In this paper, we provide a
  comprehensive survey aimed at facilitating further research of MM-LLMs.
  Specifically, we first outline general design formulations for model
  architecture and training pipeline. Subsequently, we provide brief
  introductions of 26 existing MM-LLMs, each characterized by its specific
  formulations. Additionally, we review the performance of MM-LLMs on mainstream
  benchmarks and summarize key training recipes to enhance the potency of
  MM-LLMs. Lastly, we explore promising directions for MM-LLMs while concurrently
  maintaining a real-time tracking website for the latest developments in the
  field. We hope that this survey contributes to the ongoing advancement of the
  MM-LLMs domain.
paper_authors._id: 65b1ea38ce74220b8315682e
paper_authors.name: Duzhen Zhang
paper_authors._id: 65b1ea38ce74220b8315682f
paper_authors.name: Yahan Yu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-02-27T13:49:13.051Z
paper_authors._id: 65ddd02438ad9c41e2b1f54f
paper_authors.avatarUrl: /avatars/dc584e49e26d2fa3c5bca5c90aeca913.svg
paper_authors.fullname: Yahan Yu
paper_authors.type: user
paper_authors.user: YahanYu
paper_authors._id: 65b1ea38ce74220b83156830
paper_authors.name: Chenxing Li
paper_authors._id: 65b1ea38ce74220b83156831
paper_authors.name: Jiahua Dong
paper_authors._id: 65b1ea38ce74220b83156832
paper_authors.name: Dan Su
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-25T08:03:57.150Z
paper_authors._id: 64c8fa55dc59e43a6100b382
paper_authors.avatarUrl: /avatars/0a4381fbab27e2ae2438c275a7046f43.svg
paper_authors.fullname: dan su
paper_authors.type: user
paper_authors.user: sudanenator
paper_authors._id: 65b1ea38ce74220b83156833
paper_authors.name: Chenhui Chu
paper_authors._id: 65b1ea38ce74220b83156834
paper_authors.name: Dong Yu
paper_title: MM-LLMs: Recent Advances in MultiModal Large Language Models
paper_summary: In the past year, MultiModal Large Language Models (MM-LLMs) have undergone
  substantial advancements, augmenting off-the-shelf LLMs to support MM inputs or
  outputs via cost-effective training strategies. The resulting models not only
  preserve the inherent reasoning and decision-making capabilities of LLMs but
  also empower a diverse range of MM tasks. In this paper, we provide a
  comprehensive survey aimed at facilitating further research of MM-LLMs.
  Specifically, we first outline general design formulations for model
  architecture and training pipeline. Subsequently, we provide brief
  introductions of 26 existing MM-LLMs, each characterized by its specific
  formulations. Additionally, we review the performance of MM-LLMs on mainstream
  benchmarks and summarize key training recipes to enhance the potency of
  MM-LLMs. Lastly, we explore promising directions for MM-LLMs while concurrently
  maintaining a real-time tracking website for the latest developments in the
  field. We hope that this survey contributes to the ongoing advancement of the
  MM-LLMs domain.
paper_ai_keywords: MultiModal Large Language Models MM-LLMs model architecture training pipeline MM tasks mainstream benchmarks training recipes real-time tracking website