title: Baichuan-Omni Technical Report
summary: The salient multimodal capabilities and interactive experience of GPT-4o
  highlight its critical role in practical applications, yet it lacks a
  high-performing open-source counterpart. In this paper, we introduce
  Baichuan-Omni, the first open-source 7B Multimodal Large Language Model (MLLM)
  adept at concurrently processing and analyzing modalities of image, video,
  audio, and text, while delivering an advanced multimodal interactive experience
  and strong performance. We propose an effective multimodal training schema
  starting with 7B model and proceeding through two stages of multimodal
  alignment and multitask fine-tuning across audio, image, video, and text modal.
  This approach equips the language model with the ability to handle visual and
  audio data effectively. Demonstrating strong performance across various
  omni-modal and multimodal benchmarks, we aim for this contribution to serve as
  a competitive baseline for the open-source community in advancing multimodal
  understanding and real-time interaction.
paper_authors._id: 670c7526d01f57eeed9ecb02
paper_authors.name: Yadong Li
paper_authors._id: 670c7526d01f57eeed9ecb03
paper_authors.name: Haoze Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-23T15:05:39.764Z
paper_authors._id: 6436bb0dd58a5ea528c55acb
paper_authors.avatarUrl: /avatars/df17b66780e14e07bbe4625f068a94ad.svg
paper_authors.fullname: Alvin Sun
paper_authors.type: user
paper_authors.user: AlvinSunYooo
paper_authors._id: 670c7526d01f57eeed9ecb04
paper_authors.name: Mingan Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T07:48:30.747Z
paper_authors._id: 6415947858a690df103af49f
paper_authors.avatarUrl: /avatars/38aec23b869833bceb25b9250809b419.svg
paper_authors.fullname: lma
paper_authors.type: user
paper_authors.user: lin5547
paper_authors._id: 670c7526d01f57eeed9ecb05
paper_authors.name: Tianpeng Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-14T07:58:36.547Z
paper_authors._id: 64be4c97ef8c0e42bf448469
paper_authors.avatarUrl: /avatars/a2a45b5f17d3e27c59547312bcf7bb52.svg
paper_authors.fullname: Tianpeng Li
paper_authors.type: user
paper_authors.user: TJU-Tianpengli
paper_authors._id: 670c7526d01f57eeed9ecb06
paper_authors.name: Guosheng Dong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-14T07:59:42.904Z
paper_authors._id: 668d2ff4271bd45dab6a8e71
paper_authors.avatarUrl: /avatars/4e198c35a9f09bbc6551c34148aaf560.svg
paper_authors.fullname: guosheng dong
paper_authors.type: user
paper_authors.user: dongguosheng
paper_authors._id: 670c7526d01f57eeed9ecb07
paper_authors.name: Tao Zhang
paper_authors._id: 670c7526d01f57eeed9ecb08
paper_authors.name: Bowen Ding
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-16T21:59:49.277Z
paper_authors._id: 64127b9fac08ffb707937231
paper_authors.avatarUrl: /avatars/33d406b8d1f319af5a4e3c2dc59ea7f2.svg
paper_authors.fullname: Ding Bowen
paper_authors.type: user
paper_authors.user: Daniel21Ding
paper_authors._id: 670c7526d01f57eeed9ecb09
paper_authors.name: Wei Song
paper_authors._id: 670c7526d01f57eeed9ecb0a
paper_authors.name: Zhenglin Cheng
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T07:48:32.690Z
paper_authors._id: 65028e8389707f182386588c
paper_authors.avatarUrl: /avatars/86a748a3264e6e0f4ee5eaf8f7032ecb.svg
paper_authors.fullname: Zhenglin Cheng
paper_authors.type: user
paper_authors.user: kenshinn
paper_authors._id: 670c7526d01f57eeed9ecb0b
paper_authors.name: Yuqi Huo
paper_authors._id: 670c7526d01f57eeed9ecb0c
paper_authors.name: Song Chen
paper_authors._id: 670c7526d01f57eeed9ecb0d
paper_authors.name: Xu Li
paper_authors._id: 670c7526d01f57eeed9ecb0e
paper_authors.name: Da Pan
paper_authors._id: 670c7526d01f57eeed9ecb0f
paper_authors.name: Shusen Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-14T07:48:23.098Z
paper_authors._id: 64d9c9bd95cf13a381f6cc1a
paper_authors.avatarUrl: /avatars/ce138298df376511044c69182b6419b9.svg
paper_authors.fullname: zss
paper_authors.type: user
paper_authors.user: zss9606
paper_authors._id: 670c7526d01f57eeed9ecb10
paper_authors.name: Xin Wu
paper_authors._id: 670c7526d01f57eeed9ecb11
paper_authors.name: Zheng Liang
paper_authors._id: 670c7526d01f57eeed9ecb12
paper_authors.name: Jun Liu
paper_authors._id: 670c7526d01f57eeed9ecb13
paper_authors.name: Tao Zhang
paper_authors._id: 670c7526d01f57eeed9ecb14
paper_authors.name: Keer Lu
paper_authors._id: 670c7526d01f57eeed9ecb15
paper_authors.name: Yaqi Zhao
paper_authors._id: 670c7526d01f57eeed9ecb16
paper_authors.name: Yanjun Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-14T08:11:33.707Z
paper_authors._id: 651e0a9f60e6a3ef9a1fd38b
paper_authors.avatarUrl: /avatars/50b2b5bf380fa1f754fc8e53a4d08f19.svg
paper_authors.fullname: Yanjun Sheng
paper_authors.type: user
paper_authors.user: Astroshiba
paper_authors._id: 670c7526d01f57eeed9ecb17
paper_authors.name: Fan Yang
paper_authors._id: 670c7526d01f57eeed9ecb18
paper_authors.name: Kaicheng Yu
paper_authors._id: 670c7526d01f57eeed9ecb19
paper_authors.name: Tao Lin
paper_authors._id: 670c7526d01f57eeed9ecb1a
paper_authors.name: Jianhua Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-14T08:01:31.350Z
paper_authors._id: 66042414ded78d7454b9746e
paper_authors.avatarUrl: /avatars/828feab7015b527b5ac5c9238fa3c7e0.svg
paper_authors.fullname: jianhua Xu
paper_authors.type: user
paper_authors.user: ArthurXu2020
paper_authors._id: 670c7526d01f57eeed9ecb1b
paper_authors.name: Zenan Zhou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-14T08:01:23.922Z
paper_authors._id: 668d4e50ed63008dfaa78304
paper_authors.avatarUrl: /avatars/80854a3c6b4b7c70cd46694d4cf7296a.svg
paper_authors.fullname: Zenan Zhou
paper_authors.type: user
paper_authors.user: Zenan11
paper_authors._id: 670c7526d01f57eeed9ecb1c
paper_authors.name: Weipeng Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-14T08:01:15.261Z
paper_authors._id: 6501587887b370a56ad2608e
paper_authors.avatarUrl: /avatars/6779baaa8ed9032de55a2f78e1f52e20.svg
paper_authors.fullname: Wei-Peng Chen
paper_authors.type: user
paper_authors.user: whenfra
paper_title: Baichuan-Omni Technical Report
paper_summary: The salient multimodal capabilities and interactive experience of GPT-4o
  highlight its critical role in practical applications, yet it lacks a
  high-performing open-source counterpart. In this paper, we introduce
  Baichuan-Omni, the first open-source 7B Multimodal Large Language Model (MLLM)
  adept at concurrently processing and analyzing modalities of image, video,
  audio, and text, while delivering an advanced multimodal interactive experience
  and strong performance. We propose an effective multimodal training schema
  starting with 7B model and proceeding through two stages of multimodal
  alignment and multitask fine-tuning across audio, image, video, and text modal.
  This approach equips the language model with the ability to handle visual and
  audio data effectively. Demonstrating strong performance across various
  omni-modal and multimodal benchmarks, we aim for this contribution to serve as
  a competitive baseline for the open-source community in advancing multimodal
  understanding and real-time interaction.
paper_ai_keywords: multimodal Large Language Model (LLM) multimodal training schema multimodal alignment multitask fine-tuning multimodal interactive experience visual and audio data handling multimodal benchmarks competitive baseline