Baichuan-Omni Technical Report
The salient multimodal capabilities and interactive experience of GPT-4o
highlight its critical role in practical applications, yet it lacks a
high-performing open-source counterpart. In this paper, we introduce
Baichuan-Omni, the first open-source 7B Multimodal Large Language Model (MLLM)
adept at concurrently processing and analyzing modalities of image, video,
audio, and text, while delivering an advanced multimodal interactive experience
and strong performance. We propose an effective multimodal training schema
starting with 7B model and proceeding through two stages of multimodal
alignment and multitask fine-tuning across audio, image, video, and text modal.
This approach equips the language model with the ability to handle visual and
audio data effectively. Demonstrating strong performance across various
omni-modal and multimodal benchmarks, we aim for this contribution to serve as
a competitive baseline for the open-source community in advancing multimodal
understanding and real-time interaction.
670c7526d01f57eeed9ecb02
Yadong Li
670c7526d01f57eeed9ecb03
Haoze Sun
claimed_verified
2025-01-23T15:05:39.764Z
6436bb0dd58a5ea528c55acb
/avatars/df17b66780e14e07bbe4625f068a94ad.svg
Alvin Sun
user
AlvinSunYooo
670c7526d01f57eeed9ecb04
Mingan Lin
claimed_verified
2024-10-14T07:48:30.747Z
6415947858a690df103af49f
/avatars/38aec23b869833bceb25b9250809b419.svg
lma
user
lin5547
670c7526d01f57eeed9ecb05
Tianpeng Li
admin_assigned
2024-10-14T07:58:36.547Z
64be4c97ef8c0e42bf448469
/avatars/a2a45b5f17d3e27c59547312bcf7bb52.svg
Tianpeng Li
user
TJU-Tianpengli
670c7526d01f57eeed9ecb06
Guosheng Dong
admin_assigned
2024-10-14T07:59:42.904Z
668d2ff4271bd45dab6a8e71
/avatars/4e198c35a9f09bbc6551c34148aaf560.svg
guosheng dong
user
dongguosheng
670c7526d01f57eeed9ecb07
Tao Zhang
670c7526d01f57eeed9ecb08
Bowen Ding
admin_assigned
2024-10-16T21:59:49.277Z
64127b9fac08ffb707937231
/avatars/33d406b8d1f319af5a4e3c2dc59ea7f2.svg
Ding Bowen
user
Daniel21Ding
670c7526d01f57eeed9ecb09
Wei Song
claimed_verified
2025-03-24T08:56:17.960Z
665eccf5ffd59344a22533a8
/avatars/2ae2710753ce34a04937384bc6dddf70.svg
Wei Song
user
Songweii
670c7526d01f57eeed9ecb0a
Zhenglin Cheng
claimed_verified
2024-10-14T07:48:32.690Z
65028e8389707f182386588c
/avatars/86a748a3264e6e0f4ee5eaf8f7032ecb.svg
Zhenglin Cheng
user
kenshinn
670c7526d01f57eeed9ecb0b
Yuqi Huo
670c7526d01f57eeed9ecb0c
Song Chen
670c7526d01f57eeed9ecb0d
Xu Li
670c7526d01f57eeed9ecb0e
Da Pan
670c7526d01f57eeed9ecb0f
Shusen Zhang
claimed_verified
2024-10-14T07:48:23.098Z
64d9c9bd95cf13a381f6cc1a
/avatars/ce138298df376511044c69182b6419b9.svg
zss
user
zss9606
670c7526d01f57eeed9ecb10
Xin Wu
670c7526d01f57eeed9ecb11
Zheng Liang
670c7526d01f57eeed9ecb12
Jun Liu
670c7526d01f57eeed9ecb13
Tao Zhang
670c7526d01f57eeed9ecb14
Keer Lu
670c7526d01f57eeed9ecb15
Yaqi Zhao
670c7526d01f57eeed9ecb16
Yanjun Shen
admin_assigned
2024-10-14T08:11:33.707Z
651e0a9f60e6a3ef9a1fd38b
/avatars/50b2b5bf380fa1f754fc8e53a4d08f19.svg
Yanjun Sheng
user
Astroshiba
670c7526d01f57eeed9ecb17
Fan Yang
670c7526d01f57eeed9ecb18
Kaicheng Yu
670c7526d01f57eeed9ecb19
Tao Lin
claimed_verified
2025-05-13T08:58:01.132Z
65296b8a6cdea40585bd81e2
/avatars/824452e8b8fd056eaf7549c46393c47b.svg
Tao LIN
user
tlin-taolin
670c7526d01f57eeed9ecb1a
Jianhua Xu
admin_assigned
2024-10-14T08:01:31.350Z
66042414ded78d7454b9746e
/avatars/828feab7015b527b5ac5c9238fa3c7e0.svg
jianhua Xu
user
ArthurXu2020
670c7526d01f57eeed9ecb1b
Zenan Zhou
admin_assigned
2024-10-14T08:01:23.922Z
668d4e50ed63008dfaa78304
/avatars/80854a3c6b4b7c70cd46694d4cf7296a.svg
Zenan Zhou
user
Zenan11
670c7526d01f57eeed9ecb1c
Weipeng Chen
admin_assigned
2024-10-14T08:01:15.261Z
6501587887b370a56ad2608e
/avatars/6779baaa8ed9032de55a2f78e1f52e20.svg
Wei-Peng Chen
user
whenfra
Baichuan-Omni Technical Report
The salient multimodal capabilities and interactive experience of GPT-4o
highlight its critical role in practical applications, yet it lacks a
high-performing open-source counterpart. In this paper, we introduce
Baichuan-Omni, the first open-source 7B Multimodal Large Language Model (MLLM)
adept at concurrently processing and analyzing modalities of image, video,
audio, and text, while delivering an advanced multimodal interactive experience
and strong performance. We propose an effective multimodal training schema
starting with 7B model and proceeding through two stages of multimodal
alignment and multitask fine-tuning across audio, image, video, and text modal.
This approach equips the language model with the ability to handle visual and
audio data effectively. Demonstrating strong performance across various
omni-modal and multimodal benchmarks, we aim for this contribution to serve as
a competitive baseline for the open-source community in advancing multimodal
understanding and real-time interaction.
Baichuan-Omni, a 7B open-source Multimodal Large Language Model, excels in processing image, video, audio, and text, showcasing competitive performance across multimodal benchmarks.
Multimodal Large Language Model multimodal training schema multimodal alignment multitask fine-tuning advanced multimodal interactive experience