Baichuan-Omni-1.5 Technical Report
We introduce Baichuan-Omni-1.5, an omni-modal model that not only has
omni-modal understanding capabilities but also provides end-to-end audio
generation capabilities. To achieve fluent and high-quality interaction across
modalities without compromising the capabilities of any modality, we
prioritized optimizing three key aspects. First, we establish a comprehensive
data cleaning and synthesis pipeline for multimodal data, obtaining about 500B
high-quality data (text, audio, and vision). Second, an audio-tokenizer
(Baichuan-Audio-Tokenizer) has been designed to capture both semantic and
acoustic information from audio, enabling seamless integration and enhanced
compatibility with MLLM. Lastly, we designed a multi-stage training strategy
that progressively integrates multimodal alignment and multitask fine-tuning,
ensuring effective synergy across all modalities. Baichuan-Omni-1.5 leads
contemporary models (including GPT4o-mini and MiniCPM-o 2.6) in terms of
comprehensive omni-modal capabilities. Notably, it achieves results comparable
to leading models such as Qwen2-VL-72B across various multimodal medical
benchmarks.
67986c6822990ae89bb71fb9
Yadong Li
admin_assigned
2025-01-28T10:29:02.659Z
6797cc0ff386b10d1609e3ff
/avatars/3ec1020e974ed01f60a46150501171da.svg
Yadong Li
user
AdamLee1
67986c6822990ae89bb71fba
Jun Liu
67986c6822990ae89bb71fbb
Tao Zhang
67986c6822990ae89bb71fbc
Tao Zhang
67986c6822990ae89bb71fbd
Song Chen
67986c6822990ae89bb71fbe
Tianpeng Li
67986c6822990ae89bb71fbf
Zehuan Li
67986c6822990ae89bb71fc0
Lijun Liu
67986c6822990ae89bb71fc1
Lingfeng Ming
67986c6822990ae89bb71fc2
Guosheng Dong
67986c6822990ae89bb71fc3
Da Pan
67986c6822990ae89bb71fc4
Chong Li
67986c6822990ae89bb71fc5
Yuanbo Fang
67986c6822990ae89bb71fc6
Dongdong Kuang
claimed_verified
2025-01-29T17:17:34.592Z
6455ec9bd808eebdefc4ceec
https://cdn-avatars.huggingface.co/v1/production/uploads/6455ec9bd808eebdefc4ceec/AcwIUKFKjUgPpfJq07zQs.jpeg
Dongdong Kuang
user
kingsley01
67986c6822990ae89bb71fc7
Mingrui Wang
claimed_verified
2025-01-29T17:17:32.373Z
66af35946149eb45a6730a8f
/avatars/7f613f925b5d57798e03c0320661247e.svg
Mingrui Wang
user
ruillm
67986c6822990ae89bb71fc8
Chenglin Zhu
67986c6822990ae89bb71fc9
Youwei Zhang
67986c6822990ae89bb71fca
Hongyu Guo
67986c6822990ae89bb71fcb
Fengyu Zhang
67986c6822990ae89bb71fcc
Yuran Wang
claimed_verified
2025-01-28T13:52:53.610Z
65e71ef39cf349af2940b317
/avatars/fc1cd8d3510946fc947d67b16b51834b.svg
Yuran Wang
user
Ryann829
67986c6822990ae89bb71fcd
Bowen Ding
67986c6822990ae89bb71fce
Wei Song
claimed_verified
2025-03-24T08:55:52.565Z
665eccf5ffd59344a22533a8
/avatars/2ae2710753ce34a04937384bc6dddf70.svg
Wei Song
user
Songweii
67986c6822990ae89bb71fcf
Xu Li
67986c6822990ae89bb71fd0
Yuqi Huo
67986c6822990ae89bb71fd1
Zheng Liang
67986c6822990ae89bb71fd2
Shusen Zhang
67986c6822990ae89bb71fd3
Xin Wu
67986c6822990ae89bb71fd4
Shuai Zhao
67986c6822990ae89bb71fd5
Linchu Xiong
67986c6822990ae89bb71fd6
Yozhen Wu
67986c6822990ae89bb71fd7
Jiahui Ye
67986c6822990ae89bb71fd8
Wenhao Lu
67986c6822990ae89bb71fd9
Bowen Li
67986c6822990ae89bb71fda
Yan Zhang
67986c6822990ae89bb71fdb
Yaqi Zhou
67986c6822990ae89bb71fdc
Xin Chen
67986c6822990ae89bb71fdd
Lei Su
67986c6822990ae89bb71fde
Hongda Zhang
67986c6822990ae89bb71fdf
Fuzhong Chen
67986c6822990ae89bb71fe0
Xuezhen Dong
67986c6822990ae89bb71fe1
Na Nie
67986c6822990ae89bb71fe2
Zhiying Wu
67986c6822990ae89bb71fe3
Bin Xiao
67986c6822990ae89bb71fe4
Ting Li
67986c6822990ae89bb71fe5
Shunya Dang
67986c6822990ae89bb71fe6
Ping Zhang
67986c6822990ae89bb71fe7
Yijia Sun
67986c6822990ae89bb71fe8
Jincheng Wu
67986c6822990ae89bb71fe9
Jinjie Yang
67986c6822990ae89bb71fea
Xionghai Lin
67986c6822990ae89bb71feb
Zhi Ma
67986c6822990ae89bb71fec
Kegeng Wu
67986c6822990ae89bb71fed
Jia li
67986c6822990ae89bb71fee
Aiyuan Yang
67986c6822990ae89bb71fef
Hui Liu
67986c6822990ae89bb71ff0
Jianqiang Zhang
67986c6822990ae89bb71ff1
Xiaoxi Chen
67986c6822990ae89bb71ff2
Guangwei Ai
67986c6822990ae89bb71ff3
Wentao Zhang
67986c6822990ae89bb71ff4
Yicong Chen
67986c6822990ae89bb71ff5
Xiaoqin Huang
67986c6822990ae89bb71ff6
Kun Li
67986c6822990ae89bb71ff7
Wenjing Luo
67986c6822990ae89bb71ff8
Yifei Duan
67986c6822990ae89bb71ff9
Lingling Zhu
67986c6822990ae89bb71ffa
Ran Xiao
67986c6822990ae89bb71ffb
Zhe Su
67986c6822990ae89bb71ffc
Jiani Pu
67986c6822990ae89bb71ffd
Dian Wang
67986c6822990ae89bb71ffe
Xu Jia
67986c6822990ae89bb71fff
Tianyu Zhang
67986c6822990ae89bb72000
Mengyu Ai
67986c6822990ae89bb72001
Mang Wang
67986c6822990ae89bb72002
Yujing Qiao
67986c6822990ae89bb72003
Lei Zhang
67986c6822990ae89bb72004
Yanjun Shen
67986c6822990ae89bb72005
Fan Yang
67986c6822990ae89bb72006
Miao Zhen
67986c6822990ae89bb72007
Yijie Zhou
67986c6822990ae89bb72008
Mingyang Chen
67986c6822990ae89bb72009
Fei Li
67986c6822990ae89bb7200a
Chenzheng Zhu
67986c6822990ae89bb7200b
Keer Lu
67986c6822990ae89bb7200c
Yaqi Zhao
67986c6822990ae89bb7200d
Hao Liang
67986c6822990ae89bb7200e
Youquan Li
67986c6822990ae89bb7200f
Yanzhao Qin
67986c6822990ae89bb72010
Linzhuang Sun
67986c6822990ae89bb72011
Jianhua Xu
67986c6822990ae89bb72012
Haoze Sun
claimed_verified
2025-02-11T10:03:25.944Z
6436bb0dd58a5ea528c55acb
/avatars/df17b66780e14e07bbe4625f068a94ad.svg
Alvin Sun
user
AlvinSunYooo
67986c6822990ae89bb72013
Mingan Lin
claimed_verified
2025-02-12T09:17:07.915Z
6415947858a690df103af49f
/avatars/38aec23b869833bceb25b9250809b419.svg
lma
user
lin5547
67986c6822990ae89bb72014
Zenan Zhou
67986c6822990ae89bb72015
Weipeng Chen
admin_assigned
2025-01-28T13:54:48.451Z
6501587887b370a56ad2608e
/avatars/6779baaa8ed9032de55a2f78e1f52e20.svg
Wei-Peng Chen
user
whenfra
Baichuan-Omni-1.5 Technical Report
We introduce Baichuan-Omni-1.5, an omni-modal model that not only has
omni-modal understanding capabilities but also provides end-to-end audio
generation capabilities. To achieve fluent and high-quality interaction across
modalities without compromising the capabilities of any modality, we
prioritized optimizing three key aspects. First, we establish a comprehensive
data cleaning and synthesis pipeline for multimodal data, obtaining about 500B
high-quality data (text, audio, and vision). Second, an audio-tokenizer
(Baichuan-Audio-Tokenizer) has been designed to capture both semantic and
acoustic information from audio, enabling seamless integration and enhanced
compatibility with MLLM. Lastly, we designed a multi-stage training strategy
that progressively integrates multimodal alignment and multitask fine-tuning,
ensuring effective synergy across all modalities. Baichuan-Omni-1.5 leads
contemporary models (including GPT4o-mini and MiniCPM-o 2.6) in terms of
comprehensive omni-modal capabilities. Notably, it achieves results comparable
to leading models such as Qwen2-VL-72B across various multimodal medical
benchmarks.
Baichuan-Omni-1.5 is an omni-modal model with end-to-end audio generation, featuring a comprehensive data pipeline, audio-tokenizer, and multi-stage training strategy for superior performance across multimodal tasks.
omni-modal model audio-tokenizer multimodal data semantic information acoustic information MLLM multimodal alignment multitask fine-tuning omni-modal capabilities Qwen2-VL-72B multimodal medical benchmarks