title: Baichuan-Omni-1.5 Technical Report
summary: We introduce Baichuan-Omni-1.5, an omni-modal model that not only has
  omni-modal understanding capabilities but also provides end-to-end audio
  generation capabilities. To achieve fluent and high-quality interaction across
  modalities without compromising the capabilities of any modality, we
  prioritized optimizing three key aspects. First, we establish a comprehensive
  data cleaning and synthesis pipeline for multimodal data, obtaining about 500B
  high-quality data (text, audio, and vision). Second, an audio-tokenizer
  (Baichuan-Audio-Tokenizer) has been designed to capture both semantic and
  acoustic information from audio, enabling seamless integration and enhanced
  compatibility with MLLM. Lastly, we designed a multi-stage training strategy
  that progressively integrates multimodal alignment and multitask fine-tuning,
  ensuring effective synergy across all modalities. Baichuan-Omni-1.5 leads
  contemporary models (including GPT4o-mini and MiniCPM-o 2.6) in terms of
  comprehensive omni-modal capabilities. Notably, it achieves results comparable
  to leading models such as Qwen2-VL-72B across various multimodal medical
  benchmarks.
paper_authors._id: 67986c6822990ae89bb71fb9
paper_authors.name: Yadong Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-28T10:29:02.659Z
paper_authors._id: 6797cc0ff386b10d1609e3ff
paper_authors.avatarUrl: /avatars/3ec1020e974ed01f60a46150501171da.svg
paper_authors.fullname: Yadong Li
paper_authors.type: user
paper_authors.user: AdamLee1
paper_authors._id: 67986c6822990ae89bb71fba
paper_authors.name: Jun Liu
paper_authors._id: 67986c6822990ae89bb71fbb
paper_authors.name: Tao Zhang
paper_authors._id: 67986c6822990ae89bb71fbc
paper_authors.name: Tao Zhang
paper_authors._id: 67986c6822990ae89bb71fbd
paper_authors.name: Song Chen
paper_authors._id: 67986c6822990ae89bb71fbe
paper_authors.name: Tianpeng Li
paper_authors._id: 67986c6822990ae89bb71fbf
paper_authors.name: Zehuan Li
paper_authors._id: 67986c6822990ae89bb71fc0
paper_authors.name: Lijun Liu
paper_authors._id: 67986c6822990ae89bb71fc1
paper_authors.name: Lingfeng Ming
paper_authors._id: 67986c6822990ae89bb71fc2
paper_authors.name: Guosheng Dong
paper_authors._id: 67986c6822990ae89bb71fc3
paper_authors.name: Da Pan
paper_authors._id: 67986c6822990ae89bb71fc4
paper_authors.name: Chong Li
paper_authors._id: 67986c6822990ae89bb71fc5
paper_authors.name: Yuanbo Fang
paper_authors._id: 67986c6822990ae89bb71fc6
paper_authors.name: Dongdong Kuang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-29T17:17:34.592Z
paper_authors._id: 6455ec9bd808eebdefc4ceec
paper_authors.avatarUrl: /avatars/86c989f7abf6558573409e9e42a721f9.svg
paper_authors.fullname: Dongdong Kuang
paper_authors.type: user
paper_authors.user: kingsley01
paper_authors._id: 67986c6822990ae89bb71fc7
paper_authors.name: Mingrui Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-29T17:17:32.373Z
paper_authors._id: 66af35946149eb45a6730a8f
paper_authors.avatarUrl: /avatars/7f613f925b5d57798e03c0320661247e.svg
paper_authors.fullname: Mingrui Wang
paper_authors.type: user
paper_authors.user: ruillm
paper_authors._id: 67986c6822990ae89bb71fc8
paper_authors.name: Chenglin Zhu
paper_authors._id: 67986c6822990ae89bb71fc9
paper_authors.name: Youwei Zhang
paper_authors._id: 67986c6822990ae89bb71fca
paper_authors.name: Hongyu Guo
paper_authors._id: 67986c6822990ae89bb71fcb
paper_authors.name: Fengyu Zhang
paper_authors._id: 67986c6822990ae89bb71fcc
paper_authors.name: Yuran Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-28T13:52:53.610Z
paper_authors._id: 65e71ef39cf349af2940b317
paper_authors.avatarUrl: /avatars/fc1cd8d3510946fc947d67b16b51834b.svg
paper_authors.fullname: Yuran Wang
paper_authors.type: user
paper_authors.user: Ryann829
paper_authors._id: 67986c6822990ae89bb71fcd
paper_authors.name: Bowen Ding
paper_authors._id: 67986c6822990ae89bb71fce
paper_authors.name: Wei Song
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-24T08:55:52.565Z
paper_authors._id: 665eccf5ffd59344a22533a8
paper_authors.avatarUrl: /avatars/2ae2710753ce34a04937384bc6dddf70.svg
paper_authors.fullname: Wei Song
paper_authors.type: user
paper_authors.user: Songweii
paper_authors._id: 67986c6822990ae89bb71fcf
paper_authors.name: Xu Li
paper_authors._id: 67986c6822990ae89bb71fd0
paper_authors.name: Yuqi Huo
paper_authors._id: 67986c6822990ae89bb71fd1
paper_authors.name: Zheng Liang
paper_authors._id: 67986c6822990ae89bb71fd2
paper_authors.name: Shusen Zhang
paper_authors._id: 67986c6822990ae89bb71fd3
paper_authors.name: Xin Wu
paper_authors._id: 67986c6822990ae89bb71fd4
paper_authors.name: Shuai Zhao
paper_authors._id: 67986c6822990ae89bb71fd5
paper_authors.name: Linchu Xiong
paper_authors._id: 67986c6822990ae89bb71fd6
paper_authors.name: Yozhen Wu
paper_authors._id: 67986c6822990ae89bb71fd7
paper_authors.name: Jiahui Ye
paper_authors._id: 67986c6822990ae89bb71fd8
paper_authors.name: Wenhao Lu
paper_authors._id: 67986c6822990ae89bb71fd9
paper_authors.name: Bowen Li
paper_authors._id: 67986c6822990ae89bb71fda
paper_authors.name: Yan Zhang
paper_authors._id: 67986c6822990ae89bb71fdb
paper_authors.name: Yaqi Zhou
paper_authors._id: 67986c6822990ae89bb71fdc
paper_authors.name: Xin Chen
paper_authors._id: 67986c6822990ae89bb71fdd
paper_authors.name: Lei Su
paper_authors._id: 67986c6822990ae89bb71fde
paper_authors.name: Hongda Zhang
paper_authors._id: 67986c6822990ae89bb71fdf
paper_authors.name: Fuzhong Chen
paper_authors._id: 67986c6822990ae89bb71fe0
paper_authors.name: Xuezhen Dong
paper_authors._id: 67986c6822990ae89bb71fe1
paper_authors.name: Na Nie
paper_authors._id: 67986c6822990ae89bb71fe2
paper_authors.name: Zhiying Wu
paper_authors._id: 67986c6822990ae89bb71fe3
paper_authors.name: Bin Xiao
paper_authors._id: 67986c6822990ae89bb71fe4
paper_authors.name: Ting Li
paper_authors._id: 67986c6822990ae89bb71fe5
paper_authors.name: Shunya Dang
paper_authors._id: 67986c6822990ae89bb71fe6
paper_authors.name: Ping Zhang
paper_authors._id: 67986c6822990ae89bb71fe7
paper_authors.name: Yijia Sun
paper_authors._id: 67986c6822990ae89bb71fe8
paper_authors.name: Jincheng Wu
paper_authors._id: 67986c6822990ae89bb71fe9
paper_authors.name: Jinjie Yang
paper_authors._id: 67986c6822990ae89bb71fea
paper_authors.name: Xionghai Lin
paper_authors._id: 67986c6822990ae89bb71feb
paper_authors.name: Zhi Ma
paper_authors._id: 67986c6822990ae89bb71fec
paper_authors.name: Kegeng Wu
paper_authors._id: 67986c6822990ae89bb71fed
paper_authors.name: Jia li
paper_authors._id: 67986c6822990ae89bb71fee
paper_authors.name: Aiyuan Yang
paper_authors._id: 67986c6822990ae89bb71fef
paper_authors.name: Hui Liu
paper_authors._id: 67986c6822990ae89bb71ff0
paper_authors.name: Jianqiang Zhang
paper_authors._id: 67986c6822990ae89bb71ff1
paper_authors.name: Xiaoxi Chen
paper_authors._id: 67986c6822990ae89bb71ff2
paper_authors.name: Guangwei Ai
paper_authors._id: 67986c6822990ae89bb71ff3
paper_authors.name: Wentao Zhang
paper_authors._id: 67986c6822990ae89bb71ff4
paper_authors.name: Yicong Chen
paper_authors._id: 67986c6822990ae89bb71ff5
paper_authors.name: Xiaoqin Huang
paper_authors._id: 67986c6822990ae89bb71ff6
paper_authors.name: Kun Li
paper_authors._id: 67986c6822990ae89bb71ff7
paper_authors.name: Wenjing Luo
paper_authors._id: 67986c6822990ae89bb71ff8
paper_authors.name: Yifei Duan
paper_authors._id: 67986c6822990ae89bb71ff9
paper_authors.name: Lingling Zhu
paper_authors._id: 67986c6822990ae89bb71ffa
paper_authors.name: Ran Xiao
paper_authors._id: 67986c6822990ae89bb71ffb
paper_authors.name: Zhe Su
paper_authors._id: 67986c6822990ae89bb71ffc
paper_authors.name: Jiani Pu
paper_authors._id: 67986c6822990ae89bb71ffd
paper_authors.name: Dian Wang
paper_authors._id: 67986c6822990ae89bb71ffe
paper_authors.name: Xu Jia
paper_authors._id: 67986c6822990ae89bb71fff
paper_authors.name: Tianyu Zhang
paper_authors._id: 67986c6822990ae89bb72000
paper_authors.name: Mengyu Ai
paper_authors._id: 67986c6822990ae89bb72001
paper_authors.name: Mang Wang
paper_authors._id: 67986c6822990ae89bb72002
paper_authors.name: Yujing Qiao
paper_authors._id: 67986c6822990ae89bb72003
paper_authors.name: Lei Zhang
paper_authors._id: 67986c6822990ae89bb72004
paper_authors.name: Yanjun Shen
paper_authors._id: 67986c6822990ae89bb72005
paper_authors.name: Fan Yang
paper_authors._id: 67986c6822990ae89bb72006
paper_authors.name: Miao Zhen
paper_authors._id: 67986c6822990ae89bb72007
paper_authors.name: Yijie Zhou
paper_authors._id: 67986c6822990ae89bb72008
paper_authors.name: Mingyang Chen
paper_authors._id: 67986c6822990ae89bb72009
paper_authors.name: Fei Li
paper_authors._id: 67986c6822990ae89bb7200a
paper_authors.name: Chenzheng Zhu
paper_authors._id: 67986c6822990ae89bb7200b
paper_authors.name: Keer Lu
paper_authors._id: 67986c6822990ae89bb7200c
paper_authors.name: Yaqi Zhao
paper_authors._id: 67986c6822990ae89bb7200d
paper_authors.name: Hao Liang
paper_authors._id: 67986c6822990ae89bb7200e
paper_authors.name: Youquan Li
paper_authors._id: 67986c6822990ae89bb7200f
paper_authors.name: Yanzhao Qin
paper_authors._id: 67986c6822990ae89bb72010
paper_authors.name: Linzhuang Sun
paper_authors._id: 67986c6822990ae89bb72011
paper_authors.name: Jianhua Xu
paper_authors._id: 67986c6822990ae89bb72012
paper_authors.name: Haoze Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-11T10:03:25.944Z
paper_authors._id: 6436bb0dd58a5ea528c55acb
paper_authors.avatarUrl: /avatars/df17b66780e14e07bbe4625f068a94ad.svg
paper_authors.fullname: Alvin Sun
paper_authors.type: user
paper_authors.user: AlvinSunYooo
paper_authors._id: 67986c6822990ae89bb72013
paper_authors.name: Mingan Lin
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-12T09:17:07.915Z
paper_authors._id: 6415947858a690df103af49f
paper_authors.avatarUrl: /avatars/38aec23b869833bceb25b9250809b419.svg
paper_authors.fullname: lma
paper_authors.type: user
paper_authors.user: lin5547
paper_authors._id: 67986c6822990ae89bb72014
paper_authors.name: Zenan Zhou
paper_authors._id: 67986c6822990ae89bb72015
paper_authors.name: Weipeng Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-28T13:54:48.451Z
paper_authors._id: 6501587887b370a56ad2608e
paper_authors.avatarUrl: /avatars/6779baaa8ed9032de55a2f78e1f52e20.svg
paper_authors.fullname: Wei-Peng Chen
paper_authors.type: user
paper_authors.user: whenfra
paper_title: Baichuan-Omni-1.5 Technical Report
paper_summary: We introduce Baichuan-Omni-1.5, an omni-modal model that not only has
  omni-modal understanding capabilities but also provides end-to-end audio
  generation capabilities. To achieve fluent and high-quality interaction across
  modalities without compromising the capabilities of any modality, we
  prioritized optimizing three key aspects. First, we establish a comprehensive
  data cleaning and synthesis pipeline for multimodal data, obtaining about 500B
  high-quality data (text, audio, and vision). Second, an audio-tokenizer
  (Baichuan-Audio-Tokenizer) has been designed to capture both semantic and
  acoustic information from audio, enabling seamless integration and enhanced
  compatibility with MLLM. Lastly, we designed a multi-stage training strategy
  that progressively integrates multimodal alignment and multitask fine-tuning,
  ensuring effective synergy across all modalities. Baichuan-Omni-1.5 leads
  contemporary models (including GPT4o-mini and MiniCPM-o 2.6) in terms of
  comprehensive omni-modal capabilities. Notably, it achieves results comparable
  to leading models such as Qwen2-VL-72B across various multimodal medical
  benchmarks.
paper_ai_summary: Baichuan-Omni-1.5 is an omni-modal model with end-to-end audio generation, featuring a comprehensive data pipeline, audio-tokenizer, and multi-stage training strategy for superior performance across multimodal tasks.
paper_ai_keywords: omni-modal model audio-tokenizer multimodal data semantic information acoustic information MLLM multimodal alignment multitask fine-tuning omni-modal capabilities Qwen2-VL-72B multimodal medical benchmarks