title: VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction
summary: Recent Multimodal Large Language Models (MLLMs) have typically focused on
  integrating visual and textual modalities, with less emphasis placed on the
  role of speech in enhancing interaction. However, speech plays a crucial role
  in multimodal dialogue systems, and implementing high-performance in both
  vision and speech tasks remains a significant challenge due to the fundamental
  modality differences. In this paper, we propose a carefully designed
  multi-stage training methodology that progressively trains LLM to understand
  both visual and speech information, ultimately enabling fluent vision and
  speech interaction. Our approach not only preserves strong vision-language
  capacity, but also enables efficient speech-to-speech dialogue capabilities
  without separate ASR and TTS modules, significantly accelerating multimodal
  end-to-end response speed. By comparing our method against state-of-the-art
  counterparts across benchmarks for image, video, and speech tasks, we
  demonstrate that our model is equipped with both strong visual and speech
  capabilities, making near real-time vision and speech interaction.
paper_authors._id: 677b5979a54b76dcaa4991f9
paper_authors.name: Chaoyou Fu
paper_authors._id: 677b5979a54b76dcaa4991fa
paper_authors.name: Haojia Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-06T08:24:54.539Z
paper_authors._id: 64ffd436d522560505a94b8e
paper_authors.avatarUrl: /avatars/02d4faac40ac203cb5d635cfcb39780c.svg
paper_authors.fullname: Haojia Lin
paper_authors.type: user
paper_authors.user: linhaojia13
paper_authors._id: 677b5979a54b76dcaa4991fb
paper_authors.name: Xiong Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-06T08:25:10.743Z
paper_authors._id: 664eaf0a98e93ef417c3cc42
paper_authors.avatarUrl: /avatars/67fb44351cac8964410e5b6549817182.svg
paper_authors.fullname: Xiong Wang
paper_authors.type: user
paper_authors.user: xiongwang
paper_authors._id: 677b5979a54b76dcaa4991fc
paper_authors.name: Yi-Fan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-06T08:25:17.351Z
paper_authors._id: 623d8ca4c29adf5ef6175615
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/623d8ca4c29adf5ef6175615/q7lHao7UPwU1u7YLSP56m.jpeg
paper_authors.fullname: Yi-Fan Zhang
paper_authors.type: user
paper_authors.user: yifanzhang114
paper_authors._id: 677b5979a54b76dcaa4991fd
paper_authors.name: Yunhang Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-06T08:25:29.653Z
paper_authors._id: 6483143902f98c3f05aff915
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6483143902f98c3f05aff915/ZhWFFgrlRsQf4MXiInh5p.jpeg
paper_authors.fullname: 沈云航 Yunhang Shen
paper_authors.type: user
paper_authors.user: shenyunhang
paper_authors._id: 677b5979a54b76dcaa4991fe
paper_authors.name: Xiaoyu Liu
paper_authors._id: 677b5979a54b76dcaa4991ff
paper_authors.name: Yangze Li
paper_authors._id: 677b5979a54b76dcaa499200
paper_authors.name: Zuwei Long
paper_authors._id: 677b5979a54b76dcaa499201
paper_authors.name: Heting Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-06T08:26:18.115Z
paper_authors._id: 65ff5dcf82708115869da69a
paper_authors.avatarUrl: /avatars/10edebbc559e9fb8b0e377c82eba66d4.svg
paper_authors.fullname: Heting Gao
paper_authors.type: user
paper_authors.user: hertin
paper_authors._id: 677b5979a54b76dcaa499202
paper_authors.name: Ke Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-09T10:07:25.742Z
paper_authors._id: 63280915eeee4dd858083092
paper_authors.avatarUrl: /avatars/78347af4af42527d53e88d9969c5c934.svg
paper_authors.fullname: Ke Li
paper_authors.type: user
paper_authors.user: tristanli
paper_authors._id: 677b5979a54b76dcaa499203
paper_authors.name: Xiawu Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-06T08:26:25.079Z
paper_authors._id: 665d85e35491b1e10d0d5221
paper_authors.avatarUrl: /avatars/e46ea55b197e2bf8038871ae95f59585.svg
paper_authors.fullname: Xiawu Zheng
paper_authors.type: user
paper_authors.user: zhengxiawu
paper_authors._id: 677b5979a54b76dcaa499204
paper_authors.name: Rongrong Ji
paper_authors._id: 677b5979a54b76dcaa499205
paper_authors.name: Xing Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-06T08:30:21.180Z
paper_authors._id: 647401e50da364bd0d002f2a
paper_authors.avatarUrl: /avatars/f1586f610f21ddb4f868856208c2cfab.svg
paper_authors.fullname: XING SUN
paper_authors.type: user
paper_authors.user: tedsun
paper_authors._id: 677b5979a54b76dcaa499206
paper_authors.name: Caifeng Shan
paper_authors._id: 677b5979a54b76dcaa499207
paper_authors.name: Ran He
paper_title: VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction
paper_summary: Recent Multimodal Large Language Models (MLLMs) have typically focused on
  integrating visual and textual modalities, with less emphasis placed on the
  role of speech in enhancing interaction. However, speech plays a crucial role
  in multimodal dialogue systems, and implementing high-performance in both
  vision and speech tasks remains a significant challenge due to the fundamental
  modality differences. In this paper, we propose a carefully designed
  multi-stage training methodology that progressively trains LLM to understand
  both visual and speech information, ultimately enabling fluent vision and
  speech interaction. Our approach not only preserves strong vision-language
  capacity, but also enables efficient speech-to-speech dialogue capabilities
  without separate ASR and TTS modules, significantly accelerating multimodal
  end-to-end response speed. By comparing our method against state-of-the-art
  counterparts across benchmarks for image, video, and speech tasks, we
  demonstrate that our model is equipped with both strong visual and speech
  capabilities, making near real-time vision and speech interaction.
paper_ai_summary: A multi-stage training method enhances multimodal large language models with both visual and speech capabilities, enabling real-time interaction without separate ASR and TTS modules.
paper_ai_keywords: Multimodal Large Language Models multi-stage training speech-to-speech dialogue vision-language capacity ASR TTS