title: VITA: Towards Open-Source Interactive Omni Multimodal LLM
summary: The remarkable multimodal capabilities and interactive experience of GPT-4o
  underscore their necessity in practical applications, yet open-source models
  rarely excel in both areas. In this paper, we introduce VITA, the first-ever
  open-source Multimodal Large Language Model (MLLM) adept at simultaneous
  processing and analysis of Video, Image, Text, and Audio modalities, and
  meanwhile has an advanced multimodal interactive experience. Starting from
  Mixtral 8x7B as a language foundation, we expand its Chinese vocabulary
  followed by bilingual instruction tuning. We further endow the language model
  with visual and audio capabilities through two-stage multi-task learning of
  multimodal alignment and instruction tuning. VITA demonstrates robust
  foundational capabilities of multilingual, vision, and audio understanding, as
  evidenced by its strong performance across a range of both unimodal and
  multimodal benchmarks. Beyond foundational capabilities, we have made
  considerable progress in enhancing the natural multimodal human-computer
  interaction experience. To the best of our knowledge, we are the first to
  exploit non-awakening interaction and audio interrupt in MLLM. VITA is the
  first step for the open-source community to explore the seamless integration of
  multimodal understanding and interaction. While there is still lots of work to
  be done on VITA to get close to close-source counterparts, we hope that its
  role as a pioneer can serve as a cornerstone for subsequent research. Project
  Page: https://vita-home.github.io.
paper_authors._id: 66b982ca115bf3ff568c5dd1
paper_authors.name: Chaoyou Fu
paper_authors._id: 66b982ca115bf3ff568c5dd2
paper_authors.name: Haojia Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T09:54:42.395Z
paper_authors._id: 64ffd436d522560505a94b8e
paper_authors.avatarUrl: /avatars/02d4faac40ac203cb5d635cfcb39780c.svg
paper_authors.fullname: Haojia Lin
paper_authors.type: user
paper_authors.user: linhaojia13
paper_authors._id: 66b982ca115bf3ff568c5dd3
paper_authors.name: Zuwei Long
paper_authors._id: 66b982ca115bf3ff568c5dd4
paper_authors.name: Yunhang Shen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T09:55:05.630Z
paper_authors._id: 6483143902f98c3f05aff915
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6483143902f98c3f05aff915/ZhWFFgrlRsQf4MXiInh5p.jpeg
paper_authors.fullname: 沈云航 Yunhang Shen
paper_authors.type: user
paper_authors.user: shenyunhang
paper_authors._id: 66b982ca115bf3ff568c5dd5
paper_authors.name: Meng Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-13T07:38:30.043Z
paper_authors._id: 64b77da86d1ab56458452ee0
paper_authors.avatarUrl: /avatars/0523ec058e4cdc94af543af5dece9c06.svg
paper_authors.fullname: Meng Zhao
paper_authors.type: user
paper_authors.user: Vector8421
paper_authors._id: 66b982ca115bf3ff568c5dd6
paper_authors.name: Yifan Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T09:55:44.449Z
paper_authors._id: 623d8ca4c29adf5ef6175615
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/623d8ca4c29adf5ef6175615/q7lHao7UPwU1u7YLSP56m.jpeg
paper_authors.fullname: Yi-Fan Zhang
paper_authors.type: user
paper_authors.user: yifanzhang114
paper_authors._id: 66b982ca115bf3ff568c5dd7
paper_authors.name: Xiong Wang
paper_authors._id: 66b982ca115bf3ff568c5dd8
paper_authors.name: Di Yin
paper_authors._id: 66b982ca115bf3ff568c5dd9
paper_authors.name: Long Ma
paper_authors._id: 66b982ca115bf3ff568c5dda
paper_authors.name: Xiawu Zheng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T09:56:47.955Z
paper_authors._id: 665d85e35491b1e10d0d5221
paper_authors.avatarUrl: /avatars/e46ea55b197e2bf8038871ae95f59585.svg
paper_authors.fullname: Xiawu Zheng
paper_authors.type: user
paper_authors.user: zhengxiawu
paper_authors._id: 66b982ca115bf3ff568c5ddb
paper_authors.name: Ran He
paper_authors._id: 66b982ca115bf3ff568c5ddc
paper_authors.name: Rongrong Ji
paper_authors._id: 66b982ca115bf3ff568c5ddd
paper_authors.name: Yunsheng Wu
paper_authors._id: 66b982ca115bf3ff568c5dde
paper_authors.name: Caifeng Shan
paper_authors._id: 66b982ca115bf3ff568c5ddf
paper_authors.name: Xing Sun
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-08-13T09:57:44.428Z
paper_authors._id: 647401e50da364bd0d002f2a
paper_authors.avatarUrl: /avatars/f1586f610f21ddb4f868856208c2cfab.svg
paper_authors.fullname: XING SUN
paper_authors.type: user
paper_authors.user: tedsun
paper_title: VITA: Towards Open-Source Interactive Omni Multimodal LLM
paper_summary: The remarkable multimodal capabilities and interactive experience of GPT-4o
  underscore their necessity in practical applications, yet open-source models
  rarely excel in both areas. In this paper, we introduce VITA, the first-ever
  open-source Multimodal Large Language Model (MLLM) adept at simultaneous
  processing and analysis of Video, Image, Text, and Audio modalities, and
  meanwhile has an advanced multimodal interactive experience. Starting from
  Mixtral 8x7B as a language foundation, we expand its Chinese vocabulary
  followed by bilingual instruction tuning. We further endow the language model
  with visual and audio capabilities through two-stage multi-task learning of
  multimodal alignment and instruction tuning. VITA demonstrates robust
  foundational capabilities of multilingual, vision, and audio understanding, as
  evidenced by its strong performance across a range of both unimodal and
  multimodal benchmarks. Beyond foundational capabilities, we have made
  considerable progress in enhancing the natural multimodal human-computer
  interaction experience. To the best of our knowledge, we are the first to
  exploit non-awakening interaction and audio interrupt in MLLM. VITA is the
  first step for the open-source community to explore the seamless integration of
  multimodal understanding and interaction. While there is still lots of work to
  be done on VITA to get close to close-source counterparts, we hope that its
  role as a pioneer can serve as a cornerstone for subsequent research. Project
  Page: https://vita-home.github.io.
paper_ai_summary: VITA, an open-source Multimodal Large Language Model, excels in processing Video, Image, Text, and Audio with seamless interaction, showcasing advancements in multimodal understanding and human-computer interaction.
paper_ai_keywords: Multimodal Large Language Model Mixtral bilingual instruction tuning multi-task learning multimodal alignment multilingual understanding vision understanding audio understanding non-awakening interaction audio interrupt multimodal benchmarks multimodal human-computer interaction