title: Lyra: An Efficient and Speech-Centric Framework for Omni-Cognition
summary: As Multi-modal Large Language Models (MLLMs) evolve, expanding beyond
  single-domain capabilities is essential to meet the demands for more versatile
  and efficient AI. However, previous omni-models have insufficiently explored
  speech, neglecting its integration with multi-modality. We introduce Lyra, an
  efficient MLLM that enhances multimodal abilities, including advanced
  long-speech comprehension, sound understanding, cross-modality efficiency, and
  seamless speech interaction. To achieve efficiency and speech-centric
  capabilities, Lyra employs three strategies: (1) leveraging existing
  open-source large models and a proposed multi-modality LoRA to reduce training
  costs and data requirements; (2) using a latent multi-modality regularizer and
  extractor to strengthen the relationship between speech and other modalities,
  thereby enhancing model performance; and (3) constructing a high-quality,
  extensive dataset that includes 1.5M multi-modal (language, vision, audio) data
  samples and 12K long speech samples, enabling Lyra to handle complex long
  speech inputs and achieve more robust omni-cognition. Compared to other
  omni-methods, Lyra achieves state-of-the-art performance on various
  vision-language, vision-speech, and speech-language benchmarks, while also
  using fewer computational resources and less training data.
paper_authors._id: 675bcad776d0e555d631381c
paper_authors.name: Zhisheng Zhong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:52:26.208Z
paper_authors._id: 65d882d30f35ed3f52d3ae2c
paper_authors.avatarUrl: /avatars/22cda67c3fcd7150320ec3551eda90f5.svg
paper_authors.fullname: Zhisheng Zhong
paper_authors.type: user
paper_authors.user: zszhong
paper_authors._id: 675bcad776d0e555d631381d
paper_authors.name: Chengyao Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-13T15:21:46.575Z
paper_authors._id: 6423e35b30b0e4ab36dd1b16
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6423e35b30b0e4ab36dd1b16/pea6LVDS9PQAxvQt9GUiZ.jpeg
paper_authors.fullname: Wang Chengyao
paper_authors.type: user
paper_authors.user: wcy1122
paper_authors._id: 675bcad776d0e555d631381e
paper_authors.name: Yuqi Liu
paper_authors._id: 675bcad776d0e555d631381f
paper_authors.name: Senqiao Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:51:23.068Z
paper_authors._id: 6527b7280ae663e384eb8499
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6527b7280ae663e384eb8499/73yF3eu2cUx7jVZrhXnXx.jpeg
paper_authors.fullname: Senqiao Yang
paper_authors.type: user
paper_authors.user: Senqiao
paper_authors._id: 675bcad776d0e555d6313820
paper_authors.name: Longxiang Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:50:10.607Z
paper_authors._id: 6492a0d8d4ae24c933ace44d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/DXIky2sdPwmiCOR9p-JBQ.png
paper_authors.fullname: Longxiang Tang
paper_authors.type: user
paper_authors.user: lloong
paper_authors._id: 675bcad776d0e555d6313821
paper_authors.name: Yuechen Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:50:22.552Z
paper_authors._id: 6418554a0956be7233a1023e
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6418554a0956be7233a1023e/9EKN0GoOpcDbvBDmAQEJf.png
paper_authors.fullname: zhang yuechen
paper_authors.type: user
paper_authors.user: julianjuaner
paper_authors._id: 675bcad776d0e555d6313822
paper_authors.name: Jingyao Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:50:51.188Z
paper_authors._id: 647ff779d037515be1169e35
paper_authors.avatarUrl: /avatars/741096302cbdeb3670a8f31cf2d31d5d.svg
paper_authors.fullname: Jingyao Li
paper_authors.type: user
paper_authors.user: JingyaoLi
paper_authors._id: 675bcad776d0e555d6313823
paper_authors.name: Tianyuan Qu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:45:31.921Z
paper_authors._id: 66e79b3c1c79fc2e51dc1d60
paper_authors.avatarUrl: /avatars/8706336e9e7a417505c9bb32583a662f.svg
paper_authors.fullname: QU Tianyuan
paper_authors.type: user
paper_authors.user: TainU
paper_authors._id: 675bcad776d0e555d6313824
paper_authors.name: Yanwei Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:45:47.023Z
paper_authors._id: 643ff78cdc984afcbbbc3b1a
paper_authors.avatarUrl: /avatars/eec5198ce88aaf8156840bec0d190a7f.svg
paper_authors.fullname: Yanwei Li
paper_authors.type: user
paper_authors.user: YanweiLi
paper_authors._id: 675bcad776d0e555d6313825
paper_authors.name: Yukang Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:50:59.539Z
paper_authors._id: 65b31d77f32c79fea79ba4d1
paper_authors.avatarUrl: /avatars/8c51c1ff337ac372c7caafa4fa5019dc.svg
paper_authors.fullname: Yukang Chen
paper_authors.type: user
paper_authors.user: Yukang2
paper_authors._id: 675bcad776d0e555d6313826
paper_authors.name: Shaozuo Yu
paper_authors._id: 675bcad776d0e555d6313827
paper_authors.name: Sitong Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:51:15.558Z
paper_authors._id: 64119af4ca1bb1e9253b47f1
paper_authors.avatarUrl: /avatars/aef1ce8822e66d342f541292f5e9fedf.svg
paper_authors.fullname: wusitong
paper_authors.type: user
paper_authors.user: stonewst
paper_authors._id: 675bcad776d0e555d6313828
paper_authors.name: Eric Lo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T10:49:59.436Z
paper_authors._id: 6447485270338c0376087917
paper_authors.avatarUrl: /avatars/359d17963a6d1c5f1fddf027169332a5.svg
paper_authors.fullname: Eric Lo
paper_authors.type: user
paper_authors.user: ericlo
paper_authors._id: 675bcad776d0e555d6313829
paper_authors.name: Shu Liu
paper_authors._id: 675bcad776d0e555d631382a
paper_authors.name: Jiaya Jia
paper_title: Lyra: An Efficient and Speech-Centric Framework for Omni-Cognition
paper_summary: As Multi-modal Large Language Models (MLLMs) evolve, expanding beyond
  single-domain capabilities is essential to meet the demands for more versatile
  and efficient AI. However, previous omni-models have insufficiently explored
  speech, neglecting its integration with multi-modality. We introduce Lyra, an
  efficient MLLM that enhances multimodal abilities, including advanced
  long-speech comprehension, sound understanding, cross-modality efficiency, and
  seamless speech interaction. To achieve efficiency and speech-centric
  capabilities, Lyra employs three strategies: (1) leveraging existing
  open-source large models and a proposed multi-modality LoRA to reduce training
  costs and data requirements; (2) using a latent multi-modality regularizer and
  extractor to strengthen the relationship between speech and other modalities,
  thereby enhancing model performance; and (3) constructing a high-quality,
  extensive dataset that includes 1.5M multi-modal (language, vision, audio) data
  samples and 12K long speech samples, enabling Lyra to handle complex long
  speech inputs and achieve more robust omni-cognition. Compared to other
  omni-methods, Lyra achieves state-of-the-art performance on various
  vision-language, vision-speech, and speech-language benchmarks, while also
  using fewer computational resources and less training data.
paper_ai_keywords: Multi-modal Large Language Models (MLLMs) omni-models long-speech comprehension sound understanding cross-modality efficiency seamless speech interaction multi-modality LoRA latent multi-modality regularizer latent multi-modality extractor omni-cognition vision-language benchmarks vision-speech benchmarks speech-language benchmarks