Lyra: An Efficient and Speech-Centric Framework for Omni-Cognition
As Multi-modal Large Language Models (MLLMs) evolve, expanding beyond
single-domain capabilities is essential to meet the demands for more versatile
and efficient AI. However, previous omni-models have insufficiently explored
speech, neglecting its integration with multi-modality. We introduce Lyra, an
efficient MLLM that enhances multimodal abilities, including advanced
long-speech comprehension, sound understanding, cross-modality efficiency, and
seamless speech interaction. To achieve efficiency and speech-centric
capabilities, Lyra employs three strategies: (1) leveraging existing
open-source large models and a proposed multi-modality LoRA to reduce training
costs and data requirements; (2) using a latent multi-modality regularizer and
extractor to strengthen the relationship between speech and other modalities,
thereby enhancing model performance; and (3) constructing a high-quality,
extensive dataset that includes 1.5M multi-modal (language, vision, audio) data
samples and 12K long speech samples, enabling Lyra to handle complex long
speech inputs and achieve more robust omni-cognition. Compared to other
omni-methods, Lyra achieves state-of-the-art performance on various
vision-language, vision-speech, and speech-language benchmarks, while also
using fewer computational resources and less training data.
675bcad776d0e555d631381c
Zhisheng Zhong
admin_assigned
2024-12-16T10:52:26.208Z
65d882d30f35ed3f52d3ae2c
/avatars/22cda67c3fcd7150320ec3551eda90f5.svg
Zhisheng Zhong
user
zszhong
675bcad776d0e555d631381d
Chengyao Wang
claimed_verified
2024-12-13T15:21:46.575Z
6423e35b30b0e4ab36dd1b16
https://cdn-avatars.huggingface.co/v1/production/uploads/6423e35b30b0e4ab36dd1b16/pea6LVDS9PQAxvQt9GUiZ.jpeg
Wang Chengyao
user
wcy1122
675bcad776d0e555d631381e
Yuqi Liu
675bcad776d0e555d631381f
Senqiao Yang
admin_assigned
2024-12-16T10:51:23.068Z
6527b7280ae663e384eb8499
https://cdn-avatars.huggingface.co/v1/production/uploads/6527b7280ae663e384eb8499/73yF3eu2cUx7jVZrhXnXx.jpeg
Senqiao Yang
user
Senqiao
675bcad776d0e555d6313820
Longxiang Tang
admin_assigned
2024-12-16T10:50:10.607Z
6492a0d8d4ae24c933ace44d
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/DXIky2sdPwmiCOR9p-JBQ.png
Longxiang Tang
user
lloong
675bcad776d0e555d6313821
Yuechen Zhang
admin_assigned
2024-12-16T10:50:22.552Z
6418554a0956be7233a1023e
https://cdn-avatars.huggingface.co/v1/production/uploads/6418554a0956be7233a1023e/9EKN0GoOpcDbvBDmAQEJf.png
zhang yuechen
user
julianjuaner
675bcad776d0e555d6313822
Jingyao Li
admin_assigned
2024-12-16T10:50:51.188Z
647ff779d037515be1169e35
/avatars/741096302cbdeb3670a8f31cf2d31d5d.svg
Jingyao Li
user
JingyaoLi
675bcad776d0e555d6313823
Tianyuan Qu
admin_assigned
2024-12-16T10:45:31.921Z
66e79b3c1c79fc2e51dc1d60
/avatars/8706336e9e7a417505c9bb32583a662f.svg
QU Tianyuan
user
TainU
675bcad776d0e555d6313824
Yanwei Li
admin_assigned
2024-12-16T10:45:47.023Z
643ff78cdc984afcbbbc3b1a
/avatars/eec5198ce88aaf8156840bec0d190a7f.svg
Yanwei Li
user
YanweiLi
675bcad776d0e555d6313825
Yukang Chen
admin_assigned
2024-12-16T10:50:59.539Z
65b31d77f32c79fea79ba4d1
/avatars/8c51c1ff337ac372c7caafa4fa5019dc.svg
Yukang Chen
user
Yukang2
675bcad776d0e555d6313826
Shaozuo Yu
675bcad776d0e555d6313827
Sitong Wu
admin_assigned
2024-12-16T10:51:15.558Z
64119af4ca1bb1e9253b47f1
/avatars/aef1ce8822e66d342f541292f5e9fedf.svg
wusitong
user
stonewst
675bcad776d0e555d6313828
Eric Lo
admin_assigned
2024-12-16T10:49:59.436Z
6447485270338c0376087917
/avatars/359d17963a6d1c5f1fddf027169332a5.svg
Eric Lo
user
ericlo
675bcad776d0e555d6313829
Shu Liu
675bcad776d0e555d631382a
Jiaya Jia
Lyra: An Efficient and Speech-Centric Framework for Omni-Cognition
As Multi-modal Large Language Models (MLLMs) evolve, expanding beyond
single-domain capabilities is essential to meet the demands for more versatile
and efficient AI. However, previous omni-models have insufficiently explored
speech, neglecting its integration with multi-modality. We introduce Lyra, an
efficient MLLM that enhances multimodal abilities, including advanced
long-speech comprehension, sound understanding, cross-modality efficiency, and
seamless speech interaction. To achieve efficiency and speech-centric
capabilities, Lyra employs three strategies: (1) leveraging existing
open-source large models and a proposed multi-modality LoRA to reduce training
costs and data requirements; (2) using a latent multi-modality regularizer and
extractor to strengthen the relationship between speech and other modalities,
thereby enhancing model performance; and (3) constructing a high-quality,
extensive dataset that includes 1.5M multi-modal (language, vision, audio) data
samples and 12K long speech samples, enabling Lyra to handle complex long
speech inputs and achieve more robust omni-cognition. Compared to other
omni-methods, Lyra achieves state-of-the-art performance on various
vision-language, vision-speech, and speech-language benchmarks, while also
using fewer computational resources and less training data.
Multi-modal Large Language Models (MLLMs) omni-models long-speech comprehension sound understanding cross-modality efficiency seamless speech interaction multi-modality LoRA latent multi-modality regularizer latent multi-modality extractor omni-cognition vision-language benchmarks vision-speech benchmarks speech-language benchmarks