LLaMA-Omni: Seamless Speech Interaction with Large Language Models
Models like GPT-4o enable real-time interaction with large language models
(LLMs) through speech, significantly enhancing user experience compared to
traditional text-based interaction. However, there is still a lack of
exploration on how to build speech interaction models based on open-source
LLMs. To address this, we propose LLaMA-Omni, a novel model architecture
designed for low-latency and high-quality speech interaction with LLMs.
LLaMA-Omni integrates a pretrained speech encoder, a speech adaptor, an LLM,
and a streaming speech decoder. It eliminates the need for speech
transcription, and can simultaneously generate text and speech responses
directly from speech instructions with extremely low latency. We build our
model based on the latest Llama-3.1-8B-Instruct model. To align the model with
speech interaction scenarios, we construct a dataset named InstructS2S-200K,
which includes 200K speech instructions and corresponding speech responses.
Experimental results show that compared to previous speech-language models,
LLaMA-Omni provides better responses in both content and style, with a response
latency as low as 226ms. Additionally, training LLaMA-Omni takes less than 3
days on just 4 GPUs, paving the way for the efficient development of
speech-language models in the future.
66e0fb6675d9226ba1120693
Qingkai Fang
claimed_verified
2024-09-11T07:27:09.630Z
65b7573482d384513443875e
/avatars/0f2175e4adf507f5ccb0636c1cb647de.svg
Qingkai Fang
user
poeroz
66e0fb6675d9226ba1120694
Shoutao Guo
admin_assigned
2024-09-11T10:02:01.044Z
66680c0505c407bfea87667c
/avatars/e3c26d2eb13fe8ad2b3fd16897e61e6d.svg
Shoutao Guo
user
guoshoutao
66e0fb6675d9226ba1120695
Yan Zhou
claimed_verified
2024-09-11T09:57:10.094Z
667a706ece13150b7c587de1
/avatars/ece746b324d8f121a4305ac405988f55.svg
Yan Zhou
user
zysgdd
66e0fb6675d9226ba1120696
Zhengrui Ma
admin_assigned
2024-09-11T10:02:06.688Z
64896496ffac083138eef1e9
/avatars/216167afeb302896511deb093dbf8700.svg
Zhengrui Ma
user
Paulmzr
66e0fb6675d9226ba1120697
Shaolei Zhang
admin_assigned
2024-09-11T10:02:12.088Z
64803e5dc57f629056c601f1
/avatars/a9e9c97c70714e3a29bef2cf929ee6b3.svg
Shaolei Zhang
user
zhangshaolei
66e0fb6675d9226ba1120698
Yang Feng
LLaMA-Omni: Seamless Speech Interaction with Large Language Models
Models like GPT-4o enable real-time interaction with large language models
(LLMs) through speech, significantly enhancing user experience compared to
traditional text-based interaction. However, there is still a lack of
exploration on how to build speech interaction models based on open-source
LLMs. To address this, we propose LLaMA-Omni, a novel model architecture
designed for low-latency and high-quality speech interaction with LLMs.
LLaMA-Omni integrates a pretrained speech encoder, a speech adaptor, an LLM,
and a streaming speech decoder. It eliminates the need for speech
transcription, and can simultaneously generate text and speech responses
directly from speech instructions with extremely low latency. We build our
model based on the latest Llama-3.1-8B-Instruct model. To align the model with
speech interaction scenarios, we construct a dataset named InstructS2S-200K,
which includes 200K speech instructions and corresponding speech responses.
Experimental results show that compared to previous speech-language models,
LLaMA-Omni provides better responses in both content and style, with a response
latency as low as 226ms. Additionally, training LLaMA-Omni takes less than 3
days on just 4 GPUs, paving the way for the efficient development of
speech-language models in the future.
LLaMA-Omni integrates a speech encoder, adaptor, LLM, and decoder for low-latency and high-quality speech interaction with LLMs, outperforming previous models in both content and style.
speech encoder speech adaptor LLM streaming speech decoder InstructS2S-200K speech-language models