title: LLaMA-Omni: Seamless Speech Interaction with Large Language Models
summary: Models like GPT-4o enable real-time interaction with large language models
  (LLMs) through speech, significantly enhancing user experience compared to
  traditional text-based interaction. However, there is still a lack of
  exploration on how to build speech interaction models based on open-source
  LLMs. To address this, we propose LLaMA-Omni, a novel model architecture
  designed for low-latency and high-quality speech interaction with LLMs.
  LLaMA-Omni integrates a pretrained speech encoder, a speech adaptor, an LLM,
  and a streaming speech decoder. It eliminates the need for speech
  transcription, and can simultaneously generate text and speech responses
  directly from speech instructions with extremely low latency. We build our
  model based on the latest Llama-3.1-8B-Instruct model. To align the model with
  speech interaction scenarios, we construct a dataset named InstructS2S-200K,
  which includes 200K speech instructions and corresponding speech responses.
  Experimental results show that compared to previous speech-language models,
  LLaMA-Omni provides better responses in both content and style, with a response
  latency as low as 226ms. Additionally, training LLaMA-Omni takes less than 3
  days on just 4 GPUs, paving the way for the efficient development of
  speech-language models in the future.
paper_authors._id: 66e0fb6675d9226ba1120693
paper_authors.name: Qingkai Fang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-11T07:27:09.630Z
paper_authors._id: 65b7573482d384513443875e
paper_authors.avatarUrl: /avatars/0f2175e4adf507f5ccb0636c1cb647de.svg
paper_authors.fullname: Qingkai Fang
paper_authors.type: user
paper_authors.user: poeroz
paper_authors._id: 66e0fb6675d9226ba1120694
paper_authors.name: Shoutao Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-11T10:02:01.044Z
paper_authors._id: 66680c0505c407bfea87667c
paper_authors.avatarUrl: /avatars/e3c26d2eb13fe8ad2b3fd16897e61e6d.svg
paper_authors.fullname: Shoutao Guo
paper_authors.type: user
paper_authors.user: guoshoutao
paper_authors._id: 66e0fb6675d9226ba1120695
paper_authors.name: Yan Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-11T09:57:10.094Z
paper_authors._id: 667a706ece13150b7c587de1
paper_authors.avatarUrl: /avatars/ece746b324d8f121a4305ac405988f55.svg
paper_authors.fullname: Yan Zhou
paper_authors.type: user
paper_authors.user: zysgdd
paper_authors._id: 66e0fb6675d9226ba1120696
paper_authors.name: Zhengrui Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-11T10:02:06.688Z
paper_authors._id: 64896496ffac083138eef1e9
paper_authors.avatarUrl: /avatars/216167afeb302896511deb093dbf8700.svg
paper_authors.fullname: Zhengrui Ma
paper_authors.type: user
paper_authors.user: Paulmzr
paper_authors._id: 66e0fb6675d9226ba1120697
paper_authors.name: Shaolei Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-11T10:02:12.088Z
paper_authors._id: 64803e5dc57f629056c601f1
paper_authors.avatarUrl: /avatars/a9e9c97c70714e3a29bef2cf929ee6b3.svg
paper_authors.fullname: Shaolei Zhang
paper_authors.type: user
paper_authors.user: zhangshaolei
paper_authors._id: 66e0fb6675d9226ba1120698
paper_authors.name: Yang Feng
paper_title: LLaMA-Omni: Seamless Speech Interaction with Large Language Models
paper_summary: Models like GPT-4o enable real-time interaction with large language models
  (LLMs) through speech, significantly enhancing user experience compared to
  traditional text-based interaction. However, there is still a lack of
  exploration on how to build speech interaction models based on open-source
  LLMs. To address this, we propose LLaMA-Omni, a novel model architecture
  designed for low-latency and high-quality speech interaction with LLMs.
  LLaMA-Omni integrates a pretrained speech encoder, a speech adaptor, an LLM,
  and a streaming speech decoder. It eliminates the need for speech
  transcription, and can simultaneously generate text and speech responses
  directly from speech instructions with extremely low latency. We build our
  model based on the latest Llama-3.1-8B-Instruct model. To align the model with
  speech interaction scenarios, we construct a dataset named InstructS2S-200K,
  which includes 200K speech instructions and corresponding speech responses.
  Experimental results show that compared to previous speech-language models,
  LLaMA-Omni provides better responses in both content and style, with a response
  latency as low as 226ms. Additionally, training LLaMA-Omni takes less than 3
  days on just 4 GPUs, paving the way for the efficient development of
  speech-language models in the future.
paper_ai_summary: LLaMA-Omni integrates a speech encoder, adaptor, LLM, and decoder for low-latency and high-quality speech interaction with LLMs, outperforming previous models in both content and style.
paper_ai_keywords: speech encoder speech adaptor LLM streaming speech decoder InstructS2S-200K speech-language models