title: LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive
    Streaming Speech Synthesis
summary: Real-time, intelligent, and natural speech interaction is an essential part
  of the next-generation human-computer interaction. Recent advancements have
  showcased the potential of building intelligent spoken chatbots based on large
  language models (LLMs). In this paper, we introduce LLaMA-Omni 2, a series of
  speech language models (SpeechLMs) ranging from 0.5B to 14B parameters, capable
  of achieving high-quality real-time speech interaction. LLaMA-Omni 2 is built
  upon the Qwen2.5 series models, integrating a speech encoder and an
  autoregressive streaming speech decoder. Despite being trained on only 200K
  multi-turn speech dialogue samples, LLaMA-Omni 2 demonstrates strong
  performance on several spoken question answering and speech instruction
  following benchmarks, surpassing previous state-of-the-art SpeechLMs like
  GLM-4-Voice, which was trained on millions of hours of speech data.
paper_authors._id: 681975abba26bf20601bb7ca
paper_authors.name: Qingkai Fang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T09:15:43.683Z
paper_authors._id: 65b7573482d384513443875e
paper_authors.avatarUrl: /avatars/0f2175e4adf507f5ccb0636c1cb647de.svg
paper_authors.fullname: Qingkai Fang
paper_authors.type: user
paper_authors.user: poeroz
paper_authors._id: 681975abba26bf20601bb7cb
paper_authors.name: Yan Zhou
paper_authors._id: 681975abba26bf20601bb7cc
paper_authors.name: Shoutao Guo
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T09:15:53.154Z
paper_authors._id: 66680c0505c407bfea87667c
paper_authors.avatarUrl: /avatars/e3c26d2eb13fe8ad2b3fd16897e61e6d.svg
paper_authors.fullname: Shoutao Guo
paper_authors.type: user
paper_authors.user: guoshoutao
paper_authors._id: 681975abba26bf20601bb7cd
paper_authors.name: Shaolei Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-06T09:16:00.139Z
paper_authors._id: 64803e5dc57f629056c601f1
paper_authors.avatarUrl: /avatars/a9e9c97c70714e3a29bef2cf929ee6b3.svg
paper_authors.fullname: Shaolei Zhang
paper_authors.type: user
paper_authors.user: zhangshaolei
paper_authors._id: 681975abba26bf20601bb7ce
paper_authors.name: Yang Feng
paper_title: LLaMA-Omni2: LLM-based Real-time Spoken Chatbot with Autoregressive
    Streaming Speech Synthesis
paper_summary: Real-time, intelligent, and natural speech interaction is an essential part
  of the next-generation human-computer interaction. Recent advancements have
  showcased the potential of building intelligent spoken chatbots based on large
  language models (LLMs). In this paper, we introduce LLaMA-Omni 2, a series of
  speech language models (SpeechLMs) ranging from 0.5B to 14B parameters, capable
  of achieving high-quality real-time speech interaction. LLaMA-Omni 2 is built
  upon the Qwen2.5 series models, integrating a speech encoder and an
  autoregressive streaming speech decoder. Despite being trained on only 200K
  multi-turn speech dialogue samples, LLaMA-Omni 2 demonstrates strong
  performance on several spoken question answering and speech instruction
  following benchmarks, surpassing previous state-of-the-art SpeechLMs like
  GLM-4-Voice, which was trained on millions of hours of speech data.
paper_ai_summary: LLaMA-Omni 2, a series of speech language models with parameters ranging from 0.5B to 14B, achieves high-quality real-time speech interaction through a speech encoder and autoregressive streaming speech decoder, outperforming models like GLM-4-Voice with significantly less training data.
paper_ai_keywords: speech language models SpeechLMs Qwen2.5 speech encoder autoregressive streaming speech decoder spoken question answering speech instruction following GLM-4-Voice