title: Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming
summary: Recent advances in language models have achieved significant progress.
  GPT-4o, as a new milestone, has enabled real-time conversations with humans,
  demonstrating near-human natural fluency. Such human-computer interaction
  necessitates models with the capability to perform reasoning directly with the
  audio modality and generate output in streaming. However, this remains beyond
  the reach of current academic models, as they typically depend on extra TTS
  systems for speech synthesis, resulting in undesirable latency. This paper
  introduces the Mini-Omni, an audio-based end-to-end conversational model,
  capable of real-time speech interaction. To achieve this capability, we propose
  a text-instructed speech generation method, along with batch-parallel
  strategies during inference to further boost the performance. Our method also
  helps to retain the original model's language capabilities with minimal
  degradation, enabling other works to establish real-time interaction
  capabilities. We call this training method "Any Model Can Talk". We also
  introduce the VoiceAssistant-400K dataset to fine-tune models optimized for
  speech output. To our best knowledge, Mini-Omni is the first fully end-to-end,
  open-source model for real-time speech interaction, offering valuable potential
  for future research.
paper_authors._id: 66d138638a438492b07c2e7f
paper_authors.name: Zhifei Xie
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-03T14:49:03.214Z
paper_authors._id: 66c5d81a4061fd5907443787
paper_authors.avatarUrl: /avatars/2e107195b1ff7d06bbc6c9bd4e5620cf.svg
paper_authors.fullname: zhifei
paper_authors.type: user
paper_authors.user: filicos
paper_authors._id: 66d138638a438492b07c2e80
paper_authors.name: Changqiao Wu
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-10-18T01:43:35.876Z
paper_authors._id: 66cdb20c6aee4c865fe9647e
paper_authors.avatarUrl: /avatars/d3637e8278c4b14d0f03ebfbfd3d3711.svg
paper_authors.fullname: Changqiao Wu
paper_authors.type: user
paper_authors.user: gpt-omni
paper_title: Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming
paper_summary: Recent advances in language models have achieved significant progress.
  GPT-4o, as a new milestone, has enabled real-time conversations with humans,
  demonstrating near-human natural fluency. Such human-computer interaction
  necessitates models with the capability to perform reasoning directly with the
  audio modality and generate output in streaming. However, this remains beyond
  the reach of current academic models, as they typically depend on extra TTS
  systems for speech synthesis, resulting in undesirable latency. This paper
  introduces the Mini-Omni, an audio-based end-to-end conversational model,
  capable of real-time speech interaction. To achieve this capability, we propose
  a text-instructed speech generation method, along with batch-parallel
  strategies during inference to further boost the performance. Our method also
  helps to retain the original model's language capabilities with minimal
  degradation, enabling other works to establish real-time interaction
  capabilities. We call this training method "Any Model Can Talk". We also
  introduce the VoiceAssistant-400K dataset to fine-tune models optimized for
  speech output. To our best knowledge, Mini-Omni is the first fully end-to-end,
  open-source model for real-time speech interaction, offering valuable potential
  for future research.
paper_ai_summary: Mini-Omni is an end-to-end audio-based conversational model enabling real-time speech interaction and efficient speech generation.
paper_ai_keywords: text-instructed speech generation batch-parallel strategies Any Model Can Talk VoiceAssistant-400K dataset