Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming
Recent advances in language models have achieved significant progress.
GPT-4o, as a new milestone, has enabled real-time conversations with humans,
demonstrating near-human natural fluency. Such human-computer interaction
necessitates models with the capability to perform reasoning directly with the
audio modality and generate output in streaming. However, this remains beyond
the reach of current academic models, as they typically depend on extra TTS
systems for speech synthesis, resulting in undesirable latency. This paper
introduces the Mini-Omni, an audio-based end-to-end conversational model,
capable of real-time speech interaction. To achieve this capability, we propose
a text-instructed speech generation method, along with batch-parallel
strategies during inference to further boost the performance. Our method also
helps to retain the original model's language capabilities with minimal
degradation, enabling other works to establish real-time interaction
capabilities. We call this training method "Any Model Can Talk". We also
introduce the VoiceAssistant-400K dataset to fine-tune models optimized for
speech output. To our best knowledge, Mini-Omni is the first fully end-to-end,
open-source model for real-time speech interaction, offering valuable potential
for future research.
66d138638a438492b07c2e7f
Zhifei Xie
claimed_verified
2024-09-03T14:49:03.214Z
66c5d81a4061fd5907443787
/avatars/2e107195b1ff7d06bbc6c9bd4e5620cf.svg
zhifei
user
filicos
66d138638a438492b07c2e80
Changqiao Wu
extracted_confirmed
2024-10-18T01:43:35.876Z
66cdb20c6aee4c865fe9647e
/avatars/d3637e8278c4b14d0f03ebfbfd3d3711.svg
Changqiao Wu
user
gpt-omni
Mini-Omni: Language Models Can Hear, Talk While Thinking in Streaming
Recent advances in language models have achieved significant progress.
GPT-4o, as a new milestone, has enabled real-time conversations with humans,
demonstrating near-human natural fluency. Such human-computer interaction
necessitates models with the capability to perform reasoning directly with the
audio modality and generate output in streaming. However, this remains beyond
the reach of current academic models, as they typically depend on extra TTS
systems for speech synthesis, resulting in undesirable latency. This paper
introduces the Mini-Omni, an audio-based end-to-end conversational model,
capable of real-time speech interaction. To achieve this capability, we propose
a text-instructed speech generation method, along with batch-parallel
strategies during inference to further boost the performance. Our method also
helps to retain the original model's language capabilities with minimal
degradation, enabling other works to establish real-time interaction
capabilities. We call this training method "Any Model Can Talk". We also
introduce the VoiceAssistant-400K dataset to fine-tune models optimized for
speech output. To our best knowledge, Mini-Omni is the first fully end-to-end,
open-source model for real-time speech interaction, offering valuable potential
for future research.
Mini-Omni is an end-to-end audio-based conversational model enabling real-time speech interaction and efficient speech generation.
text-instructed speech generation batch-parallel strategies Any Model Can Talk VoiceAssistant-400K dataset