MinMo: A Multimodal Large Language Model for Seamless Voice Interaction
Recent advancements in large language models (LLMs) and multimodal
speech-text models have laid the groundwork for seamless voice interactions,
enabling real-time, natural, and human-like conversations. Previous models for
voice interactions are categorized as native and aligned. Native models
integrate speech and text processing in one framework but struggle with issues
like differing sequence lengths and insufficient pre-training. Aligned models
maintain text LLM capabilities but are often limited by small datasets and a
narrow focus on speech tasks. In this work, we introduce MinMo, a Multimodal
Large Language Model with approximately 8B parameters for seamless voice
interaction. We address the main limitations of prior aligned multimodal
models. We train MinMo through multiple stages of speech-to-text alignment,
text-to-speech alignment, speech-to-speech alignment, and duplex interaction
alignment, on 1.4 million hours of diverse speech data and a broad range of
speech tasks. After the multi-stage training, MinMo achieves state-of-the-art
performance across various benchmarks for voice comprehension and generation
while maintaining the capabilities of text LLMs, and also facilitates
full-duplex conversation, that is, simultaneous two-way communication between
the user and the system. Moreover, we propose a novel and simple voice decoder
that outperforms prior models in voice generation. The enhanced
instruction-following capabilities of MinMo supports controlling speech
generation based on user instructions, with various nuances including emotions,
dialects, and speaking rates, and mimicking specific voices. For MinMo, the
speech-to-text latency is approximately 100ms, full-duplex latency is
approximately 600ms in theory and 800ms in practice. The MinMo project web page
is https://funaudiollm.github.io/minmo, and the code and models will be
released soon.
6785e88a6bc168b98ae46a55
Qian Chen
6785e88a6bc168b98ae46a56
Yafeng Chen
6785e88a6bc168b98ae46a57
Yanni Chen
6785e88a6bc168b98ae46a58
Mengzhe Chen
admin_assigned
2025-01-14T09:16:32.211Z
632aa1565f2ff1958c053f9f
https://cdn-avatars.huggingface.co/v1/production/uploads/1663738082852-noauth.png
chenmengzhe
user
chenmengzhe
6785e88a6bc168b98ae46a59
Yingda Chen
admin_assigned
2025-01-14T09:16:41.268Z
6215ca5692c0ecfba9186921
https://cdn-avatars.huggingface.co/v1/production/uploads/1645595695434-6215ca5692c0ecfba9186921.png
Yingda Chen
user
Yingda
6785e88a6bc168b98ae46a5a
Chong Deng
6785e88a6bc168b98ae46a5b
Zhihao Du
admin_assigned
2025-01-14T09:16:59.075Z
64f15d3041425edddeec9410
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/caz_i5eCbGk-B9013DHqb.jpeg
zhihaodu
user
zhihaodu
6785e88a6bc168b98ae46a5c
Ruize Gao
admin_assigned
2025-01-14T09:17:22.485Z
6629ed94aabce1b25c3db90c
/avatars/cbc39db81c8e8f950d3bd2c2e03f71c8.svg
Ruize Gao
user
gaoruize
6785e88a6bc168b98ae46a5d
Changfeng Gao
6785e88a6bc168b98ae46a5e
Zhifu Gao
admin_assigned
2025-01-14T09:17:48.070Z
63d7dcec635499f2163d36b2
/avatars/1e4afa5807f22025e80af2ebe2ac08d1.svg
zhifu gao
user
langgz
6785e88a6bc168b98ae46a5f
Yabin Li
6785e88a6bc168b98ae46a60
Xiang Lv
6785e88a6bc168b98ae46a61
Jiaqing Liu
admin_assigned
2025-01-14T09:18:16.920Z
63f268f8be95ed4c9a9b4948
/avatars/5e8cee51637c5c66595bb22c0734145e.svg
Jiaqing Liu
user
Nick6
6785e88a6bc168b98ae46a62
Haoneng Luo
6785e88a6bc168b98ae46a63
Bin Ma
6785e88a6bc168b98ae46a64
Chongjia Ni
admin_assigned
2025-01-14T09:16:04.510Z
64af6a1c7ab7586520eb6f64
/avatars/8cd025c8e6a1a5ecc94f206bfa90ef65.svg
Ni
user
Chongjia
6785e88a6bc168b98ae46a65
Xian Shi
6785e88a6bc168b98ae46a66
Jialong Tang
admin_assigned
2025-01-14T09:18:50.994Z
63281d05ac205d01918b5fc7
/avatars/fc3e0f7285bb2869a92670f764dfc535.svg
Jialong Tang
user
Jialong
6785e88a6bc168b98ae46a67
Hui Wang
6785e88a6bc168b98ae46a68
Hao Wang
6785e88a6bc168b98ae46a69
Wen Wang
6785e88a6bc168b98ae46a6a
Yuxuan Wang
6785e88a6bc168b98ae46a6b
Yunlan Xu
6785e88a6bc168b98ae46a6c
Fan Yu
6785e88a6bc168b98ae46a6d
Zhijie Yan
6785e88a6bc168b98ae46a6e
Yexin Yang
6785e88a6bc168b98ae46a6f
Baosong Yang
admin_assigned
2025-01-14T09:24:22.882Z
64b0a77df12b47366663884c
/avatars/a212ea862abb5966060e439dd0e7656f.svg
Baosong Yang
user
Baosong
6785e88a6bc168b98ae46a70
Xian Yang
6785e88a6bc168b98ae46a71
Guanrou Yang
6785e88a6bc168b98ae46a72
Tianyu Zhao
claimed_verified
2025-01-15T08:49:13.682Z
66fa63785ea4244ba16b9dce
/avatars/c0382731298fb9888922e1a469a7749d.svg
Tianyu Zhao
user
hengwu-zty
6785e88a6bc168b98ae46a73
Qinglin Zhang
6785e88a6bc168b98ae46a74
Shiliang Zhang
admin_assigned
2025-01-14T09:25:14.833Z
632493f2a099f5a84cee66d9
/avatars/6c0ca937e35d2ef32eed6b99bd1d7313.svg
Zhang
user
shiliang
6785e88a6bc168b98ae46a75
Nan Zhao
6785e88a6bc168b98ae46a76
Pei Zhang
6785e88a6bc168b98ae46a77
Chong Zhang
claimed_verified
2025-03-18T08:12:26.037Z
649aa7d0ef592b26c3d4ead4
/avatars/1174fe2a396b553312896914401947cb.svg
iris2c
user
iris2c
6785e88a6bc168b98ae46a78
Jinren Zhou
MinMo: A Multimodal Large Language Model for Seamless Voice Interaction
Recent advancements in large language models (LLMs) and multimodal
speech-text models have laid the groundwork for seamless voice interactions,
enabling real-time, natural, and human-like conversations. Previous models for
voice interactions are categorized as native and aligned. Native models
integrate speech and text processing in one framework but struggle with issues
like differing sequence lengths and insufficient pre-training. Aligned models
maintain text LLM capabilities but are often limited by small datasets and a
narrow focus on speech tasks. In this work, we introduce MinMo, a Multimodal
Large Language Model with approximately 8B parameters for seamless voice
interaction. We address the main limitations of prior aligned multimodal
models. We train MinMo through multiple stages of speech-to-text alignment,
text-to-speech alignment, speech-to-speech alignment, and duplex interaction
alignment, on 1.4 million hours of diverse speech data and a broad range of
speech tasks. After the multi-stage training, MinMo achieves state-of-the-art
performance across various benchmarks for voice comprehension and generation
while maintaining the capabilities of text LLMs, and also facilitates
full-duplex conversation, that is, simultaneous two-way communication between
the user and the system. Moreover, we propose a novel and simple voice decoder
that outperforms prior models in voice generation. The enhanced
instruction-following capabilities of MinMo supports controlling speech
generation based on user instructions, with various nuances including emotions,
dialects, and speaking rates, and mimicking specific voices. For MinMo, the
speech-to-text latency is approximately 100ms, full-duplex latency is
approximately 600ms in theory and 800ms in practice. The MinMo project web page
is https://funaudiollm.github.io/minmo, and the code and models will be
released soon.
MinMo, a multimodal large language model, integrates speech and text processing to achieve state-of-the-art performance in voice comprehension and generation, while enabling full-duplex conversation and instruction-following capabilities.
multimodal large language model native models aligned models speech-to-text alignment text-to-speech alignment speech-to-speech alignment duplex interaction alignment full-duplex conversation voice decoder voice generation instruction-following capabilities