title: MinMo: A Multimodal Large Language Model for Seamless Voice Interaction
summary: Recent advancements in large language models (LLMs) and multimodal
  speech-text models have laid the groundwork for seamless voice interactions,
  enabling real-time, natural, and human-like conversations. Previous models for
  voice interactions are categorized as native and aligned. Native models
  integrate speech and text processing in one framework but struggle with issues
  like differing sequence lengths and insufficient pre-training. Aligned models
  maintain text LLM capabilities but are often limited by small datasets and a
  narrow focus on speech tasks. In this work, we introduce MinMo, a Multimodal
  Large Language Model with approximately 8B parameters for seamless voice
  interaction. We address the main limitations of prior aligned multimodal
  models. We train MinMo through multiple stages of speech-to-text alignment,
  text-to-speech alignment, speech-to-speech alignment, and duplex interaction
  alignment, on 1.4 million hours of diverse speech data and a broad range of
  speech tasks. After the multi-stage training, MinMo achieves state-of-the-art
  performance across various benchmarks for voice comprehension and generation
  while maintaining the capabilities of text LLMs, and also facilitates
  full-duplex conversation, that is, simultaneous two-way communication between
  the user and the system. Moreover, we propose a novel and simple voice decoder
  that outperforms prior models in voice generation. The enhanced
  instruction-following capabilities of MinMo supports controlling speech
  generation based on user instructions, with various nuances including emotions,
  dialects, and speaking rates, and mimicking specific voices. For MinMo, the
  speech-to-text latency is approximately 100ms, full-duplex latency is
  approximately 600ms in theory and 800ms in practice. The MinMo project web page
  is https://funaudiollm.github.io/minmo, and the code and models will be
  released soon.
paper_authors._id: 6785e88a6bc168b98ae46a55
paper_authors.name: Qian Chen
paper_authors._id: 6785e88a6bc168b98ae46a56
paper_authors.name: Yafeng Chen
paper_authors._id: 6785e88a6bc168b98ae46a57
paper_authors.name: Yanni Chen
paper_authors._id: 6785e88a6bc168b98ae46a58
paper_authors.name: Mengzhe Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-14T09:16:32.211Z
paper_authors._id: 632aa1565f2ff1958c053f9f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1663738082852-noauth.png
paper_authors.fullname: chenmengzhe
paper_authors.type: user
paper_authors.user: chenmengzhe
paper_authors._id: 6785e88a6bc168b98ae46a59
paper_authors.name: Yingda Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-14T09:16:41.268Z
paper_authors._id: 6215ca5692c0ecfba9186921
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1645595695434-6215ca5692c0ecfba9186921.png
paper_authors.fullname: Yingda Chen
paper_authors.type: user
paper_authors.user: Yingda
paper_authors._id: 6785e88a6bc168b98ae46a5a
paper_authors.name: Chong Deng
paper_authors._id: 6785e88a6bc168b98ae46a5b
paper_authors.name: Zhihao Du
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-14T09:16:59.075Z
paper_authors._id: 64f15d3041425edddeec9410
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/caz_i5eCbGk-B9013DHqb.jpeg
paper_authors.fullname: zhihaodu
paper_authors.type: user
paper_authors.user: zhihaodu
paper_authors._id: 6785e88a6bc168b98ae46a5c
paper_authors.name: Ruize Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-14T09:17:22.485Z
paper_authors._id: 6629ed94aabce1b25c3db90c
paper_authors.avatarUrl: /avatars/cbc39db81c8e8f950d3bd2c2e03f71c8.svg
paper_authors.fullname: Ruize Gao
paper_authors.type: user
paper_authors.user: gaoruize
paper_authors._id: 6785e88a6bc168b98ae46a5d
paper_authors.name: Changfeng Gao
paper_authors._id: 6785e88a6bc168b98ae46a5e
paper_authors.name: Zhifu Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-14T09:17:48.070Z
paper_authors._id: 63d7dcec635499f2163d36b2
paper_authors.avatarUrl: /avatars/1e4afa5807f22025e80af2ebe2ac08d1.svg
paper_authors.fullname: zhifu gao
paper_authors.type: user
paper_authors.user: langgz
paper_authors._id: 6785e88a6bc168b98ae46a5f
paper_authors.name: Yabin Li
paper_authors._id: 6785e88a6bc168b98ae46a60
paper_authors.name: Xiang Lv
paper_authors._id: 6785e88a6bc168b98ae46a61
paper_authors.name: Jiaqing Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-14T09:18:16.920Z
paper_authors._id: 63f268f8be95ed4c9a9b4948
paper_authors.avatarUrl: /avatars/5e8cee51637c5c66595bb22c0734145e.svg
paper_authors.fullname: Jiaqing Liu
paper_authors.type: user
paper_authors.user: Nick6
paper_authors._id: 6785e88a6bc168b98ae46a62
paper_authors.name: Haoneng Luo
paper_authors._id: 6785e88a6bc168b98ae46a63
paper_authors.name: Bin Ma
paper_authors._id: 6785e88a6bc168b98ae46a64
paper_authors.name: Chongjia Ni
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-14T09:16:04.510Z
paper_authors._id: 64af6a1c7ab7586520eb6f64
paper_authors.avatarUrl: /avatars/8cd025c8e6a1a5ecc94f206bfa90ef65.svg
paper_authors.fullname: Ni
paper_authors.type: user
paper_authors.user: Chongjia
paper_authors._id: 6785e88a6bc168b98ae46a65
paper_authors.name: Xian Shi
paper_authors._id: 6785e88a6bc168b98ae46a66
paper_authors.name: Jialong Tang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-14T09:18:50.994Z
paper_authors._id: 63281d05ac205d01918b5fc7
paper_authors.avatarUrl: /avatars/fc3e0f7285bb2869a92670f764dfc535.svg
paper_authors.fullname: Jialong Tang
paper_authors.type: user
paper_authors.user: Jialong
paper_authors._id: 6785e88a6bc168b98ae46a67
paper_authors.name: Hui Wang
paper_authors._id: 6785e88a6bc168b98ae46a68
paper_authors.name: Hao Wang
paper_authors._id: 6785e88a6bc168b98ae46a69
paper_authors.name: Wen Wang
paper_authors._id: 6785e88a6bc168b98ae46a6a
paper_authors.name: Yuxuan Wang
paper_authors._id: 6785e88a6bc168b98ae46a6b
paper_authors.name: Yunlan Xu
paper_authors._id: 6785e88a6bc168b98ae46a6c
paper_authors.name: Fan Yu
paper_authors._id: 6785e88a6bc168b98ae46a6d
paper_authors.name: Zhijie Yan
paper_authors._id: 6785e88a6bc168b98ae46a6e
paper_authors.name: Yexin Yang
paper_authors._id: 6785e88a6bc168b98ae46a6f
paper_authors.name: Baosong Yang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-14T09:24:22.882Z
paper_authors._id: 64b0a77df12b47366663884c
paper_authors.avatarUrl: /avatars/a212ea862abb5966060e439dd0e7656f.svg
paper_authors.fullname: Baosong Yang
paper_authors.type: user
paper_authors.user: Baosong
paper_authors._id: 6785e88a6bc168b98ae46a70
paper_authors.name: Xian Yang
paper_authors._id: 6785e88a6bc168b98ae46a71
paper_authors.name: Guanrou Yang
paper_authors._id: 6785e88a6bc168b98ae46a72
paper_authors.name: Tianyu Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-15T08:49:13.682Z
paper_authors._id: 66fa63785ea4244ba16b9dce
paper_authors.avatarUrl: /avatars/c0382731298fb9888922e1a469a7749d.svg
paper_authors.fullname: Tianyu Zhao
paper_authors.type: user
paper_authors.user: hengwu-zty
paper_authors._id: 6785e88a6bc168b98ae46a73
paper_authors.name: Qinglin Zhang
paper_authors._id: 6785e88a6bc168b98ae46a74
paper_authors.name: Shiliang Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-14T09:25:14.833Z
paper_authors._id: 632493f2a099f5a84cee66d9
paper_authors.avatarUrl: /avatars/6c0ca937e35d2ef32eed6b99bd1d7313.svg
paper_authors.fullname: Zhang
paper_authors.type: user
paper_authors.user: shiliang
paper_authors._id: 6785e88a6bc168b98ae46a75
paper_authors.name: Nan Zhao
paper_authors._id: 6785e88a6bc168b98ae46a76
paper_authors.name: Pei Zhang
paper_authors._id: 6785e88a6bc168b98ae46a77
paper_authors.name: Chong Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-18T08:12:26.037Z
paper_authors._id: 649aa7d0ef592b26c3d4ead4
paper_authors.avatarUrl: /avatars/1174fe2a396b553312896914401947cb.svg
paper_authors.fullname: Chong Zhang
paper_authors.type: user
paper_authors.user: iris2c
paper_authors._id: 6785e88a6bc168b98ae46a78
paper_authors.name: Jinren Zhou
paper_title: MinMo: A Multimodal Large Language Model for Seamless Voice Interaction
paper_summary: Recent advancements in large language models (LLMs) and multimodal
  speech-text models have laid the groundwork for seamless voice interactions,
  enabling real-time, natural, and human-like conversations. Previous models for
  voice interactions are categorized as native and aligned. Native models
  integrate speech and text processing in one framework but struggle with issues
  like differing sequence lengths and insufficient pre-training. Aligned models
  maintain text LLM capabilities but are often limited by small datasets and a
  narrow focus on speech tasks. In this work, we introduce MinMo, a Multimodal
  Large Language Model with approximately 8B parameters for seamless voice
  interaction. We address the main limitations of prior aligned multimodal
  models. We train MinMo through multiple stages of speech-to-text alignment,
  text-to-speech alignment, speech-to-speech alignment, and duplex interaction
  alignment, on 1.4 million hours of diverse speech data and a broad range of
  speech tasks. After the multi-stage training, MinMo achieves state-of-the-art
  performance across various benchmarks for voice comprehension and generation
  while maintaining the capabilities of text LLMs, and also facilitates
  full-duplex conversation, that is, simultaneous two-way communication between
  the user and the system. Moreover, we propose a novel and simple voice decoder
  that outperforms prior models in voice generation. The enhanced
  instruction-following capabilities of MinMo supports controlling speech
  generation based on user instructions, with various nuances including emotions,
  dialects, and speaking rates, and mimicking specific voices. For MinMo, the
  speech-to-text latency is approximately 100ms, full-duplex latency is
  approximately 600ms in theory and 800ms in practice. The MinMo project web page
  is https://funaudiollm.github.io/minmo, and the code and models will be
  released soon.
paper_ai_keywords: large language models (LLMs) multimodal speech-text models voice interactions native models aligned models MinMo Multimodal Large Language Model speech-to-text alignment text-to-speech alignment speech-to-speech alignment duplex interaction alignment speech data speech tasks benchmarks voice comprehension voice generation full-duplex conversation voice decoder instruction-following capabilities emotions dialects speaking rates specific voices speech-to-text latency full-duplex latency