FunAudioLLM: Voice Understanding and Generation Foundation Models for
  Natural Interaction Between Humans and LLMs
This report introduces FunAudioLLM, a model family designed to enhance
natural voice interactions between humans and large language models (LLMs). At
its core are two innovative models: SenseVoice, which handles multilingual
speech recognition, emotion recognition, and audio event detection; and
CosyVoice, which facilitates natural speech generation with control over
multiple languages, timbre, speaking style, and speaker identity.
SenseVoice-Small delivers exceptionally low-latency ASR for 5 languages, and
SenseVoice-Large supports high-precision ASR for over 50 languages, while
CosyVoice excels in multi-lingual voice generation, zero-shot in-context
learning, cross-lingual voice cloning, and instruction-following capabilities.
The models related to SenseVoice and CosyVoice have been open-sourced on
Modelscope and Huggingface, along with the corresponding training, inference,
and fine-tuning codes released on GitHub. By integrating these models with
LLMs, FunAudioLLM enables applications such as speech-to-speech translation,
emotional voice chat, interactive podcasts, and expressive audiobook narration,
thereby pushing the boundaries of voice interaction technology. Demos are
available at https://fun-audio-llm.github.io, and the code can be accessed at
https://github.com/FunAudioLLM.
668b611fa9ee6373c751c9b6
Tongyi SpeechTeam
FunAudioLLM: Voice Understanding and Generation Foundation Models for
  Natural Interaction Between Humans and LLMs
This report introduces FunAudioLLM, a model family designed to enhance
natural voice interactions between humans and large language models (LLMs). At
its core are two innovative models: SenseVoice, which handles multilingual
speech recognition, emotion recognition, and audio event detection; and
CosyVoice, which facilitates natural speech generation with control over
multiple languages, timbre, speaking style, and speaker identity.
SenseVoice-Small delivers exceptionally low-latency ASR for 5 languages, and
SenseVoice-Large supports high-precision ASR for over 50 languages, while
CosyVoice excels in multi-lingual voice generation, zero-shot in-context
learning, cross-lingual voice cloning, and instruction-following capabilities.
The models related to SenseVoice and CosyVoice have been open-sourced on
Modelscope and Huggingface, along with the corresponding training, inference,
and fine-tuning codes released on GitHub. By integrating these models with
LLMs, FunAudioLLM enables applications such as speech-to-speech translation,
emotional voice chat, interactive podcasts, and expressive audiobook narration,
thereby pushing the boundaries of voice interaction technology. Demos are
available at https://fun-audio-llm.github.io, and the code can be accessed at
https://github.com/FunAudioLLM.
FunAudioLLM enhances voice interactions by integrating SenseVoice for multilingual speech recognition, emotion detection, and audio event detection with CosyVoice for natural speech generation across languages, timbres, and styles.
multi-lingual speech recognition emotion recognition audio event detection natural speech generation multi-lingual voice generation zero-shot in-context learning cross-lingual voice cloning instruction-following capabilities speech-to-speech translation emotional voice chat interactive podcasts expressive audiobook narration