title: FunAudioLLM: Voice Understanding and Generation Foundation Models for
    Natural Interaction Between Humans and LLMs
summary: This report introduces FunAudioLLM, a model family designed to enhance
  natural voice interactions between humans and large language models (LLMs). At
  its core are two innovative models: SenseVoice, which handles multilingual
  speech recognition, emotion recognition, and audio event detection; and
  CosyVoice, which facilitates natural speech generation with control over
  multiple languages, timbre, speaking style, and speaker identity.
  SenseVoice-Small delivers exceptionally low-latency ASR for 5 languages, and
  SenseVoice-Large supports high-precision ASR for over 50 languages, while
  CosyVoice excels in multi-lingual voice generation, zero-shot in-context
  learning, cross-lingual voice cloning, and instruction-following capabilities.
  The models related to SenseVoice and CosyVoice have been open-sourced on
  Modelscope and Huggingface, along with the corresponding training, inference,
  and fine-tuning codes released on GitHub. By integrating these models with
  LLMs, FunAudioLLM enables applications such as speech-to-speech translation,
  emotional voice chat, interactive podcasts, and expressive audiobook narration,
  thereby pushing the boundaries of voice interaction technology. Demos are
  available at https://fun-audio-llm.github.io, and the code can be accessed at
  https://github.com/FunAudioLLM.
paper_authors._id: 668b611fa9ee6373c751c9b6
paper_authors.name: Tongyi SpeechTeam
paper_title: FunAudioLLM: Voice Understanding and Generation Foundation Models for
    Natural Interaction Between Humans and LLMs
paper_summary: This report introduces FunAudioLLM, a model family designed to enhance
  natural voice interactions between humans and large language models (LLMs). At
  its core are two innovative models: SenseVoice, which handles multilingual
  speech recognition, emotion recognition, and audio event detection; and
  CosyVoice, which facilitates natural speech generation with control over
  multiple languages, timbre, speaking style, and speaker identity.
  SenseVoice-Small delivers exceptionally low-latency ASR for 5 languages, and
  SenseVoice-Large supports high-precision ASR for over 50 languages, while
  CosyVoice excels in multi-lingual voice generation, zero-shot in-context
  learning, cross-lingual voice cloning, and instruction-following capabilities.
  The models related to SenseVoice and CosyVoice have been open-sourced on
  Modelscope and Huggingface, along with the corresponding training, inference,
  and fine-tuning codes released on GitHub. By integrating these models with
  LLMs, FunAudioLLM enables applications such as speech-to-speech translation,
  emotional voice chat, interactive podcasts, and expressive audiobook narration,
  thereby pushing the boundaries of voice interaction technology. Demos are
  available at https://fun-audio-llm.github.io, and the code can be accessed at
  https://github.com/FunAudioLLM.
paper_ai_keywords: ASR (Automatic Speech Recognition) emotion recognition audio event detection multilingual speech recognition natural speech generation zero-shot in-context learning cross-lingual voice cloning parameter-following high-precision ASR speech-to-speech translation emotional voice chat interactive podcasts expressive audiobook narration voice interaction technology