title: WavLLM: Towards Robust and Adaptive Speech Large Language Model
summary: The recent advancements in large language models (LLMs) have revolutionized
  the field of natural language processing, progressively broadening their scope
  to multimodal perception and generation. However, effectively integrating
  listening capabilities into LLMs poses significant challenges, particularly
  with respect to generalizing across varied contexts and executing complex
  auditory tasks. In this work, we introduce WavLLM, a robust and adaptive speech
  large language model with dual encoders, and a prompt-aware LoRA weight
  adapter, optimized by a two-stage curriculum learning approach. Leveraging dual
  encoders, we decouple different types of speech information, utilizing a
  Whisper encoder to process the semantic content of speech, and a WavLM encoder
  to capture the unique characteristics of the speaker's identity. Within the
  curriculum learning framework, WavLLM first builds its foundational
  capabilities by optimizing on mixed elementary single tasks, followed by
  advanced multi-task training on more complex tasks such as combinations of the
  elementary tasks. To enhance the flexibility and adherence to different tasks
  and instructions, a prompt-aware LoRA weight adapter is introduced in the
  second advanced multi-task training stage. We validate the proposed model on
  universal speech benchmarks including tasks such as ASR, ST, SV, ER, and also
  apply it to specialized datasets like Gaokao English listening comprehension
  set for SQA, and speech Chain-of-Thought (CoT) evaluation set. Experiments
  demonstrate that the proposed model achieves state-of-the-art performance
  across a range of speech tasks on the same model size, exhibiting robust
  generalization capabilities in executing complex tasks using CoT approach.
  Furthermore, our model successfully completes Gaokao tasks without specialized
  training. The codes, models, audio, and Gaokao evaluation set can be accessed
  at aka.ms/wavllm.
paper_authors._id: 660bd1a1c518ed66653f4f64
paper_authors.name: Shujie Hu
paper_authors._id: 660bd1a1c518ed66653f4f65
paper_authors.name: Long Zhou
paper_authors._id: 660bd1a1c518ed66653f4f66
paper_authors.name: Shujie Liu
paper_authors._id: 660bd1a1c518ed66653f4f67
paper_authors.name: Sanyuan Chen
paper_authors._id: 660bd1a1c518ed66653f4f68
paper_authors.name: Hongkun Hao
paper_authors._id: 660bd1a1c518ed66653f4f69
paper_authors.name: Jing Pan
paper_authors._id: 660bd1a1c518ed66653f4f6a
paper_authors.name: Xunying Liu
paper_authors._id: 660bd1a1c518ed66653f4f6b
paper_authors.name: Jinyu Li
paper_authors._id: 660bd1a1c518ed66653f4f6c
paper_authors.name: Sunit Sivasankaran
paper_authors._id: 660bd1a1c518ed66653f4f6d
paper_authors.name: Linquan Liu
paper_authors._id: 660bd1a1c518ed66653f4f6e
paper_authors.name: Furu Wei
paper_title: WavLLM: Towards Robust and Adaptive Speech Large Language Model
paper_summary: The recent advancements in large language models (LLMs) have revolutionized
  the field of natural language processing, progressively broadening their scope
  to multimodal perception and generation. However, effectively integrating
  listening capabilities into LLMs poses significant challenges, particularly
  with respect to generalizing across varied contexts and executing complex
  auditory tasks. In this work, we introduce WavLLM, a robust and adaptive speech
  large language model with dual encoders, and a prompt-aware LoRA weight
  adapter, optimized by a two-stage curriculum learning approach. Leveraging dual
  encoders, we decouple different types of speech information, utilizing a
  Whisper encoder to process the semantic content of speech, and a WavLM encoder
  to capture the unique characteristics of the speaker's identity. Within the
  curriculum learning framework, WavLLM first builds its foundational
  capabilities by optimizing on mixed elementary single tasks, followed by
  advanced multi-task training on more complex tasks such as combinations of the
  elementary tasks. To enhance the flexibility and adherence to different tasks
  and instructions, a prompt-aware LoRA weight adapter is introduced in the
  second advanced multi-task training stage. We validate the proposed model on
  universal speech benchmarks including tasks such as ASR, ST, SV, ER, and also
  apply it to specialized datasets like Gaokao English listening comprehension
  set for SQA, and speech Chain-of-Thought (CoT) evaluation set. Experiments
  demonstrate that the proposed model achieves state-of-the-art performance
  across a range of speech tasks on the same model size, exhibiting robust
  generalization capabilities in executing complex tasks using CoT approach.
  Furthermore, our model successfully completes Gaokao tasks without specialized
  training. The codes, models, audio, and Gaokao evaluation set can be accessed
  at aka.ms/wavllm.
paper_ai_summary: WavLLM, a speech LLM with dual encoders and prompt-aware LoRA weight adapter, achieves state-of-the-art performance across various speech tasks using a two-stage curriculum learning approach.
paper_ai_keywords: LLMs multimodal perception speech large language model dual encoders Whisper encoder WavLM encoder curriculum learning prompt-aware LoRA weight adapter ASR ST SV ER SQA speech Chain-of-Thought