title: SpeechVerse: A Large-scale Generalizable Audio Language Model
summary: Large language models (LLMs) have shown incredible proficiency in performing
  tasks that require semantic understanding of natural language instructions.
  Recently, many works have further expanded this capability to perceive
  multimodal audio and text inputs, but their capabilities are often limited to
  specific fine-tuned tasks such as automatic speech recognition and translation.
  We therefore develop SpeechVerse, a robust multi-task training and curriculum
  learning framework that combines pre-trained speech and text foundation models
  via a small set of learnable parameters, while keeping the pre-trained models
  frozen during training. The models are instruction finetuned using continuous
  latent representations extracted from the speech foundation model to achieve
  optimal zero-shot performance on a diverse range of speech processing tasks
  using natural language instructions. We perform extensive benchmarking that
  includes comparing our model performance against traditional baselines across
  several datasets and tasks. Furthermore, we evaluate the model's capability for
  generalized instruction following by testing on out-of-domain datasets, novel
  prompts, and unseen tasks. Our empirical experiments reveal that our multi-task
  SpeechVerse model is even superior to conventional task-specific baselines on 9
  out of the 11 tasks.
paper_authors._id: 6644f82fb037ed8abd89b72a
paper_authors.name: Nilaksh Das
paper_authors._id: 6644f82fb037ed8abd89b72b
paper_authors.name: Saket Dingliwal
paper_authors._id: 6644f82fb037ed8abd89b72c
paper_authors.name: Srikanth Ronanki
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-05-17T06:53:32.267Z
paper_authors._id: 66467366a6d12d11aa1907b7
paper_authors.avatarUrl: /avatars/58e8cfc785e94ea415575408a2e78f24.svg
paper_authors.fullname: Srikanth Ronanki
paper_authors.type: user
paper_authors.user: ronanki123
paper_authors._id: 6644f82fb037ed8abd89b72d
paper_authors.name: Rohit Paturi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-15T20:29:57.732Z
paper_authors._id: 649d336965b8e583e1ba7b8e
paper_authors.avatarUrl: /avatars/7e3850f98c07b081902ee84d73141aec.svg
paper_authors.fullname: Rohit Paturi
paper_authors.type: user
paper_authors.user: roitpaturi
paper_authors._id: 6644f82fb037ed8abd89b72e
paper_authors.name: David Huang
paper_authors._id: 6644f82fb037ed8abd89b72f
paper_authors.name: Prashant Mathur
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-15T20:30:46.235Z
paper_authors._id: 64b559445154d0a506acab69
paper_authors.avatarUrl: /avatars/502590d6ea2ae56f7dd43505914560a0.svg
paper_authors.fullname: Prashant
paper_authors.type: user
paper_authors.user: prashantmathur
paper_authors._id: 6644f82fb037ed8abd89b730
paper_authors.name: Jie Yuan
paper_authors._id: 6644f82fb037ed8abd89b731
paper_authors.name: Dhanush Bekal
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-15T20:31:18.256Z
paper_authors._id: 63d22014b30415240fdcc74c
paper_authors.avatarUrl: /avatars/82160f411ac320149dbab97489246737.svg
paper_authors.fullname: Dhanush Bekal
paper_authors.type: user
paper_authors.user: Ddhdddh
paper_authors._id: 6644f82fb037ed8abd89b732
paper_authors.name: Xing Niu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-15T20:31:27.154Z
paper_authors._id: 63dbf06306e5ca38798438ed
paper_authors.avatarUrl: /avatars/a2c608e4e7ded6d4a2fca9f7d4750ab7.svg
paper_authors.fullname: Xing Niu
paper_authors.type: user
paper_authors.user: xingniu
paper_authors._id: 6644f82fb037ed8abd89b733
paper_authors.name: Sai Muralidhar Jayanthi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-15T20:31:34.639Z
paper_authors._id: 5faf4c4984389b139cf3b4e0
paper_authors.avatarUrl: /avatars/0fa06885e2999fcab8bc833e2211779f.svg
paper_authors.fullname: Sai Muralidhar Jayanthi
paper_authors.type: user
paper_authors.user: murali1996
paper_authors._id: 6644f82fb037ed8abd89b734
paper_authors.name: Xilai Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-15T20:31:42.147Z
paper_authors._id: 64a7971b1e4dd9f35498426d
paper_authors.avatarUrl: /avatars/c6fd98cabb6b60a66fec436008c49f00.svg
paper_authors.fullname: Xilai Li
paper_authors.type: user
paper_authors.user: YuzakiKokuban
paper_authors._id: 6644f82fb037ed8abd89b735
paper_authors.name: Karel Mundnich
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-15T20:31:49.418Z
paper_authors._id: 65b03e4dbf7d3327d066e7af
paper_authors.avatarUrl: /avatars/ad60d2f207e80f742e5aaed93281b8a8.svg
paper_authors.fullname: Karel Mundnich
paper_authors.type: user
paper_authors.user: kmundnic
paper_authors._id: 6644f82fb037ed8abd89b736
paper_authors.name: Monica Sunkara
paper_authors._id: 6644f82fb037ed8abd89b737
paper_authors.name: Sundararajan Srinivasan
paper_authors._id: 6644f82fb037ed8abd89b738
paper_authors.name: Kyu J Han
paper_authors._id: 6644f82fb037ed8abd89b739
paper_authors.name: Katrin Kirchhoff
paper_title: SpeechVerse: A Large-scale Generalizable Audio Language Model
paper_summary: Large language models (LLMs) have shown incredible proficiency in performing
  tasks that require semantic understanding of natural language instructions.
  Recently, many works have further expanded this capability to perceive
  multimodal audio and text inputs, but their capabilities are often limited to
  specific fine-tuned tasks such as automatic speech recognition and translation.
  We therefore develop SpeechVerse, a robust multi-task training and curriculum
  learning framework that combines pre-trained speech and text foundation models
  via a small set of learnable parameters, while keeping the pre-trained models
  frozen during training. The models are instruction finetuned using continuous
  latent representations extracted from the speech foundation model to achieve
  optimal zero-shot performance on a diverse range of speech processing tasks
  using natural language instructions. We perform extensive benchmarking that
  includes comparing our model performance against traditional baselines across
  several datasets and tasks. Furthermore, we evaluate the model's capability for
  generalized instruction following by testing on out-of-domain datasets, novel
  prompts, and unseen tasks. Our empirical experiments reveal that our multi-task
  SpeechVerse model is even superior to conventional task-specific baselines on 9
  out of the 11 tasks.
paper_ai_summary: SpeechVerse, a multi-task training framework, enhances performance on diverse speech processing tasks with natural language instructions by combining pre-trained speech and text models.
paper_ai_keywords: SpeechVerse multi-task training curriculum learning pre-trained speech models pre-trained text models learnable parameters instruction finetuning continuous latent representations zero-shot performance out-of-domain datasets generalized instruction following