SpeechVerse: A Large-scale Generalizable Audio Language Model
Large language models (LLMs) have shown incredible proficiency in performing
tasks that require semantic understanding of natural language instructions.
Recently, many works have further expanded this capability to perceive
multimodal audio and text inputs, but their capabilities are often limited to
specific fine-tuned tasks such as automatic speech recognition and translation.
We therefore develop SpeechVerse, a robust multi-task training and curriculum
learning framework that combines pre-trained speech and text foundation models
via a small set of learnable parameters, while keeping the pre-trained models
frozen during training. The models are instruction finetuned using continuous
latent representations extracted from the speech foundation model to achieve
optimal zero-shot performance on a diverse range of speech processing tasks
using natural language instructions. We perform extensive benchmarking that
includes comparing our model performance against traditional baselines across
several datasets and tasks. Furthermore, we evaluate the model's capability for
generalized instruction following by testing on out-of-domain datasets, novel
prompts, and unseen tasks. Our empirical experiments reveal that our multi-task
SpeechVerse model is even superior to conventional task-specific baselines on 9
out of the 11 tasks.
6644f82fb037ed8abd89b72a
Nilaksh Das
6644f82fb037ed8abd89b72b
Saket Dingliwal
6644f82fb037ed8abd89b72c
Srikanth Ronanki
claimed_verified
2024-05-17T06:53:32.267Z
66467366a6d12d11aa1907b7
/avatars/58e8cfc785e94ea415575408a2e78f24.svg
Srikanth Ronanki
user
ronanki123
6644f82fb037ed8abd89b72d
Rohit Paturi
admin_assigned
2024-05-15T20:29:57.732Z
649d336965b8e583e1ba7b8e
/avatars/7e3850f98c07b081902ee84d73141aec.svg
Rohit Paturi
user
roitpaturi
6644f82fb037ed8abd89b72e
David Huang
6644f82fb037ed8abd89b72f
Prashant Mathur
admin_assigned
2024-05-15T20:30:46.235Z
64b559445154d0a506acab69
/avatars/502590d6ea2ae56f7dd43505914560a0.svg
Prashant
user
prashantmathur
6644f82fb037ed8abd89b730
Jie Yuan
6644f82fb037ed8abd89b731
Dhanush Bekal
admin_assigned
2024-05-15T20:31:18.256Z
63d22014b30415240fdcc74c
/avatars/82160f411ac320149dbab97489246737.svg
Dhanush Bekal
user
Ddhdddh
6644f82fb037ed8abd89b732
Xing Niu
admin_assigned
2024-05-15T20:31:27.154Z
63dbf06306e5ca38798438ed
/avatars/a2c608e4e7ded6d4a2fca9f7d4750ab7.svg
Xing Niu
user
xingniu
6644f82fb037ed8abd89b733
Sai Muralidhar Jayanthi
admin_assigned
2024-05-15T20:31:34.639Z
5faf4c4984389b139cf3b4e0
/avatars/0fa06885e2999fcab8bc833e2211779f.svg
Sai Muralidhar Jayanthi
user
murali1996
6644f82fb037ed8abd89b734
Xilai Li
admin_assigned
2024-05-15T20:31:42.147Z
64a7971b1e4dd9f35498426d
/avatars/c6fd98cabb6b60a66fec436008c49f00.svg
Xilai Li
user
YuzakiKokuban
6644f82fb037ed8abd89b735
Karel Mundnich
admin_assigned
2024-05-15T20:31:49.418Z
65b03e4dbf7d3327d066e7af
/avatars/ad60d2f207e80f742e5aaed93281b8a8.svg
Karel Mundnich
user
kmundnic
6644f82fb037ed8abd89b736
Monica Sunkara
6644f82fb037ed8abd89b737
Sundararajan Srinivasan
6644f82fb037ed8abd89b738
Kyu J Han
6644f82fb037ed8abd89b739
Katrin Kirchhoff
SpeechVerse: A Large-scale Generalizable Audio Language Model
Large language models (LLMs) have shown incredible proficiency in performing
tasks that require semantic understanding of natural language instructions.
Recently, many works have further expanded this capability to perceive
multimodal audio and text inputs, but their capabilities are often limited to
specific fine-tuned tasks such as automatic speech recognition and translation.
We therefore develop SpeechVerse, a robust multi-task training and curriculum
learning framework that combines pre-trained speech and text foundation models
via a small set of learnable parameters, while keeping the pre-trained models
frozen during training. The models are instruction finetuned using continuous
latent representations extracted from the speech foundation model to achieve
optimal zero-shot performance on a diverse range of speech processing tasks
using natural language instructions. We perform extensive benchmarking that
includes comparing our model performance against traditional baselines across
several datasets and tasks. Furthermore, we evaluate the model's capability for
generalized instruction following by testing on out-of-domain datasets, novel
prompts, and unseen tasks. Our empirical experiments reveal that our multi-task
SpeechVerse model is even superior to conventional task-specific baselines on 9
out of the 11 tasks.
SpeechVerse, a multi-task training framework, enhances performance on diverse speech processing tasks with natural language instructions by combining pre-trained speech and text models.
SpeechVerse multi-task training curriculum learning pre-trained speech models pre-trained text models learnable parameters instruction finetuning continuous latent representations zero-shot performance out-of-domain datasets generalized instruction following