Prompting Large Language Models with Speech Recognition Abilities
Large language models have proven themselves highly flexible, able to solve a
wide range of generative tasks, such as abstractive summarization and
open-ended question answering. In this paper we extend the capabilities of LLMs
by directly attaching a small audio encoder allowing it to perform speech
recognition. By directly prepending a sequence of audial embeddings to the text
token embeddings, the LLM can be converted to an automatic speech recognition
(ASR) system, and be used in the exact same manner as its textual counterpart.
Experiments on Multilingual LibriSpeech (MLS) show that incorporating a
conformer encoder into the open sourced LLaMA-7B allows it to outperform
monolingual baselines by 18% and perform multilingual speech recognition
despite LLaMA being trained overwhelmingly on English text. Furthermore, we
perform ablation studies to investigate whether the LLM can be completely
frozen during training to maintain its original capabilities, scaling up the
audio encoder, and increasing the audio encoder striding to generate fewer
embeddings. The results from these studies show that multilingual ASR is
possible even when the LLM is frozen or when strides of almost 1 second are
used in the audio encoder opening up the possibility for LLMs to operate on
long-form audio.
64bf817af671da974e74a317
Yassir Fathullah
admin_assigned
2023-07-25T08:42:58.849Z
64106057b27543634e3922bd
/avatars/c4eea7d419453802c67c4f7355315cac.svg
Yassir Fathullah
user
yassirf
64bf817af671da974e74a318
Chunyang Wu
admin_assigned
2023-07-25T08:43:28.503Z
641d8fb516ef0e5a7274cf1a
/avatars/97a8565e7518f2a9b9e58dd4b98207d9.svg
wuchunyang
user
Picc
64bf817af671da974e74a319
Egor Lakomkin
admin_assigned
2023-07-25T08:43:46.359Z
61e1bb4a210b4f412674d08d
/avatars/b0455785ca48ec93fc43a36dd4b5e417.svg
Egor Lakomkin
user
EgorL
64bf817af671da974e74a31a
Junteng Jia
admin_assigned
2023-07-25T08:44:06.812Z
6171c03ff37773b575f8b39f
/avatars/feb21515dfd111fe6a222469ea7d6950.svg
Junteng Jia
user
JuntengJia
64bf817af671da974e74a31b
Yuan Shangguan
64bf817af671da974e74a31c
Ke Li
64bf817af671da974e74a31d
Jinxi Guo
64bf817af671da974e74a31e
Wenhan Xiong
64bf817af671da974e74a31f
Jay Mahadeokar
64bf817af671da974e74a320
Ozlem Kalinli
64bf817af671da974e74a321
Christian Fuegen
64bf817af671da974e74a322
Mike Seltzer
Prompting Large Language Models with Speech Recognition Abilities
Large language models have proven themselves highly flexible, able to solve a
wide range of generative tasks, such as abstractive summarization and
open-ended question answering. In this paper we extend the capabilities of LLMs
by directly attaching a small audio encoder allowing it to perform speech
recognition. By directly prepending a sequence of audial embeddings to the text
token embeddings, the LLM can be converted to an automatic speech recognition
(ASR) system, and be used in the exact same manner as its textual counterpart.
Experiments on Multilingual LibriSpeech (MLS) show that incorporating a
conformer encoder into the open sourced LLaMA-7B allows it to outperform
monolingual baselines by 18% and perform multilingual speech recognition
despite LLaMA being trained overwhelmingly on English text. Furthermore, we
perform ablation studies to investigate whether the LLM can be completely
frozen during training to maintain its original capabilities, scaling up the
audio encoder, and increasing the audio encoder striding to generate fewer
embeddings. The results from these studies show that multilingual ASR is
possible even when the LLM is frozen or when strides of almost 1 second are
used in the audio encoder opening up the possibility for LLMs to operate on
long-form audio.
Attaching a conformer audio encoder to LLaMA-7B allows it to perform multilingual speech recognition, outperforming monolingual models and maintaining its original text capabilities, even with limited training and longer strides.
large language models speech recognition abstractive summarization open-ended question answering automatic speech recognition MLS conformer encoder LLaMA-7B multilingual ASR ablation studies audio encoder audio encoder striding