title: Prompting Large Language Models with Speech Recognition Abilities
summary: Large language models have proven themselves highly flexible, able to solve a
  wide range of generative tasks, such as abstractive summarization and
  open-ended question answering. In this paper we extend the capabilities of LLMs
  by directly attaching a small audio encoder allowing it to perform speech
  recognition. By directly prepending a sequence of audial embeddings to the text
  token embeddings, the LLM can be converted to an automatic speech recognition
  (ASR) system, and be used in the exact same manner as its textual counterpart.
  Experiments on Multilingual LibriSpeech (MLS) show that incorporating a
  conformer encoder into the open sourced LLaMA-7B allows it to outperform
  monolingual baselines by 18% and perform multilingual speech recognition
  despite LLaMA being trained overwhelmingly on English text. Furthermore, we
  perform ablation studies to investigate whether the LLM can be completely
  frozen during training to maintain its original capabilities, scaling up the
  audio encoder, and increasing the audio encoder striding to generate fewer
  embeddings. The results from these studies show that multilingual ASR is
  possible even when the LLM is frozen or when strides of almost 1 second are
  used in the audio encoder opening up the possibility for LLMs to operate on
  long-form audio.
paper_authors._id: 64bf817af671da974e74a317
paper_authors.name: Yassir Fathullah
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-25T08:42:58.849Z
paper_authors._id: 64106057b27543634e3922bd
paper_authors.avatarUrl: /avatars/c4eea7d419453802c67c4f7355315cac.svg
paper_authors.fullname: Yassir Fathullah
paper_authors.type: user
paper_authors.user: yassirf
paper_authors._id: 64bf817af671da974e74a318
paper_authors.name: Chunyang Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-25T08:43:28.503Z
paper_authors._id: 641d8fb516ef0e5a7274cf1a
paper_authors.avatarUrl: /avatars/97a8565e7518f2a9b9e58dd4b98207d9.svg
paper_authors.fullname: wuchunyang
paper_authors.type: user
paper_authors.user: Picc
paper_authors._id: 64bf817af671da974e74a319
paper_authors.name: Egor Lakomkin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-25T08:43:46.359Z
paper_authors._id: 61e1bb4a210b4f412674d08d
paper_authors.avatarUrl: /avatars/b0455785ca48ec93fc43a36dd4b5e417.svg
paper_authors.fullname: Egor Lakomkin
paper_authors.type: user
paper_authors.user: EgorL
paper_authors._id: 64bf817af671da974e74a31a
paper_authors.name: Junteng Jia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-25T08:44:06.812Z
paper_authors._id: 6171c03ff37773b575f8b39f
paper_authors.avatarUrl: /avatars/feb21515dfd111fe6a222469ea7d6950.svg
paper_authors.fullname: Junteng Jia
paper_authors.type: user
paper_authors.user: JuntengJia
paper_authors._id: 64bf817af671da974e74a31b
paper_authors.name: Yuan Shangguan
paper_authors._id: 64bf817af671da974e74a31c
paper_authors.name: Ke Li
paper_authors._id: 64bf817af671da974e74a31d
paper_authors.name: Jinxi Guo
paper_authors._id: 64bf817af671da974e74a31e
paper_authors.name: Wenhan Xiong
paper_authors._id: 64bf817af671da974e74a31f
paper_authors.name: Jay Mahadeokar
paper_authors._id: 64bf817af671da974e74a320
paper_authors.name: Ozlem Kalinli
paper_authors._id: 64bf817af671da974e74a321
paper_authors.name: Christian Fuegen
paper_authors._id: 64bf817af671da974e74a322
paper_authors.name: Mike Seltzer
paper_title: Prompting Large Language Models with Speech Recognition Abilities
paper_summary: Large language models have proven themselves highly flexible, able to solve a
  wide range of generative tasks, such as abstractive summarization and
  open-ended question answering. In this paper we extend the capabilities of LLMs
  by directly attaching a small audio encoder allowing it to perform speech
  recognition. By directly prepending a sequence of audial embeddings to the text
  token embeddings, the LLM can be converted to an automatic speech recognition
  (ASR) system, and be used in the exact same manner as its textual counterpart.
  Experiments on Multilingual LibriSpeech (MLS) show that incorporating a
  conformer encoder into the open sourced LLaMA-7B allows it to outperform
  monolingual baselines by 18% and perform multilingual speech recognition
  despite LLaMA being trained overwhelmingly on English text. Furthermore, we
  perform ablation studies to investigate whether the LLM can be completely
  frozen during training to maintain its original capabilities, scaling up the
  audio encoder, and increasing the audio encoder striding to generate fewer
  embeddings. The results from these studies show that multilingual ASR is
  possible even when the LLM is frozen or when strides of almost 1 second are
  used in the audio encoder opening up the possibility for LLMs to operate on
  long-form audio.
paper_ai_keywords: audio encoder audial embeddings text token embeddings automatic speech recognition (ASR) Multilingual LibriSpeech (MLS) conformer encoder LLaMA-7B monolingual baselines ablation studies frozen audio encoder striding