title: Towards General-Purpose Speech Abilities for Large Language Models Using
    Unpaired Data
summary: In this work, we extend the instruction-tuned Llama-2 model with end-to-end
  general-purpose speech processing and reasoning abilities while maintaining the
  wide range of LLM capabilities, without using any carefully curated paired
  data. The proposed model can utilize audio prompts as a replacement for text
  and sustain a conversation. Such a model also has extended cross-modal
  capabilities such as being able to perform speech question answering, speech
  translation, and audio summarization amongst many other closed and open-domain
  tasks. This is unlike prior approaches in speech, in which LLMs are extended to
  handle audio for a limited number of pre-designated tasks. Experiments show
  that our end-to-end approach is on par with or outperforms a cascaded system
  (speech recognizer + LLM) in terms of modeling the response to a prompt.
  Furthermore, unlike a cascade, our approach shows the ability to interchange
  text and audio modalities and utilize the prior context in a conversation to
  provide better results.
paper_authors._id: 6552edb8f2a8f0b4c6cc0c8b
paper_authors.name: Yassir Fathullah
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T10:52:51.763Z
paper_authors._id: 64106057b27543634e3922bd
paper_authors.avatarUrl: /avatars/c4eea7d419453802c67c4f7355315cac.svg
paper_authors.fullname: Yassir Fathullah
paper_authors.type: user
paper_authors.user: yassirf
paper_authors._id: 6552edb8f2a8f0b4c6cc0c8c
paper_authors.name: Chunyang Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T10:53:03.412Z
paper_authors._id: 641d8fb516ef0e5a7274cf1a
paper_authors.avatarUrl: /avatars/97a8565e7518f2a9b9e58dd4b98207d9.svg
paper_authors.fullname: wuchunyang
paper_authors.type: user
paper_authors.user: Picc
paper_authors._id: 6552edb8f2a8f0b4c6cc0c8d
paper_authors.name: Egor Lakomkin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T10:53:10.687Z
paper_authors._id: 61e1bb4a210b4f412674d08d
paper_authors.avatarUrl: /avatars/b0455785ca48ec93fc43a36dd4b5e417.svg
paper_authors.fullname: Egor Lakomkin
paper_authors.type: user
paper_authors.user: EgorL
paper_authors._id: 6552edb8f2a8f0b4c6cc0c8e
paper_authors.name: Junteng Jia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-11-14T10:53:23.148Z
paper_authors._id: 6171c03ff37773b575f8b39f
paper_authors.avatarUrl: /avatars/feb21515dfd111fe6a222469ea7d6950.svg
paper_authors.fullname: Junteng Jia
paper_authors.type: user
paper_authors.user: JuntengJia
paper_authors._id: 6552edb8f2a8f0b4c6cc0c8f
paper_authors.name: Yuan Shangguan
paper_authors._id: 6552edb8f2a8f0b4c6cc0c90
paper_authors.name: Jay Mahadeokar
paper_authors._id: 6552edb8f2a8f0b4c6cc0c91
paper_authors.name: Ozlem Kalinli
paper_authors._id: 6552edb8f2a8f0b4c6cc0c92
paper_authors.name: Christian Fuegen
paper_authors._id: 6552edb8f2a8f0b4c6cc0c93
paper_authors.name: Mike Seltzer
paper_title: Towards General-Purpose Speech Abilities for Large Language Models Using
    Unpaired Data
paper_summary: In this work, we extend the instruction-tuned Llama-2 model with end-to-end
  general-purpose speech processing and reasoning abilities while maintaining the
  wide range of LLM capabilities, without using any carefully curated paired
  data. The proposed model can utilize audio prompts as a replacement for text
  and sustain a conversation. Such a model also has extended cross-modal
  capabilities such as being able to perform speech question answering, speech
  translation, and audio summarization amongst many other closed and open-domain
  tasks. This is unlike prior approaches in speech, in which LLMs are extended to
  handle audio for a limited number of pre-designated tasks. Experiments show
  that our end-to-end approach is on par with or outperforms a cascaded system
  (speech recognizer + LLM) in terms of modeling the response to a prompt.
  Furthermore, unlike a cascade, our approach shows the ability to interchange
  text and audio modalities and utilize the prior context in a conversation to
  provide better results.
paper_ai_summary: An end-to-end extended Llama-2 model incorporating speech processing and reasoning supports a variety of tasks with audio and text, matching or outperforming cascaded systems in response modeling and context utilization.
paper_ai_keywords: instruction-tuned Llama-2 end-to-end speech processing reasoning abilities LLM capabilities audio prompts conversation speech question answering speech translation audio summarization cross-modal capabilities cascaded system speech recognizer text and audio modalities prior context