title: AudioPaLM: A Large Language Model That Can Speak and Listen
summary: We introduce AudioPaLM, a large language model for speech understanding and
  generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2
  [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified
  multimodal architecture that can process and generate text and speech with
  applications including speech recognition and speech-to-speech translation.
  AudioPaLM inherits the capability to preserve paralinguistic information such
  as speaker identity and intonation from AudioLM and the linguistic knowledge
  present only in text large language models such as PaLM-2. We demonstrate that
  initializing AudioPaLM with the weights of a text-only large language model
  improves speech processing, successfully leveraging the larger quantity of text
  training data used in pretraining to assist with the speech tasks. The
  resulting model significantly outperforms existing systems for speech
  translation tasks and has the ability to perform zero-shot speech-to-text
  translation for many languages for which input/target language combinations
  were not seen in training. AudioPaLM also demonstrates features of audio
  language models, such as transferring a voice across languages based on a short
  spoken prompt. We release examples of our method at
  https://google-research.github.io/seanet/audiopalm/examples
paper_authors._id: 6494ef61771d68df8d060909
paper_authors.name: Paul K. Rubenstein
paper_authors._id: 6494ef61771d68df8d06090a
paper_authors.name: Chulayuth Asawaroengchai
paper_authors._id: 6494ef61771d68df8d06090b
paper_authors.name: Duc Dung Nguyen
paper_authors._id: 6494ef61771d68df8d06090c
paper_authors.name: Ankur Bapna
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-23T12:09:54.990Z
paper_authors._id: 62cdba279403ec4719d384f8
paper_authors.avatarUrl: /avatars/c62c7d16b5cf382cdeb7fbfb5e291917.svg
paper_authors.fullname: Ankur Bapna
paper_authors.type: user
paper_authors.user: ankurbpn
paper_authors._id: 6494ef61771d68df8d06090d
paper_authors.name: Zalán Borsos
paper_authors._id: 6494ef61771d68df8d06090e
paper_authors.name: Félix de Chaumont Quitry
paper_authors._id: 6494ef61771d68df8d06090f
paper_authors.name: Peter Chen
paper_authors._id: 6494ef61771d68df8d060910
paper_authors.name: Dalia El Badawy
paper_authors._id: 6494ef61771d68df8d060911
paper_authors.name: Wei Han
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-23T07:31:27.769Z
paper_authors._id: 6489e3992f9e11a0c26973b6
paper_authors.avatarUrl: /avatars/e637e6caab6df56da6005f73c613a3d0.svg
paper_authors.fullname: Wei Han
paper_authors.type: user
paper_authors.user: weihan3
paper_authors._id: 6494ef61771d68df8d060912
paper_authors.name: Eugene Kharitonov
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-23T12:11:20.840Z
paper_authors._id: 64509292fc5e5ff919b4a97d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64509292fc5e5ff919b4a97d/OOn5EaCXw39ePmgKptOuT.png
paper_authors.fullname: Eugene Kharitonov
paper_authors.type: user
paper_authors.user: ekharitonov
paper_authors._id: 6494ef61771d68df8d060913
paper_authors.name: Hannah Muckenhirn
paper_authors._id: 6494ef61771d68df8d060914
paper_authors.name: Dirk Padfield
paper_authors._id: 6494ef61771d68df8d060915
paper_authors.name: James Qin
paper_authors._id: 6494ef61771d68df8d060916
paper_authors.name: Danny Rozenberg
paper_authors._id: 6494ef61771d68df8d060917
paper_authors.name: Tara Sainath
paper_authors._id: 6494ef61771d68df8d060918
paper_authors.name: Johan Schalkwyk
paper_authors._id: 6494ef61771d68df8d060919
paper_authors.name: Matt Sharifi
paper_authors._id: 6494ef61771d68df8d06091a
paper_authors.name: Michelle Tadmor Ramanovich
paper_authors._id: 6494ef61771d68df8d06091c
paper_authors.name: Marco Tagliasacchi
paper_authors._id: 6494ef61771d68df8d06091d
paper_authors.name: Alexandru Tudor
paper_authors._id: 6494ef61771d68df8d06091e
paper_authors.name: Mihajlo Velimirović
paper_authors._id: 6494ef61771d68df8d06091f
paper_authors.name: Damien Vincent
paper_authors._id: 6494ef61771d68df8d060920
paper_authors.name: Jiahui Yu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-06-23T12:18:09.782Z
paper_authors._id: 6174394801833e378dfc2517
paper_authors.avatarUrl: /avatars/4989bcdda7ba1ae51cab8235c3e39275.svg
paper_authors.fullname: Jiahui Yu
paper_authors.type: user
paper_authors.user: jiahuiyu
paper_authors._id: 6494ef61771d68df8d060921
paper_authors.name: Yongqiang Wang
paper_authors._id: 6494ef61771d68df8d060922
paper_authors.name: Vicky Zayats
paper_authors._id: 6494ef61771d68df8d060923
paper_authors.name: Neil Zeghidour
paper_authors._id: 6494ef61771d68df8d060924
paper_authors.name: Yu Zhang
paper_authors._id: 6494ef61771d68df8d060925
paper_authors.name: Zhishuai Zhang
paper_authors._id: 6494ef61771d68df8d060926
paper_authors.name: Lukas Zilka
paper_authors._id: 6494ef61771d68df8d060927
paper_authors.name: Christian Frank
paper_title: AudioPaLM: A Large Language Model That Can Speak and Listen
paper_summary: We introduce AudioPaLM, a large language model for speech understanding and
  generation. AudioPaLM fuses text-based and speech-based language models, PaLM-2
  [Anil et al., 2023] and AudioLM [Borsos et al., 2022], into a unified
  multimodal architecture that can process and generate text and speech with
  applications including speech recognition and speech-to-speech translation.
  AudioPaLM inherits the capability to preserve paralinguistic information such
  as speaker identity and intonation from AudioLM and the linguistic knowledge
  present only in text large language models such as PaLM-2. We demonstrate that
  initializing AudioPaLM with the weights of a text-only large language model
  improves speech processing, successfully leveraging the larger quantity of text
  training data used in pretraining to assist with the speech tasks. The
  resulting model significantly outperforms existing systems for speech
  translation tasks and has the ability to perform zero-shot speech-to-text
  translation for many languages for which input/target language combinations
  were not seen in training. AudioPaLM also demonstrates features of audio
  language models, such as transferring a voice across languages based on a short
  spoken prompt. We release examples of our method at
  https://google-research.github.io/seanet/audiopalm/examples
paper_ai_keywords: AudioPaLM large language model speech understanding speech generation PaLM-2 AudioLM multimodal architecture paralinguistic information speaker identity intonation speech recognition speech-to-speech translation zero-shot speech-to-text translation audio language models voice transfer short spoken prompt