title: Toward Joint Language Modeling for Speech Units and Text
summary: Speech and text are two major forms of human language. The research community
  has been focusing on mapping speech to text or vice versa for many years.
  However, in the field of language modeling, very little effort has been made to
  model them jointly. In light of this, we explore joint language modeling for
  speech units and text. Specifically, we compare different speech tokenizers to
  transform continuous speech signals into discrete units and use different
  methods to construct mixed speech-text data. We introduce automatic metrics to
  evaluate how well the joint LM mixes speech and text. We also fine-tune the LM
  on downstream spoken language understanding (SLU) tasks with different
  modalities (speech or text) and test its performance to assess the model's
  learning of shared representations. Our results show that by mixing speech
  units and text with our proposed mixing techniques, the joint LM improves over
  a speech-only baseline on SLU tasks and shows zero-shot cross-modal
  transferability.
paper_authors._id: 652cbb3d90e317f243af2d00
paper_authors.name: Ju-Chieh Chou
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-16T08:13:36.551Z
paper_authors._id: 652588df2ceb2f426c31c2e9
paper_authors.avatarUrl: /avatars/d9111152ad7c79c23798682c53facf47.svg
paper_authors.fullname: Ju-Chieh Chou
paper_authors.type: user
paper_authors.user: jjery2243542
paper_authors._id: 652cbb3d90e317f243af2d01
paper_authors.name: Chung-Ming Chien
paper_authors._id: 652cbb3d90e317f243af2d02
paper_authors.name: Wei-Ning Hsu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-16T08:14:09.565Z
paper_authors._id: 64c143d484887aa2b43bac0d
paper_authors.avatarUrl: /avatars/817dba24739676aa0f34692702bb00d6.svg
paper_authors.fullname: Wei-Ning Hsu
paper_authors.type: user
paper_authors.user: wnhsu
paper_authors._id: 652cbb3d90e317f243af2d03
paper_authors.name: Karen Livescu
paper_authors._id: 652cbb3d90e317f243af2d04
paper_authors.name: Arun Babu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-16T08:14:46.037Z
paper_authors._id: 61776ccc924026cc8c692630
paper_authors.avatarUrl: /avatars/44c669649402a52867861990e8b17d27.svg
paper_authors.fullname: Arun Babu
paper_authors.type: user
paper_authors.user: arbabu123
paper_authors._id: 652cbb3d90e317f243af2d05
paper_authors.name: Alexis Conneau
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-16T08:14:53.956Z
paper_authors._id: 61776a8f1e3e9ecbf77ddeea
paper_authors.avatarUrl: /avatars/f2b9215fa02178194d0cbf2819681824.svg
paper_authors.fullname: Alexis C
paper_authors.type: user
paper_authors.user: aconneau
paper_authors._id: 652cbb3d90e317f243af2d06
paper_authors.name: Alexei Baevski
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-16T08:15:01.514Z
paper_authors._id: 6048123112aa89499a978a21
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1615335958554-noauth.jpeg
paper_authors.fullname: Alexei Baevski
paper_authors.type: user
paper_authors.user: abaevski
paper_authors._id: 652cbb3d90e317f243af2d07
paper_authors.name: Michael Auli
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-16T08:15:11.962Z
paper_authors._id: 6176e9d475f6ae94d250042c
paper_authors.avatarUrl: /avatars/ce0eafa09525b741c22717bbde6c6a16.svg
paper_authors.fullname: Michael Auli
paper_authors.type: user
paper_authors.user: michaelauli
paper_title: Toward Joint Language Modeling for Speech Units and Text
paper_summary: Speech and text are two major forms of human language. The research community
  has been focusing on mapping speech to text or vice versa for many years.
  However, in the field of language modeling, very little effort has been made to
  model them jointly. In light of this, we explore joint language modeling for
  speech units and text. Specifically, we compare different speech tokenizers to
  transform continuous speech signals into discrete units and use different
  methods to construct mixed speech-text data. We introduce automatic metrics to
  evaluate how well the joint LM mixes speech and text. We also fine-tune the LM
  on downstream spoken language understanding (SLU) tasks with different
  modalities (speech or text) and test its performance to assess the model's
  learning of shared representations. Our results show that by mixing speech
  units and text with our proposed mixing techniques, the joint LM improves over
  a speech-only baseline on SLU tasks and shows zero-shot cross-modal
  transferability.
paper_ai_summary: A joint language model for speech and text improves performance on spoken language understanding tasks and demonstrates zero-shot cross-modal transferability.
paper_ai_keywords: speech tokenizers mixed speech-text data joint language modeling automatic metrics spoken language understanding (SLU) zero-shot cross-modal transferability