title: Textually Pretrained Speech Language Models
summary: Speech language models (SpeechLMs) process and generate acoustic data only,
  without textual supervision. In this work, we propose TWIST, a method for
  training SpeechLMs using a warm-start from a pretrained textual language
  models. We show using both automatic and human evaluations that TWIST
  outperforms a cold-start SpeechLM across the board. We empirically analyze the
  effect of different model design choices such as the speech tokenizer, the
  pretrained textual model, and the dataset size. We find that model and dataset
  scale both play an important role in constructing better-performing SpeechLMs.
  Based on our observations, we present the largest (to the best of our
  knowledge) SpeechLM both in terms of number of parameters and training data. We
  additionally introduce two spoken versions of the StoryCloze textual benchmark
  to further improve model evaluation and advance future research in the field.
  Speech samples can be found on our website:
  https://pages.cs.huji.ac.il/adiyoss-lab/twist/ .
paper_authors._id: 646c282eb1202bc77c149334
paper_authors.name: Michael Hassid
paper_authors._id: 646c282eb1202bc77c149335
paper_authors.name: Tal Remez
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-06-08T16:08:46.167Z
paper_authors._id: 647f1b66838ac3601fc762d1
paper_authors.avatarUrl: /avatars/24aaaeb700690bc84ad0212ce4ae9bd4.svg
paper_authors.fullname: Tal Remez
paper_authors.type: user
paper_authors.user: TalRemez
paper_authors._id: 646c282eb1202bc77c149336
paper_authors.name: Tu Anh Nguyen
paper_authors._id: 646c282eb1202bc77c149337
paper_authors.name: Itai Gat
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-23T08:11:22.030Z
paper_authors._id: 62a8fa984d933c74bf410c16
paper_authors.avatarUrl: /avatars/73519deba3176be9c23d49f749aee5da.svg
paper_authors.fullname: Itai Gat
paper_authors.type: user
paper_authors.user: itaigat
paper_authors._id: 646c282eb1202bc77c149338
paper_authors.name: Alexis Conneau
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-23T08:47:24.758Z
paper_authors._id: 61776a8f1e3e9ecbf77ddeea
paper_authors.avatarUrl: /avatars/f2b9215fa02178194d0cbf2819681824.svg
paper_authors.fullname: Alexis C
paper_authors.type: user
paper_authors.user: aconneau
paper_authors._id: 646c282eb1202bc77c149339
paper_authors.name: Felix Kreuk
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-05-23T09:42:05.792Z
paper_authors._id: 645d0a4cf4f49de5809de159
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/IOWMd17Iwls0dXsY1OWjK.jpeg
paper_authors.fullname: Felix Kreuk
paper_authors.type: user
paper_authors.user: felixkreuk
paper_authors._id: 646c282eb1202bc77c14933a
paper_authors.name: Jade Copet
paper_authors._id: 646c282eb1202bc77c14933b
paper_authors.name: Alexandre Defossez
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-23T08:59:13.900Z
paper_authors._id: 6357f5f50dce5559211a3503
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1666708948380-noauth.jpeg
paper_authors.fullname: Alexandre Défossez
paper_authors.type: user
paper_authors.user: adefossez
paper_authors._id: 646c282eb1202bc77c14933c
paper_authors.name: Gabriel Synnaeve
paper_authors._id: 646c282eb1202bc77c14933d
paper_authors.name: Emmanuel Dupoux
paper_authors._id: 646c282eb1202bc77c14933e
paper_authors.name: Roy Schwartz
paper_authors._id: 646c282eb1202bc77c14933f
paper_authors.name: Yossi Adi
paper_title: Textually Pretrained Speech Language Models
paper_summary: Speech language models (SpeechLMs) process and generate acoustic data only,
  without textual supervision. In this work, we propose TWIST, a method for
  training SpeechLMs using a warm-start from a pretrained textual language
  models. We show using both automatic and human evaluations that TWIST
  outperforms a cold-start SpeechLM across the board. We empirically analyze the
  effect of different model design choices such as the speech tokenizer, the
  pretrained textual model, and the dataset size. We find that model and dataset
  scale both play an important role in constructing better-performing SpeechLMs.
  Based on our observations, we present the largest (to the best of our
  knowledge) SpeechLM both in terms of number of parameters and training data. We
  additionally introduce two spoken versions of the StoryCloze textual benchmark
  to further improve model evaluation and advance future research in the field.
  Speech samples can be found on our website:
  https://pages.cs.huji.ac.il/adiyoss-lab/twist/ .
paper_ai_keywords: SpeechLMs (Speech language models) warm-start pretrained textual models cold-start model specialization speech tokenizer dataset size Discrimin choices parameter-efficient fine-tuning Storycloze spoken versions