title: Scaling Analysis of Interleaved Speech-Text Language Models
summary: Existing Speech Language Model (SLM) scaling analysis paints a bleak picture.
  They predict that SLMs require much more compute and data compared to text,
  leading some to question the feasibility of training high-quality SLMs.
  However, modern SLMs are often initialised from pre-trained TextLMs using
  speech-text interleaving to allow knowledge transfer. This raises the question
  - Do interleaved SLMs scale more efficiently than textless-SLMs? In this paper
  we answer a resounding, yes! We conduct scaling analysis of interleaved SLMs by
  training several dozen and analysing the scaling trends. We see that under this
  setup SLMs scale more efficiently with compute. Additionally, our results
  indicate that the scaling-dynamics are significantly different than
  textless-SLMs, suggesting one should allocate notably more of the compute
  budget for increasing model size over training tokens. We also study the role
  of synthetic data and TextLM model families in unlocking this potential.
  Results suggest, that our scaled up model achieves comparable performance with
  leading models on speech semantic metrics while using less compute and data
  than other approaches. We open source models, samples, and data -
  https://pages.cs.huji.ac.il/adiyoss-lab/sims.
paper_authors._id: 67ef63b5e8b932ae7a8d3043
paper_authors.name: Gallil Maimon
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-04T07:08:06.509Z
paper_authors._id: 66b9bc2dacdbc1d0b39c3b50
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/hwR0pVfP_E8XjimXIxDOU.jpeg
paper_authors.fullname: Gallil Maimon
paper_authors.type: user
paper_authors.user: gallilmaimon
paper_authors._id: 67ef63b5e8b932ae7a8d3044
paper_authors.name: Michael Hassid
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-04T07:24:39.934Z
paper_authors._id: 6547411a9295970f878aa52e
paper_authors.avatarUrl: /avatars/6e240f0add27bf1a6c04a9618eccdf83.svg
paper_authors.fullname: Michael Hassid
paper_authors.type: user
paper_authors.user: hassid
paper_authors._id: 67ef63b5e8b932ae7a8d3045
paper_authors.name: Amit Roth
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-04T07:24:48.483Z
paper_authors._id: 64b7b7b38ba7d6c922d753d6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64b7b7b38ba7d6c922d753d6/rt0thjYa84VZHy1BEcW4p.jpeg
paper_authors.fullname: Amit Roth
paper_authors.type: user
paper_authors.user: MajoRoth
paper_authors._id: 67ef63b5e8b932ae7a8d3046
paper_authors.name: Yossi Adi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-04T07:24:54.851Z
paper_authors._id: 6481e135578646b5c2386728
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6481e135578646b5c2386728/SPva4iNw0pORiCXD45cx9.jpeg
paper_authors.fullname: Yossi Adi
paper_authors.type: user
paper_authors.user: adiyoss
paper_title: Scaling Analysis of Interleaved Speech-Text Language Models
paper_summary: Existing Speech Language Model (SLM) scaling analysis paints a bleak picture.
  They predict that SLMs require much more compute and data compared to text,
  leading some to question the feasibility of training high-quality SLMs.
  However, modern SLMs are often initialised from pre-trained TextLMs using
  speech-text interleaving to allow knowledge transfer. This raises the question
  - Do interleaved SLMs scale more efficiently than textless-SLMs? In this paper
  we answer a resounding, yes! We conduct scaling analysis of interleaved SLMs by
  training several dozen and analysing the scaling trends. We see that under this
  setup SLMs scale more efficiently with compute. Additionally, our results
  indicate that the scaling-dynamics are significantly different than
  textless-SLMs, suggesting one should allocate notably more of the compute
  budget for increasing model size over training tokens. We also study the role
  of synthetic data and TextLM model families in unlocking this potential.
  Results suggest, that our scaled up model achieves comparable performance with
  leading models on speech semantic metrics while using less compute and data
  than other approaches. We open source models, samples, and data -
  https://pages.cs.huji.ac.il/adiyoss-lab/sims.
paper_ai_summary: Interleaved Speech Language Models scale more efficiently than textless models, achieving comparable performance with less compute and data.
paper_ai_keywords: Speech Language Model SLM TextLM speech-text interleaving knowledge transfer scaling analysis scaling trends synthetic data TextLM model families speech semantic metrics