title: From TOWER to SPIRE: Adding the Speech Modality to a Text-Only LLM
summary: Large language models (LLMs) have shown remarkable performance and
  generalization capabilities across multiple languages and tasks, making them
  very attractive targets for multi-modality integration (e.g., images or
  speech). In this work, we extend an existing LLM to the speech modality via
  speech discretization and continued pre-training. In particular, we are
  interested in multilingual LLMs, such as TOWER, as their pre-training setting
  allows us to treat discretized speech input as an additional translation
  language. The resulting open-source model, SPIRE, is able to transcribe and
  translate English speech input while maintaining TOWER's original performance
  on translation-related tasks, showcasing that discretized speech input
  integration as an additional language is feasible during LLM adaptation. We
  make our code and models available to the community.
paper_authors._id: 67d46c53f624d5a39650024d
paper_authors.name: Kshitij Ambilduke
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-17T14:56:05.985Z
paper_authors._id: 65bce8265604c1d38309e3d7
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/7_JdljgybU6AWglzwMr93.png
paper_authors.fullname: Kshitij Ambilduke
paper_authors.type: user
paper_authors.user: KshitijAmbilduke
paper_authors._id: 67d46c53f624d5a39650024e
paper_authors.name: Ben Peters
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-16T21:13:33.209Z
paper_authors._id: 6568a0efdd4a892a1491605c
paper_authors.avatarUrl: /avatars/0ec70d96c0e78538fc8dd9153bb4220f.svg
paper_authors.fullname: Ben Peters
paper_authors.type: user
paper_authors.user: bpop
paper_authors._id: 67d46c53f624d5a39650024f
paper_authors.name: Sonal Sannigrahi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-17T14:56:12.084Z
paper_authors._id: 65c37598f7296c2feeddc531
paper_authors.avatarUrl: /avatars/cc2242f8822b422556449227cac9762c.svg
paper_authors.fullname: Sonal Sannigrahi
paper_authors.type: user
paper_authors.user: sonalsannigrahi
paper_authors._id: 67d46c53f624d5a396500250
paper_authors.name: Anil Keshwani
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-17T14:56:18.745Z
paper_authors._id: 637cfe296231c36c63711f3c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/637cfe296231c36c63711f3c/IVT-o57LMsOpNE_ddT_tW.jpeg
paper_authors.fullname: Anil Keshwani
paper_authors.type: user
paper_authors.user: anilkeshwani
paper_authors._id: 67d46c53f624d5a396500251
paper_authors.name: Tsz Kin Lam
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-17T14:56:24.816Z
paper_authors._id: 65d629f8f67b1764166e2ab0
paper_authors.avatarUrl: /avatars/492da03e63a3a647b14132f1ea39cca7.svg
paper_authors.fullname: Tsz Kin Lam
paper_authors.type: user
paper_authors.user: edinkin
paper_authors._id: 67d46c53f624d5a396500252
paper_authors.name: Bruno Martins
paper_authors._id: 67d46c53f624d5a396500253
paper_authors.name: Marcely Zanon Boito
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-17T14:57:03.534Z
paper_authors._id: 62262e19d36494a6f743a28d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1667383417385-62262e19d36494a6f743a28d.jpeg
paper_authors.fullname: Marcely Zanon Boito
paper_authors.type: user
paper_authors.user: mzboito
paper_authors._id: 67d46c53f624d5a396500254
paper_authors.name: André F. T. Martins
paper_title: From TOWER to SPIRE: Adding the Speech Modality to a Text-Only LLM
paper_summary: Large language models (LLMs) have shown remarkable performance and
  generalization capabilities across multiple languages and tasks, making them
  very attractive targets for multi-modality integration (e.g., images or
  speech). In this work, we extend an existing LLM to the speech modality via
  speech discretization and continued pre-training. In particular, we are
  interested in multilingual LLMs, such as TOWER, as their pre-training setting
  allows us to treat discretized speech input as an additional translation
  language. The resulting open-source model, SPIRE, is able to transcribe and
  translate English speech input while maintaining TOWER's original performance
  on translation-related tasks, showcasing that discretized speech input
  integration as an additional language is feasible during LLM adaptation. We
  make our code and models available to the community.
paper_ai_summary: A multilingual LLM adapted for speech transcription and translation through speech discretization and continued pre-training can maintain original translation performance.
paper_ai_keywords: LLMs large language models speech modality speech discretization continued pre-training multilingual LLMs TOWER SPIRE transcription translation