From TOWER to SPIRE: Adding the Speech Modality to a Text-Only LLM
Large language models (LLMs) have shown remarkable performance and
generalization capabilities across multiple languages and tasks, making them
very attractive targets for multi-modality integration (e.g., images or
speech). In this work, we extend an existing LLM to the speech modality via
speech discretization and continued pre-training. In particular, we are
interested in multilingual LLMs, such as TOWER, as their pre-training setting
allows us to treat discretized speech input as an additional translation
language. The resulting open-source model, SPIRE, is able to transcribe and
translate English speech input while maintaining TOWER's original performance
on translation-related tasks, showcasing that discretized speech input
integration as an additional language is feasible during LLM adaptation. We
make our code and models available to the community.
67d46c53f624d5a39650024d
Kshitij Ambilduke
admin_assigned
2025-03-17T14:56:05.985Z
65bce8265604c1d38309e3d7
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/7_JdljgybU6AWglzwMr93.png
Kshitij Ambilduke
user
KshitijAmbilduke
67d46c53f624d5a39650024e
Ben Peters
claimed_verified
2025-03-16T21:13:33.209Z
6568a0efdd4a892a1491605c
/avatars/0ec70d96c0e78538fc8dd9153bb4220f.svg
Ben Peters
user
bpop
67d46c53f624d5a39650024f
Sonal Sannigrahi
admin_assigned
2025-03-17T14:56:12.084Z
65c37598f7296c2feeddc531
/avatars/cc2242f8822b422556449227cac9762c.svg
Sonal Sannigrahi
user
sonalsannigrahi
67d46c53f624d5a396500250
Anil Keshwani
admin_assigned
2025-03-17T14:56:18.745Z
637cfe296231c36c63711f3c
https://cdn-avatars.huggingface.co/v1/production/uploads/637cfe296231c36c63711f3c/IVT-o57LMsOpNE_ddT_tW.jpeg
Anil Keshwani
user
anilkeshwani
67d46c53f624d5a396500251
Tsz Kin Lam
admin_assigned
2025-03-17T14:56:24.816Z
65d629f8f67b1764166e2ab0
/avatars/492da03e63a3a647b14132f1ea39cca7.svg
Tsz Kin Lam
user
edinkin
67d46c53f624d5a396500252
Bruno Martins
67d46c53f624d5a396500253
Marcely Zanon Boito
admin_assigned
2025-03-17T14:57:03.534Z
62262e19d36494a6f743a28d
https://cdn-avatars.huggingface.co/v1/production/uploads/1667383417385-62262e19d36494a6f743a28d.jpeg
Marcely Zanon Boito
user
mzboito
67d46c53f624d5a396500254
André F. T. Martins
From TOWER to SPIRE: Adding the Speech Modality to a Text-Only LLM
Large language models (LLMs) have shown remarkable performance and
generalization capabilities across multiple languages and tasks, making them
very attractive targets for multi-modality integration (e.g., images or
speech). In this work, we extend an existing LLM to the speech modality via
speech discretization and continued pre-training. In particular, we are
interested in multilingual LLMs, such as TOWER, as their pre-training setting
allows us to treat discretized speech input as an additional translation
language. The resulting open-source model, SPIRE, is able to transcribe and
translate English speech input while maintaining TOWER's original performance
on translation-related tasks, showcasing that discretized speech input
integration as an additional language is feasible during LLM adaptation. We
make our code and models available to the community.
A multilingual LLM adapted for speech transcription and translation through speech discretization and continued pre-training can maintain original translation performance.
LLMs large language models speech modality speech discretization continued pre-training multilingual LLMs TOWER SPIRE transcription translation