Speechless: Speech Instruction Training Without Speech for Low Resource
  Languages
The rapid growth of voice assistants powered by large language models (LLM)
has highlighted a need for speech instruction data to train these systems.
Despite the abundance of speech recognition data, there is a notable scarcity
of speech instruction data, which is essential for fine-tuning models to
understand and execute spoken commands. Generating high-quality synthetic
speech requires a good text-to-speech (TTS) model, which may not be available
to low resource languages. Our novel approach addresses this challenge by
halting synthesis at the semantic representation level, bypassing the need for
TTS. We achieve this by aligning synthetic semantic representations with the
pre-trained Whisper encoder, enabling an LLM to be fine-tuned on text
instructions while maintaining the ability to understand spoken instructions
during inference. This simplified training process is a promising approach to
building voice assistant for low-resource languages.
6833d2df73bebebe5cd6604e
Alan Dao
extracted_confirmed
2025-05-26T02:38:31.379Z
62d7b2339b629105a5d6888a
https://cdn-avatars.huggingface.co/v1/production/uploads/62d7b2339b629105a5d6888a/pYi994fqzXctgoed79oUo.png
Alan Dao
user
alandao
6833d2df73bebebe5cd6604f
Dinh Bach Vu
6833d2df73bebebe5cd66050
Huy Hoang Ha
6833d2df73bebebe5cd66051
Tuan Le Duc Anh
6833d2df73bebebe5cd66052
Shreyas Gopal
6833d2df73bebebe5cd66053
Yue Heng Yeo
6833d2df73bebebe5cd66054
Warren Keng Hoong Low
6833d2df73bebebe5cd66055
Eng Siong Chng
6833d2df73bebebe5cd66056
Jia Qi Yip
Speechless: Speech Instruction Training Without Speech for Low Resource
  Languages
The rapid growth of voice assistants powered by large language models (LLM)
has highlighted a need for speech instruction data to train these systems.
Despite the abundance of speech recognition data, there is a notable scarcity
of speech instruction data, which is essential for fine-tuning models to
understand and execute spoken commands. Generating high-quality synthetic
speech requires a good text-to-speech (TTS) model, which may not be available
to low resource languages. Our novel approach addresses this challenge by
halting synthesis at the semantic representation level, bypassing the need for
TTS. We achieve this by aligning synthetic semantic representations with the
pre-trained Whisper encoder, enabling an LLM to be fine-tuned on text
instructions while maintaining the ability to understand spoken instructions
during inference. This simplified training process is a promising approach to
building voice assistant for low-resource languages.
A method bypasses the need for TTS models by aligning semantic representations with a Whisper encoder, enabling LLMs to understand both text and spoken instructions for low-resource languages.
large language models LLM speech instruction data TTS semantic representation Whisper encoder fine-tuning low-resource languages