title: Speechless: Speech Instruction Training Without Speech for Low Resource
    Languages
summary: The rapid growth of voice assistants powered by large language models (LLM)
  has highlighted a need for speech instruction data to train these systems.
  Despite the abundance of speech recognition data, there is a notable scarcity
  of speech instruction data, which is essential for fine-tuning models to
  understand and execute spoken commands. Generating high-quality synthetic
  speech requires a good text-to-speech (TTS) model, which may not be available
  to low resource languages. Our novel approach addresses this challenge by
  halting synthesis at the semantic representation level, bypassing the need for
  TTS. We achieve this by aligning synthetic semantic representations with the
  pre-trained Whisper encoder, enabling an LLM to be fine-tuned on text
  instructions while maintaining the ability to understand spoken instructions
  during inference. This simplified training process is a promising approach to
  building voice assistant for low-resource languages.
paper_authors._id: 6833d2df73bebebe5cd6604e
paper_authors.name: Alan Dao
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-05-26T02:38:31.379Z
paper_authors._id: 62d7b2339b629105a5d6888a
paper_authors.avatarUrl: /avatars/c3f164fde6b8f9a671890e08ce8a3e75.svg
paper_authors.fullname: Alan Dao
paper_authors.type: user
paper_authors.user: alandao
paper_authors._id: 6833d2df73bebebe5cd6604f
paper_authors.name: Dinh Bach Vu
paper_authors._id: 6833d2df73bebebe5cd66050
paper_authors.name: Huy Hoang Ha
paper_authors._id: 6833d2df73bebebe5cd66051
paper_authors.name: Tuan Le Duc Anh
paper_authors._id: 6833d2df73bebebe5cd66052
paper_authors.name: Shreyas Gopal
paper_authors._id: 6833d2df73bebebe5cd66053
paper_authors.name: Yue Heng Yeo
paper_authors._id: 6833d2df73bebebe5cd66054
paper_authors.name: Warren Keng Hoong Low
paper_authors._id: 6833d2df73bebebe5cd66055
paper_authors.name: Eng Siong Chng
paper_authors._id: 6833d2df73bebebe5cd66056
paper_authors.name: Jia Qi Yip
paper_title: Speechless: Speech Instruction Training Without Speech for Low Resource
    Languages
paper_summary: The rapid growth of voice assistants powered by large language models (LLM)
  has highlighted a need for speech instruction data to train these systems.
  Despite the abundance of speech recognition data, there is a notable scarcity
  of speech instruction data, which is essential for fine-tuning models to
  understand and execute spoken commands. Generating high-quality synthetic
  speech requires a good text-to-speech (TTS) model, which may not be available
  to low resource languages. Our novel approach addresses this challenge by
  halting synthesis at the semantic representation level, bypassing the need for
  TTS. We achieve this by aligning synthetic semantic representations with the
  pre-trained Whisper encoder, enabling an LLM to be fine-tuned on text
  instructions while maintaining the ability to understand spoken instructions
  during inference. This simplified training process is a promising approach to
  building voice assistant for low-resource languages.
paper_ai_summary: A method bypasses the need for TTS models by aligning semantic representations with a Whisper encoder, enabling LLMs to understand both text and spoken instructions for low-resource languages.
paper_ai_keywords: large language models LLM speech instruction data TTS semantic representation Whisper encoder fine-tuning low-resource languages