Toward Joint Language Modeling for Speech Units and Text
Speech and text are two major forms of human language. The research community
has been focusing on mapping speech to text or vice versa for many years.
However, in the field of language modeling, very little effort has been made to
model them jointly. In light of this, we explore joint language modeling for
speech units and text. Specifically, we compare different speech tokenizers to
transform continuous speech signals into discrete units and use different
methods to construct mixed speech-text data. We introduce automatic metrics to
evaluate how well the joint LM mixes speech and text. We also fine-tune the LM
on downstream spoken language understanding (SLU) tasks with different
modalities (speech or text) and test its performance to assess the model's
learning of shared representations. Our results show that by mixing speech
units and text with our proposed mixing techniques, the joint LM improves over
a speech-only baseline on SLU tasks and shows zero-shot cross-modal
transferability.
652cbb3d90e317f243af2d00
Ju-Chieh Chou
admin_assigned
2023-10-16T08:13:36.551Z
652588df2ceb2f426c31c2e9
/avatars/d9111152ad7c79c23798682c53facf47.svg
Ju-Chieh Chou
user
jjery2243542
652cbb3d90e317f243af2d01
Chung-Ming Chien
652cbb3d90e317f243af2d02
Wei-Ning Hsu
admin_assigned
2023-10-16T08:14:09.565Z
64c143d484887aa2b43bac0d
/avatars/817dba24739676aa0f34692702bb00d6.svg
Wei-Ning Hsu
user
wnhsu
652cbb3d90e317f243af2d03
Karen Livescu
652cbb3d90e317f243af2d04
Arun Babu
admin_assigned
2023-10-16T08:14:46.037Z
61776ccc924026cc8c692630
/avatars/44c669649402a52867861990e8b17d27.svg
Arun Babu
user
arbabu123
652cbb3d90e317f243af2d05
Alexis Conneau
admin_assigned
2023-10-16T08:14:53.956Z
61776a8f1e3e9ecbf77ddeea
/avatars/f2b9215fa02178194d0cbf2819681824.svg
Alexis C
user
aconneau
652cbb3d90e317f243af2d06
Alexei Baevski
admin_assigned
2023-10-16T08:15:01.514Z
6048123112aa89499a978a21
https://cdn-avatars.huggingface.co/v1/production/uploads/1615335958554-noauth.jpeg
Alexei Baevski
user
abaevski
652cbb3d90e317f243af2d07
Michael Auli
admin_assigned
2023-10-16T08:15:11.962Z
6176e9d475f6ae94d250042c
/avatars/ce0eafa09525b741c22717bbde6c6a16.svg
Michael Auli
user
michaelauli
Toward Joint Language Modeling for Speech Units and Text
Speech and text are two major forms of human language. The research community
has been focusing on mapping speech to text or vice versa for many years.
However, in the field of language modeling, very little effort has been made to
model them jointly. In light of this, we explore joint language modeling for
speech units and text. Specifically, we compare different speech tokenizers to
transform continuous speech signals into discrete units and use different
methods to construct mixed speech-text data. We introduce automatic metrics to
evaluate how well the joint LM mixes speech and text. We also fine-tune the LM
on downstream spoken language understanding (SLU) tasks with different
modalities (speech or text) and test its performance to assess the model's
learning of shared representations. Our results show that by mixing speech
units and text with our proposed mixing techniques, the joint LM improves over
a speech-only baseline on SLU tasks and shows zero-shot cross-modal
transferability.
A joint language model for speech and text improves performance on spoken language understanding tasks and demonstrates zero-shot cross-modal transferability.
speech tokenizers mixed speech-text data joint language modeling automatic metrics spoken language understanding (SLU) zero-shot cross-modal transferability