DM-Codec: Distilling Multimodal Representations for Speech Tokenization
Recent advancements in speech-language models have yielded significant
improvements in speech tokenization and synthesis. However, effectively mapping
the complex, multidimensional attributes of speech into discrete tokens remains
challenging. This process demands acoustic, semantic, and contextual
information for precise speech representations. Existing speech representations
generally fall into two categories: acoustic tokens from audio codecs and
semantic tokens from speech self-supervised learning models. Although recent
efforts have unified acoustic and semantic tokens for improved performance,
they overlook the crucial role of contextual representation in comprehensive
speech modeling. Our empirical investigations reveal that the absence of
contextual representations results in elevated Word Error Rate (WER) and Word
Information Lost (WIL) scores in speech transcriptions. To address these
limitations, we propose two novel distillation approaches: (1) a language model
(LM)-guided distillation method that incorporates contextual information, and
(2) a combined LM and self-supervised speech model (SM)-guided distillation
technique that effectively distills multimodal representations (acoustic,
semantic, and contextual) into a comprehensive speech tokenizer, termed
DM-Codec. The DM-Codec architecture adopts a streamlined encoder-decoder
framework with a Residual Vector Quantizer (RVQ) and incorporates the LM and SM
during the training process. Experiments show DM-Codec significantly
outperforms state-of-the-art speech tokenization models, reducing WER by up to
13.46%, WIL by 9.82%, and improving speech quality by 5.84% and intelligibility
by 1.85% on the LibriSpeech benchmark dataset. The code, samples, and model
checkpoints are available at https://github.com/mubtasimahasan/DM-Codec.
67171043c3d0f3134dc15bde
Md Mubtasim Ahasan
extracted_confirmed
2024-10-22T03:07:46.167Z
6401a3d63e3d0f2745aad104
/avatars/8c458237830c814676386b023401a80d.svg
Md Mubtasim Ahasan
user
mubtasim
67171043c3d0f3134dc15bdf
Md Fahim
67171043c3d0f3134dc15be0
Tasnim Mohiuddin
admin_assigned
2024-10-22T09:43:10.038Z
60d34afe0c54a218bc775297
/avatars/fd560b0f858be5ed534f85b7a6df3e14.svg
Tasnim Mohiuddin
user
tasnim
67171043c3d0f3134dc15be1
A K M Mahbubur Rahman
67171043c3d0f3134dc15be2
Aman Chadha
claimed_verified
2024-10-22T08:01:02.592Z
63a4754927f1f64ed7238dac
https://cdn-avatars.huggingface.co/v1/production/uploads/63a4754927f1f64ed7238dac/aH-eJF-31g4vof9jv2gmI.jpeg
Aman Chadha
user
amanchadha
67171043c3d0f3134dc15be3
Tariq Iqbal
67171043c3d0f3134dc15be4
M Ashraful Amin
67171043c3d0f3134dc15be5
Md Mofijul Islam
67171043c3d0f3134dc15be6
Amin Ahsan Ali
DM-Codec: Distilling Multimodal Representations for Speech Tokenization
Recent advancements in speech-language models have yielded significant
improvements in speech tokenization and synthesis. However, effectively mapping
the complex, multidimensional attributes of speech into discrete tokens remains
challenging. This process demands acoustic, semantic, and contextual
information for precise speech representations. Existing speech representations
generally fall into two categories: acoustic tokens from audio codecs and
semantic tokens from speech self-supervised learning models. Although recent
efforts have unified acoustic and semantic tokens for improved performance,
they overlook the crucial role of contextual representation in comprehensive
speech modeling. Our empirical investigations reveal that the absence of
contextual representations results in elevated Word Error Rate (WER) and Word
Information Lost (WIL) scores in speech transcriptions. To address these
limitations, we propose two novel distillation approaches: (1) a language model
(LM)-guided distillation method that incorporates contextual information, and
(2) a combined LM and self-supervised speech model (SM)-guided distillation
technique that effectively distills multimodal representations (acoustic,
semantic, and contextual) into a comprehensive speech tokenizer, termed
DM-Codec. The DM-Codec architecture adopts a streamlined encoder-decoder
framework with a Residual Vector Quantizer (RVQ) and incorporates the LM and SM
during the training process. Experiments show DM-Codec significantly
outperforms state-of-the-art speech tokenization models, reducing WER by up to
13.46%, WIL by 9.82%, and improving speech quality by 5.84% and intelligibility
by 1.85% on the LibriSpeech benchmark dataset. The code, samples, and model
checkpoints are available at https://github.com/mubtasimahasan/DM-Codec.
A new speech tokenizer, DM-Codec, integrates contextual information from language and self-supervised speech models to improve speech tokenization and quality.
speech tokenization speech synthesis acoustic tokens semantic tokens contextual representation Word Error Rate Word Information Lost LM-guided distillation SM-guided distillation multimodal representations Residual Vector Quantizer DM-Codec encoder-decoder framework LibriSpeech benchmark dataset