title: DM-Codec: Distilling Multimodal Representations for Speech Tokenization
summary: Recent advancements in speech-language models have yielded significant
  improvements in speech tokenization and synthesis. However, effectively mapping
  the complex, multidimensional attributes of speech into discrete tokens remains
  challenging. This process demands acoustic, semantic, and contextual
  information for precise speech representations. Existing speech representations
  generally fall into two categories: acoustic tokens from audio codecs and
  semantic tokens from speech self-supervised learning models. Although recent
  efforts have unified acoustic and semantic tokens for improved performance,
  they overlook the crucial role of contextual representation in comprehensive
  speech modeling. Our empirical investigations reveal that the absence of
  contextual representations results in elevated Word Error Rate (WER) and Word
  Information Lost (WIL) scores in speech transcriptions. To address these
  limitations, we propose two novel distillation approaches: (1) a language model
  (LM)-guided distillation method that incorporates contextual information, and
  (2) a combined LM and self-supervised speech model (SM)-guided distillation
  technique that effectively distills multimodal representations (acoustic,
  semantic, and contextual) into a comprehensive speech tokenizer, termed
  DM-Codec. The DM-Codec architecture adopts a streamlined encoder-decoder
  framework with a Residual Vector Quantizer (RVQ) and incorporates the LM and SM
  during the training process. Experiments show DM-Codec significantly
  outperforms state-of-the-art speech tokenization models, reducing WER by up to
  13.46%, WIL by 9.82%, and improving speech quality by 5.84% and intelligibility
  by 1.85% on the LibriSpeech benchmark dataset. The code, samples, and model
  checkpoints are available at https://github.com/mubtasimahasan/DM-Codec.
paper_authors._id: 67171043c3d0f3134dc15bde
paper_authors.name: Md Mubtasim Ahasan
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-10-22T03:07:46.167Z
paper_authors._id: 6401a3d63e3d0f2745aad104
paper_authors.avatarUrl: /avatars/8c458237830c814676386b023401a80d.svg
paper_authors.fullname: Md Mubtasim Ahasan
paper_authors.type: user
paper_authors.user: mubtasim
paper_authors._id: 67171043c3d0f3134dc15bdf
paper_authors.name: Md Fahim
paper_authors._id: 67171043c3d0f3134dc15be0
paper_authors.name: Tasnim Mohiuddin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-22T09:43:10.038Z
paper_authors._id: 60d34afe0c54a218bc775297
paper_authors.avatarUrl: /avatars/fd560b0f858be5ed534f85b7a6df3e14.svg
paper_authors.fullname: Tasnim Mohiuddin
paper_authors.type: user
paper_authors.user: tasnim
paper_authors._id: 67171043c3d0f3134dc15be1
paper_authors.name: A K M Mahbubur Rahman
paper_authors._id: 67171043c3d0f3134dc15be2
paper_authors.name: Aman Chadha
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-22T08:01:02.592Z
paper_authors._id: 63a4754927f1f64ed7238dac
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63a4754927f1f64ed7238dac/aH-eJF-31g4vof9jv2gmI.jpeg
paper_authors.fullname: Aman Chadha
paper_authors.type: user
paper_authors.user: amanchadha
paper_authors._id: 67171043c3d0f3134dc15be3
paper_authors.name: Tariq Iqbal
paper_authors._id: 67171043c3d0f3134dc15be4
paper_authors.name: M Ashraful Amin
paper_authors._id: 67171043c3d0f3134dc15be5
paper_authors.name: Md Mofijul Islam
paper_authors._id: 67171043c3d0f3134dc15be6
paper_authors.name: Amin Ahsan Ali
paper_title: DM-Codec: Distilling Multimodal Representations for Speech Tokenization
paper_summary: Recent advancements in speech-language models have yielded significant
  improvements in speech tokenization and synthesis. However, effectively mapping
  the complex, multidimensional attributes of speech into discrete tokens remains
  challenging. This process demands acoustic, semantic, and contextual
  information for precise speech representations. Existing speech representations
  generally fall into two categories: acoustic tokens from audio codecs and
  semantic tokens from speech self-supervised learning models. Although recent
  efforts have unified acoustic and semantic tokens for improved performance,
  they overlook the crucial role of contextual representation in comprehensive
  speech modeling. Our empirical investigations reveal that the absence of
  contextual representations results in elevated Word Error Rate (WER) and Word
  Information Lost (WIL) scores in speech transcriptions. To address these
  limitations, we propose two novel distillation approaches: (1) a language model
  (LM)-guided distillation method that incorporates contextual information, and
  (2) a combined LM and self-supervised speech model (SM)-guided distillation
  technique that effectively distills multimodal representations (acoustic,
  semantic, and contextual) into a comprehensive speech tokenizer, termed
  DM-Codec. The DM-Codec architecture adopts a streamlined encoder-decoder
  framework with a Residual Vector Quantizer (RVQ) and incorporates the LM and SM
  during the training process. Experiments show DM-Codec significantly
  outperforms state-of-the-art speech tokenization models, reducing WER by up to
  13.46%, WIL by 9.82%, and improving speech quality by 5.84% and intelligibility
  by 1.85% on the LibriSpeech benchmark dataset. The code, samples, and model
  checkpoints are available at https://github.com/mubtasimahasan/DM-Codec.
paper_ai_keywords: speech-language models speech tokenization speech synthesis acoustic tokens semantic tokens speech self-supervised learning models contextual information Word Error Rate (WER) Word Information Lost (WIL) language model (LM)-guided distillation self-supervised speech model (SM)-guided distillation multimodal representations residual vector quantizer (RVQ) LibriSpeech benchmark dataset DM-Codec