title: FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networks
summary: Large language models have revolutionized natural language processing through
  self-supervised pretraining on massive datasets. Inspired by this success,
  researchers have explored adapting these methods to speech by discretizing
  continuous audio into tokens using neural audio codecs. However, existing
  approaches face limitations, including high bitrates, the loss of either
  semantic or acoustic information, and the reliance on multi-codebook designs
  when trying to capture both, which increases architectural complexity for
  downstream tasks. To address these challenges, we introduce FocalCodec, an
  efficient low-bitrate codec based on focal modulation that utilizes a single
  binary codebook to compress speech between 0.16 and 0.65 kbps. FocalCodec
  delivers competitive performance in speech resynthesis and voice conversion at
  lower bitrates than the current state-of-the-art, while effectively handling
  multilingual speech and noisy environments. Evaluation on downstream tasks
  shows that FocalCodec successfully preserves sufficient semantic and acoustic
  information, while also being well-suited for generative modeling. Demo
  samples, code and checkpoints are available at
  https://lucadellalib.github.io/focalcodec-web/.
paper_authors._id: 67a953844ea315a67e02461d
paper_authors.name: Luca Della Libera
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-11T10:03:10.257Z
paper_authors._id: 63195d0582e7eec0eac040e3
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63195d0582e7eec0eac040e3/0tXOYkMfmv9e53zBWgqz7.png
paper_authors.fullname: Luca Della Libera
paper_authors.type: user
paper_authors.user: lucadellalib
paper_authors._id: 67a953844ea315a67e02461e
paper_authors.name: Francesco Paissan
paper_authors._id: 67a953844ea315a67e02461f
paper_authors.name: Cem Subakan
paper_authors._id: 67a953844ea315a67e024620
paper_authors.name: Mirco Ravanelli
paper_title: FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networks
paper_summary: Large language models have revolutionized natural language processing through
  self-supervised pretraining on massive datasets. Inspired by this success,
  researchers have explored adapting these methods to speech by discretizing
  continuous audio into tokens using neural audio codecs. However, existing
  approaches face limitations, including high bitrates, the loss of either
  semantic or acoustic information, and the reliance on multi-codebook designs
  when trying to capture both, which increases architectural complexity for
  downstream tasks. To address these challenges, we introduce FocalCodec, an
  efficient low-bitrate codec based on focal modulation that utilizes a single
  binary codebook to compress speech between 0.16 and 0.65 kbps. FocalCodec
  delivers competitive performance in speech resynthesis and voice conversion at
  lower bitrates than the current state-of-the-art, while effectively handling
  multilingual speech and noisy environments. Evaluation on downstream tasks
  shows that FocalCodec successfully preserves sufficient semantic and acoustic
  information, while also being well-suited for generative modeling. Demo
  samples, code and checkpoints are available at
  https://lucadellalib.github.io/focalcodec-web/.
paper_ai_summary: FocalCodec, a low-bitrate codec using focal modulation, efficiently compresses speech with minimal loss of semantic and acoustic information, excelling in resynthesis, voice conversion, and generative modeling tasks.
paper_ai_keywords: self-supervised pretraining neural audio codecs focal modulation speech resynthesis voice conversion generative modeling