FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networks
Large language models have revolutionized natural language processing through
self-supervised pretraining on massive datasets. Inspired by this success,
researchers have explored adapting these methods to speech by discretizing
continuous audio into tokens using neural audio codecs. However, existing
approaches face limitations, including high bitrates, the loss of either
semantic or acoustic information, and the reliance on multi-codebook designs
when trying to capture both, which increases architectural complexity for
downstream tasks. To address these challenges, we introduce FocalCodec, an
efficient low-bitrate codec based on focal modulation that utilizes a single
binary codebook to compress speech between 0.16 and 0.65 kbps. FocalCodec
delivers competitive performance in speech resynthesis and voice conversion at
lower bitrates than the current state-of-the-art, while effectively handling
multilingual speech and noisy environments. Evaluation on downstream tasks
shows that FocalCodec successfully preserves sufficient semantic and acoustic
information, while also being well-suited for generative modeling. Demo
samples, code and checkpoints are available at
https://lucadellalib.github.io/focalcodec-web/.
67a953844ea315a67e02461d
Luca Della Libera
claimed_verified
2025-02-11T10:03:10.257Z
63195d0582e7eec0eac040e3
https://cdn-avatars.huggingface.co/v1/production/uploads/63195d0582e7eec0eac040e3/0tXOYkMfmv9e53zBWgqz7.png
Luca Della Libera
user
lucadellalib
67a953844ea315a67e02461e
Francesco Paissan
67a953844ea315a67e02461f
Cem Subakan
67a953844ea315a67e024620
Mirco Ravanelli
FocalCodec: Low-Bitrate Speech Coding via Focal Modulation Networks
Large language models have revolutionized natural language processing through
self-supervised pretraining on massive datasets. Inspired by this success,
researchers have explored adapting these methods to speech by discretizing
continuous audio into tokens using neural audio codecs. However, existing
approaches face limitations, including high bitrates, the loss of either
semantic or acoustic information, and the reliance on multi-codebook designs
when trying to capture both, which increases architectural complexity for
downstream tasks. To address these challenges, we introduce FocalCodec, an
efficient low-bitrate codec based on focal modulation that utilizes a single
binary codebook to compress speech between 0.16 and 0.65 kbps. FocalCodec
delivers competitive performance in speech resynthesis and voice conversion at
lower bitrates than the current state-of-the-art, while effectively handling
multilingual speech and noisy environments. Evaluation on downstream tasks
shows that FocalCodec successfully preserves sufficient semantic and acoustic
information, while also being well-suited for generative modeling. Demo
samples, code and checkpoints are available at
https://lucadellalib.github.io/focalcodec-web/.
FocalCodec, a low-bitrate codec using focal modulation, efficiently compresses speech with minimal loss of semantic and acoustic information, excelling in resynthesis, voice conversion, and generative modeling tasks.
self-supervised pretraining neural audio codecs focal modulation speech resynthesis voice conversion generative modeling