SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General
  Sound
Large language models (LLMs) have significantly advanced audio processing
through audio codecs that convert audio into discrete tokens, enabling the
application of language modelling techniques to audio data. However,
traditional codecs often operate at high bitrates or within narrow domains such
as speech and lack the semantic clues required for efficient language
modelling. Addressing these challenges, we introduce SemantiCodec, a novel
codec designed to compress audio into fewer than a hundred tokens per second
across diverse audio types, including speech, general audio, and music, without
compromising quality. SemantiCodec features a dual-encoder architecture: a
semantic encoder using a self-supervised AudioMAE, discretized using k-means
clustering on extensive audio data, and an acoustic encoder to capture the
remaining details. The semantic and acoustic encoder outputs are used to
reconstruct audio via a diffusion-model-based decoder. SemantiCodec is
presented in three variants with token rates of 25, 50, and 100 per second,
supporting a range of ultra-low bit rates between 0.31 kbps and 1.43 kbps.
Experimental results demonstrate that SemantiCodec significantly outperforms
the state-of-the-art Descript codec on reconstruction quality. Our results also
suggest that SemantiCodec contains significantly richer semantic information
than all evaluated audio codecs, even at significantly lower bitrates. Our code
and demos are available at https://haoheliu.github.io/SemantiCodec/.
66331a1c13f8fa61cd0d5ac5
Haohe Liu
admin_assigned
2024-05-02T07:39:56.863Z
6155245d1c762d4d61b51d5d
https://cdn-avatars.huggingface.co/v1/production/uploads/1677233708065-6155245d1c762d4d61b51d5d.png
haoheliu
user
haoheliu
66331a1c13f8fa61cd0d5ac6
Xuenan Xu
admin_assigned
2024-05-02T07:40:07.169Z
6457623c90bf61c87f9ffdc4
https://cdn-avatars.huggingface.co/v1/production/uploads/6457623c90bf61c87f9ffdc4/pKG76YipWfYFhSnorepsl.jpeg
Xuenan Xu
user
wsntxxn
66331a1c13f8fa61cd0d5ac7
Yi Yuan
66331a1c13f8fa61cd0d5ac8
Mengyue Wu
66331a1c13f8fa61cd0d5ac9
Wenwu Wang
admin_assigned
2024-05-02T07:40:31.908Z
65c30ec23d8075e46a3da733
/avatars/164d4da7ba66f451ed08f45b76a42ffe.svg
wangwenwu
user
wangwenwu2
66331a1c13f8fa61cd0d5aca
Mark D. Plumbley
SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General
  Sound
Large language models (LLMs) have significantly advanced audio processing
through audio codecs that convert audio into discrete tokens, enabling the
application of language modelling techniques to audio data. However,
traditional codecs often operate at high bitrates or within narrow domains such
as speech and lack the semantic clues required for efficient language
modelling. Addressing these challenges, we introduce SemantiCodec, a novel
codec designed to compress audio into fewer than a hundred tokens per second
across diverse audio types, including speech, general audio, and music, without
compromising quality. SemantiCodec features a dual-encoder architecture: a
semantic encoder using a self-supervised AudioMAE, discretized using k-means
clustering on extensive audio data, and an acoustic encoder to capture the
remaining details. The semantic and acoustic encoder outputs are used to
reconstruct audio via a diffusion-model-based decoder. SemantiCodec is
presented in three variants with token rates of 25, 50, and 100 per second,
supporting a range of ultra-low bit rates between 0.31 kbps and 1.43 kbps.
Experimental results demonstrate that SemantiCodec significantly outperforms
the state-of-the-art Descript codec on reconstruction quality. Our results also
suggest that SemantiCodec contains significantly richer semantic information
than all evaluated audio codecs, even at significantly lower bitrates. Our code
and demos are available at https://haoheliu.github.io/SemantiCodec/.
SemantiCodec is a dual-encoder codec that compresses diverse audio types into fewer tokens using AudioMAE and k-means clustering, improved by a diffusion-model-based decoder, achieving higher semantic information and superior reconstruction quality at lower bitrates than existing codecs.
AudioMAE k-means clustering diffusion-model-based decoder dual-encoder architecture