title: SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General
    Sound
summary: Large language models (LLMs) have significantly advanced audio processing
  through audio codecs that convert audio into discrete tokens, enabling the
  application of language modelling techniques to audio data. However,
  traditional codecs often operate at high bitrates or within narrow domains such
  as speech and lack the semantic clues required for efficient language
  modelling. Addressing these challenges, we introduce SemantiCodec, a novel
  codec designed to compress audio into fewer than a hundred tokens per second
  across diverse audio types, including speech, general audio, and music, without
  compromising quality. SemantiCodec features a dual-encoder architecture: a
  semantic encoder using a self-supervised AudioMAE, discretized using k-means
  clustering on extensive audio data, and an acoustic encoder to capture the
  remaining details. The semantic and acoustic encoder outputs are used to
  reconstruct audio via a diffusion-model-based decoder. SemantiCodec is
  presented in three variants with token rates of 25, 50, and 100 per second,
  supporting a range of ultra-low bit rates between 0.31 kbps and 1.43 kbps.
  Experimental results demonstrate that SemantiCodec significantly outperforms
  the state-of-the-art Descript codec on reconstruction quality. Our results also
  suggest that SemantiCodec contains significantly richer semantic information
  than all evaluated audio codecs, even at significantly lower bitrates. Our code
  and demos are available at https://haoheliu.github.io/SemantiCodec/.
paper_authors._id: 66331a1c13f8fa61cd0d5ac5
paper_authors.name: Haohe Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T07:39:56.863Z
paper_authors._id: 6155245d1c762d4d61b51d5d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1677233708065-6155245d1c762d4d61b51d5d.png
paper_authors.fullname: haoheliu
paper_authors.type: user
paper_authors.user: haoheliu
paper_authors._id: 66331a1c13f8fa61cd0d5ac6
paper_authors.name: Xuenan Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T07:40:07.169Z
paper_authors._id: 6457623c90bf61c87f9ffdc4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6457623c90bf61c87f9ffdc4/pKG76YipWfYFhSnorepsl.jpeg
paper_authors.fullname: Xuenan Xu
paper_authors.type: user
paper_authors.user: wsntxxn
paper_authors._id: 66331a1c13f8fa61cd0d5ac7
paper_authors.name: Yi Yuan
paper_authors._id: 66331a1c13f8fa61cd0d5ac8
paper_authors.name: Mengyue Wu
paper_authors._id: 66331a1c13f8fa61cd0d5ac9
paper_authors.name: Wenwu Wang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-05-02T07:40:31.908Z
paper_authors._id: 65c30ec23d8075e46a3da733
paper_authors.avatarUrl: /avatars/164d4da7ba66f451ed08f45b76a42ffe.svg
paper_authors.fullname: wangwenwu
paper_authors.type: user
paper_authors.user: wangwenwu2
paper_authors._id: 66331a1c13f8fa61cd0d5aca
paper_authors.name: Mark D. Plumbley
paper_title: SemantiCodec: An Ultra Low Bitrate Semantic Audio Codec for General
    Sound
paper_summary: Large language models (LLMs) have significantly advanced audio processing
  through audio codecs that convert audio into discrete tokens, enabling the
  application of language modelling techniques to audio data. However,
  traditional codecs often operate at high bitrates or within narrow domains such
  as speech and lack the semantic clues required for efficient language
  modelling. Addressing these challenges, we introduce SemantiCodec, a novel
  codec designed to compress audio into fewer than a hundred tokens per second
  across diverse audio types, including speech, general audio, and music, without
  compromising quality. SemantiCodec features a dual-encoder architecture: a
  semantic encoder using a self-supervised AudioMAE, discretized using k-means
  clustering on extensive audio data, and an acoustic encoder to capture the
  remaining details. The semantic and acoustic encoder outputs are used to
  reconstruct audio via a diffusion-model-based decoder. SemantiCodec is
  presented in three variants with token rates of 25, 50, and 100 per second,
  supporting a range of ultra-low bit rates between 0.31 kbps and 1.43 kbps.
  Experimental results demonstrate that SemantiCodec significantly outperforms
  the state-of-the-art Descript codec on reconstruction quality. Our results also
  suggest that SemantiCodec contains significantly richer semantic information
  than all evaluated audio codecs, even at significantly lower bitrates. Our code
  and demos are available at https://haoheliu.github.io/SemantiCodec/.
paper_ai_summary: SemantiCodec is a dual-encoder codec that compresses diverse audio types into fewer tokens using AudioMAE and k-means clustering, improved by a diffusion-model-based decoder, achieving higher semantic information and superior reconstruction quality at lower bitrates than existing codecs.
paper_ai_keywords: AudioMAE k-means clustering diffusion-model-based decoder dual-encoder architecture