title: MuCodec: Ultra Low-Bitrate Music Codec
summary: Music codecs are a vital aspect of audio codec research, and ultra
  low-bitrate compression holds significant importance for music transmission and
  generation. Due to the complexity of music backgrounds and the richness of
  vocals, solely relying on modeling semantic or acoustic information cannot
  effectively reconstruct music with both vocals and backgrounds. To address this
  issue, we propose MuCodec, specifically targeting music compression and
  reconstruction tasks at ultra low bitrates. MuCodec employs MuEncoder to
  extract both acoustic and semantic features, discretizes them with RVQ, and
  obtains Mel-VAE features via flow-matching. The music is then reconstructed
  using a pre-trained MEL-VAE decoder and HiFi-GAN. MuCodec can reconstruct
  high-fidelity music at ultra low (0.35kbps) or high bitrates (1.35kbps),
  achieving the best results to date in both subjective and objective metrics.
  Code and Demo: https://xuyaoxun.github.io/MuCodec_demo/.
paper_authors._id: 66f0c6b4831eacfcad5a970b
paper_authors.name: Yaoxun Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-23T07:32:36.922Z
paper_authors._id: 642fd482450c0de9a1d1c8b5
paper_authors.avatarUrl: /avatars/3765b121feb979e410f665bf20191161.svg
paper_authors.fullname: yaoxunxu
paper_authors.type: user
paper_authors.user: yaoxunxu
paper_authors._id: 66f0c6b4831eacfcad5a970c
paper_authors.name: Hangting Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-23T07:32:42.807Z
paper_authors._id: 631b2dbd0f22997ce6728e27
paper_authors.avatarUrl: /avatars/84dc35edbda6c071fcecec01918c5c56.svg
paper_authors.fullname: Hangting Chen
paper_authors.type: user
paper_authors.user: hangtingchen
paper_authors._id: 66f0c6b4831eacfcad5a970d
paper_authors.name: Jianwei Yu
paper_authors._id: 66f0c6b4831eacfcad5a970e
paper_authors.name: Wei Tan
paper_authors._id: 66f0c6b4831eacfcad5a970f
paper_authors.name: Rongzhi Gu
paper_authors._id: 66f0c6b4831eacfcad5a9710
paper_authors.name: Shun Lei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-23T07:33:26.089Z
paper_authors._id: 6531768050f0ed02addafa95
paper_authors.avatarUrl: /avatars/b6689d42ab2f994ba0f149fba7732261.svg
paper_authors.fullname: Shun Lei
paper_authors.type: user
paper_authors.user: lglg666
paper_authors._id: 66f0c6b4831eacfcad5a9711
paper_authors.name: Zhiwei Lin
paper_authors._id: 66f0c6b4831eacfcad5a9712
paper_authors.name: Zhiyong Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-23T07:34:09.118Z
paper_authors._id: 6280e830e99dccaac4bbfde5
paper_authors.avatarUrl: /avatars/9242b8d2826ce2f79af9bb794bba2b61.svg
paper_authors.fullname: Zhiyong Wu
paper_authors.type: user
paper_authors.user: zy001
paper_title: MuCodec: Ultra Low-Bitrate Music Codec
paper_summary: Music codecs are a vital aspect of audio codec research, and ultra
  low-bitrate compression holds significant importance for music transmission and
  generation. Due to the complexity of music backgrounds and the richness of
  vocals, solely relying on modeling semantic or acoustic information cannot
  effectively reconstruct music with both vocals and backgrounds. To address this
  issue, we propose MuCodec, specifically targeting music compression and
  reconstruction tasks at ultra low bitrates. MuCodec employs MuEncoder to
  extract both acoustic and semantic features, discretizes them with RVQ, and
  obtains Mel-VAE features via flow-matching. The music is then reconstructed
  using a pre-trained MEL-VAE decoder and HiFi-GAN. MuCodec can reconstruct
  high-fidelity music at ultra low (0.35kbps) or high bitrates (1.35kbps),
  achieving the best results to date in both subjective and objective metrics.
  Code and Demo: https://xuyaoxun.github.io/MuCodec_demo/.
paper_ai_summary: MuCodec, using MuEncoder, RVQ, flow-matching, MEL-VAE, and HiFi-GAN, achieves high-fidelity music reconstruction at ultra-low bitrates.
paper_ai_keywords: MuEncoder RVQ flow-matching MEL-VAE HiFi-GAN