title: From Discrete Tokens to High-Fidelity Audio Using Multi-Band Diffusion
summary: Deep generative models can generate high-fidelity audio conditioned on
  various types of representations (e.g., mel-spectrograms, Mel-frequency
  Cepstral Coefficients (MFCC)). Recently, such models have been used to
  synthesize audio waveforms conditioned on highly compressed representations.
  Although such methods produce impressive results, they are prone to generate
  audible artifacts when the conditioning is flawed or imperfect. An alternative
  modeling approach is to use diffusion models. However, these have mainly been
  used as speech vocoders (i.e., conditioned on mel-spectrograms) or generating
  relatively low sampling rate signals. In this work, we propose a high-fidelity
  multi-band diffusion-based framework that generates any type of audio modality
  (e.g., speech, music, environmental sounds) from low-bitrate discrete
  representations. At equal bit rate, the proposed approach outperforms
  state-of-the-art generative techniques in terms of perceptual quality. Training
  and, evaluation code, along with audio samples, are available on the
  facebookresearch/audiocraft Github page.
paper_authors._id: 64d1cea7228324a28b17be2f
paper_authors.name: Robin San Roman
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-31T11:48:08.814Z
paper_authors._id: 65ba260b3167e14767093ca7
paper_authors.avatarUrl: /avatars/f23fa3e8429ac603dd512cc52481a9bc.svg
paper_authors.fullname: Robin San Roman
paper_authors.type: user
paper_authors.user: Sparker17
paper_authors._id: 64d1cea7228324a28b17be30
paper_authors.name: Yossi Adi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-08T15:14:38.465Z
paper_authors._id: 6481e135578646b5c2386728
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6481e135578646b5c2386728/SPva4iNw0pORiCXD45cx9.jpeg
paper_authors.fullname: Yossi Adi
paper_authors.type: user
paper_authors.user: adiyoss
paper_authors._id: 64d1cea7228324a28b17be31
paper_authors.name: Antoine Deleforge
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-09-26T13:02:17.807Z
paper_authors._id: 6512a992ec7fac9ec22566d1
paper_authors.avatarUrl: /avatars/61df825d651f5950b27adc814db658a7.svg
paper_authors.fullname: Antoine Deleforge
paper_authors.type: user
paper_authors.user: AntoineDeleforge
paper_authors._id: 64d1cea7228324a28b17be32
paper_authors.name: Romain Serizel
paper_authors._id: 64d1cea7228324a28b17be33
paper_authors.name: Gabriel Synnaeve
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-08T15:15:07.921Z
paper_authors._id: 630eac7931970d1cd4fbacf2
paper_authors.avatarUrl: /avatars/b7ccbddfa745db854dc342be1327cd53.svg
paper_authors.fullname: Gabriel Synnaeve
paper_authors.type: user
paper_authors.user: gsynnaeve
paper_authors._id: 64d1cea7228324a28b17be34
paper_authors.name: Alexandre Défossez
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-08T15:15:16.444Z
paper_authors._id: 6357f5f50dce5559211a3503
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1666708948380-noauth.jpeg
paper_authors.fullname: Alexandre Défossez
paper_authors.type: user
paper_authors.user: adefossez
paper_title: From Discrete Tokens to High-Fidelity Audio Using Multi-Band Diffusion
paper_summary: Deep generative models can generate high-fidelity audio conditioned on
  various types of representations (e.g., mel-spectrograms, Mel-frequency
  Cepstral Coefficients (MFCC)). Recently, such models have been used to
  synthesize audio waveforms conditioned on highly compressed representations.
  Although such methods produce impressive results, they are prone to generate
  audible artifacts when the conditioning is flawed or imperfect. An alternative
  modeling approach is to use diffusion models. However, these have mainly been
  used as speech vocoders (i.e., conditioned on mel-spectrograms) or generating
  relatively low sampling rate signals. In this work, we propose a high-fidelity
  multi-band diffusion-based framework that generates any type of audio modality
  (e.g., speech, music, environmental sounds) from low-bitrate discrete
  representations. At equal bit rate, the proposed approach outperforms
  state-of-the-art generative techniques in terms of perceptual quality. Training
  and, evaluation code, along with audio samples, are available on the
  facebookresearch/audiocraft Github page.
paper_ai_summary: A high-fidelity multi-band diffusion model generates high-quality audio from low-bitrate representations, outperforming existing techniques in perceptual quality.
paper_ai_keywords: mel-spectrograms Mel-frequency Cepstral Coefficients (MFCC) diffusion models speech vocoders multi-band diffusion-based framework audio modality perceptual quality