From Discrete Tokens to High-Fidelity Audio Using Multi-Band Diffusion
Deep generative models can generate high-fidelity audio conditioned on
various types of representations (e.g., mel-spectrograms, Mel-frequency
Cepstral Coefficients (MFCC)). Recently, such models have been used to
synthesize audio waveforms conditioned on highly compressed representations.
Although such methods produce impressive results, they are prone to generate
audible artifacts when the conditioning is flawed or imperfect. An alternative
modeling approach is to use diffusion models. However, these have mainly been
used as speech vocoders (i.e., conditioned on mel-spectrograms) or generating
relatively low sampling rate signals. In this work, we propose a high-fidelity
multi-band diffusion-based framework that generates any type of audio modality
(e.g., speech, music, environmental sounds) from low-bitrate discrete
representations. At equal bit rate, the proposed approach outperforms
state-of-the-art generative techniques in terms of perceptual quality. Training
and, evaluation code, along with audio samples, are available on the
facebookresearch/audiocraft Github page.
64d1cea7228324a28b17be2f
Robin San Roman
claimed_verified
2024-01-31T11:48:08.814Z
65ba260b3167e14767093ca7
/avatars/f23fa3e8429ac603dd512cc52481a9bc.svg
Robin San Roman
user
Sparker17
64d1cea7228324a28b17be30
Yossi Adi
admin_assigned
2023-08-08T15:14:38.465Z
6481e135578646b5c2386728
https://cdn-avatars.huggingface.co/v1/production/uploads/6481e135578646b5c2386728/SPva4iNw0pORiCXD45cx9.jpeg
Yossi Adi
user
adiyoss
64d1cea7228324a28b17be31
Antoine Deleforge
claimed_verified
2023-09-26T13:02:17.807Z
6512a992ec7fac9ec22566d1
/avatars/61df825d651f5950b27adc814db658a7.svg
Antoine Deleforge
user
AntoineDeleforge
64d1cea7228324a28b17be32
Romain Serizel
64d1cea7228324a28b17be33
Gabriel Synnaeve
admin_assigned
2023-08-08T15:15:07.921Z
630eac7931970d1cd4fbacf2
/avatars/b7ccbddfa745db854dc342be1327cd53.svg
Gabriel Synnaeve
user
gsynnaeve
64d1cea7228324a28b17be34
Alexandre Défossez
admin_assigned
2023-08-08T15:15:16.444Z
6357f5f50dce5559211a3503
https://cdn-avatars.huggingface.co/v1/production/uploads/1666708948380-noauth.jpeg
Alexandre Défossez
user
adefossez
From Discrete Tokens to High-Fidelity Audio Using Multi-Band Diffusion
Deep generative models can generate high-fidelity audio conditioned on
various types of representations (e.g., mel-spectrograms, Mel-frequency
Cepstral Coefficients (MFCC)). Recently, such models have been used to
synthesize audio waveforms conditioned on highly compressed representations.
Although such methods produce impressive results, they are prone to generate
audible artifacts when the conditioning is flawed or imperfect. An alternative
modeling approach is to use diffusion models. However, these have mainly been
used as speech vocoders (i.e., conditioned on mel-spectrograms) or generating
relatively low sampling rate signals. In this work, we propose a high-fidelity
multi-band diffusion-based framework that generates any type of audio modality
(e.g., speech, music, environmental sounds) from low-bitrate discrete
representations. At equal bit rate, the proposed approach outperforms
state-of-the-art generative techniques in terms of perceptual quality. Training
and, evaluation code, along with audio samples, are available on the
facebookresearch/audiocraft Github page.
A high-fidelity multi-band diffusion model generates high-quality audio from low-bitrate representations, outperforming existing techniques in perceptual quality.
mel-spectrograms Mel-frequency Cepstral Coefficients (MFCC) diffusion models speech vocoders multi-band diffusion-based framework audio modality perceptual quality