MusicHiFi: Fast High-Fidelity Stereo Vocoding
Diffusion-based audio and music generation models commonly generate music by
constructing an image representation of audio (e.g., a mel-spectrogram) and
then converting it to audio using a phase reconstruction model or vocoder.
Typical vocoders, however, produce monophonic audio at lower resolutions (e.g.,
16-24 kHz), which limits their effectiveness. We propose MusicHiFi -- an
efficient high-fidelity stereophonic vocoder. Our method employs a cascade of
three generative adversarial networks (GANs) that convert low-resolution
mel-spectrograms to audio, upsamples to high-resolution audio via bandwidth
expansion, and upmixes to stereophonic audio. Compared to previous work, we
propose 1) a unified GAN-based generator and discriminator architecture and
training procedure for each stage of our cascade, 2) a new fast, near
downsampling-compatible bandwidth extension module, and 3) a new fast
downmix-compatible mono-to-stereo upmixer that ensures the preservation of
monophonic content in the output. We evaluate our approach using both objective
and subjective listening tests and find our approach yields comparable or
better audio quality, better spatialization control, and significantly faster
inference speed compared to past work. Sound examples are at
https://MusicHiFi.github.io/web/.
65f7a0f8d64a62e375b1f4f3
Ge Zhu
admin_assigned
2024-03-18T09:11:02.008Z
6526f16b00040a5395b82c96
/avatars/592366207bc919a6984b3a0d816314c7.svg
Ge Zhu
user
gzhu06
65f7a0f8d64a62e375b1f4f4
Juan-Pablo Caceres
65f7a0f8d64a62e375b1f4f5
Zhiyao Duan
admin_assigned
2024-03-18T09:11:26.168Z
6556f3c3ee35f7d8bcf7706d
/avatars/423682fbf354f085232237cbf0f51bbe.svg
Zhiyao Duan
user
zhiyaoduan
65f7a0f8d64a62e375b1f4f6
Nicholas J. Bryan
admin_assigned
2024-03-18T09:11:32.465Z
648119ee7a741c7f33f49f25
/avatars/183d2143ba20e1cc8712c63c055aadd7.svg
Nicholas J. Bryan
user
Njb
MusicHiFi: Fast High-Fidelity Stereo Vocoding
Diffusion-based audio and music generation models commonly generate music by
constructing an image representation of audio (e.g., a mel-spectrogram) and
then converting it to audio using a phase reconstruction model or vocoder.
Typical vocoders, however, produce monophonic audio at lower resolutions (e.g.,
16-24 kHz), which limits their effectiveness. We propose MusicHiFi -- an
efficient high-fidelity stereophonic vocoder. Our method employs a cascade of
three generative adversarial networks (GANs) that convert low-resolution
mel-spectrograms to audio, upsamples to high-resolution audio via bandwidth
expansion, and upmixes to stereophonic audio. Compared to previous work, we
propose 1) a unified GAN-based generator and discriminator architecture and
training procedure for each stage of our cascade, 2) a new fast, near
downsampling-compatible bandwidth extension module, and 3) a new fast
downmix-compatible mono-to-stereo upmixer that ensures the preservation of
monophonic content in the output. We evaluate our approach using both objective
and subjective listening tests and find our approach yields comparable or
better audio quality, better spatialization control, and significantly faster
inference speed compared to past work. Sound examples are at
https://MusicHiFi.github.io/web/.
MusicHiFi is an efficient high-fidelity stereophonic vocoder using a cascade of GANs to improve audio quality and spatialization control.
diffusion-based audio music generation models mel-spectrogram vocoder generative adversarial networks GANs bandwidth expansion mono-to-stereo upmixer subjective listening tests