AudioLDM 2: Learning Holistic Audio Generation with Self-supervised
  Pretraining
Although audio generation shares commonalities across different types of
audio, such as speech, music, and sound effects, designing models for each type
requires careful consideration of specific objectives and biases that can
significantly differ from those of other types. To bring us closer to a unified
perspective of audio generation, this paper proposes a framework that utilizes
the same learning method for speech, music, and sound effect generation. Our
framework introduces a general representation of audio, called language of
audio (LOA). Any audio can be translated into LOA based on AudioMAE, a
self-supervised pre-trained representation learning model. In the generation
process, we translate any modalities into LOA by using a GPT-2 model, and we
perform self-supervised audio generation learning with a latent diffusion model
conditioned on LOA. The proposed framework naturally brings advantages such as
in-context learning abilities and reusable self-supervised pretrained AudioMAE
and latent diffusion models. Experiments on the major benchmarks of
text-to-audio, text-to-music, and text-to-speech demonstrate new
state-of-the-art or competitive performance to previous approaches. Our demo
and code are available at https://audioldm.github.io/audioldm2.
64d59c3254bb9eb704f0738d
Haohe Liu
claimed_verified
2023-08-11T13:30:27.575Z
6155245d1c762d4d61b51d5d
https://cdn-avatars.huggingface.co/v1/production/uploads/1677233708065-6155245d1c762d4d61b51d5d.png
haoheliu
user
haoheliu
64d59c3254bb9eb704f0738e
Qiao Tian
64d59c3254bb9eb704f0738f
Yi Yuan
64d59c3254bb9eb704f07390
Xubo Liu
admin_assigned
2023-08-11T13:42:25.305Z
64c260105f9e98008d498081
https://cdn-avatars.huggingface.co/v1/production/uploads/64c260105f9e98008d498081/TwJKuPtGMn9WL7T-54WGg.jpeg
Xubo Liu
user
Xubo-Liu
64d59c3254bb9eb704f07391
Xinhao Mei
admin_assigned
2023-08-11T13:43:40.742Z
626c5f8af451470f862534c6
https://cdn-avatars.huggingface.co/v1/production/uploads/626c5f8af451470f862534c6/qL5ubYstO0Won5VOFHviF.jpeg
Xinhao Mei
user
Xinhao
64d59c3254bb9eb704f07392
Qiuqiang Kong
admin_assigned
2023-08-11T13:44:33.368Z
64311ee3288c97756736a5ca
/avatars/e6dc01880149450ebc2b9a07c19920cc.svg
Qiuqiang Kong
user
qiuqiangkong
64d59c3254bb9eb704f07393
Yuping Wang
64d59c3254bb9eb704f07394
Wenwu Wang
64d59c3254bb9eb704f07395
Yuxuan Wang
64d59c3254bb9eb704f07396
Mark D. Plumbley
AudioLDM 2: Learning Holistic Audio Generation with Self-supervised
  Pretraining
Although audio generation shares commonalities across different types of
audio, such as speech, music, and sound effects, designing models for each type
requires careful consideration of specific objectives and biases that can
significantly differ from those of other types. To bring us closer to a unified
perspective of audio generation, this paper proposes a framework that utilizes
the same learning method for speech, music, and sound effect generation. Our
framework introduces a general representation of audio, called language of
audio (LOA). Any audio can be translated into LOA based on AudioMAE, a
self-supervised pre-trained representation learning model. In the generation
process, we translate any modalities into LOA by using a GPT-2 model, and we
perform self-supervised audio generation learning with a latent diffusion model
conditioned on LOA. The proposed framework naturally brings advantages such as
in-context learning abilities and reusable self-supervised pretrained AudioMAE
and latent diffusion models. Experiments on the major benchmarks of
text-to-audio, text-to-music, and text-to-speech demonstrate new
state-of-the-art or competitive performance to previous approaches. Our demo
and code are available at https://audioldm.github.io/audioldm2.
A unified audio generation framework using a language of audio representation, AudioMAE, and latent diffusion model achieves state-of-the-art performance across different audio types.
language of audio AudioMAE self-supervised pre-trained GPT-2 latent diffusion model in-context learning