title: AudioLDM 2: Learning Holistic Audio Generation with Self-supervised
    Pretraining
summary: Although audio generation shares commonalities across different types of
  audio, such as speech, music, and sound effects, designing models for each type
  requires careful consideration of specific objectives and biases that can
  significantly differ from those of other types. To bring us closer to a unified
  perspective of audio generation, this paper proposes a framework that utilizes
  the same learning method for speech, music, and sound effect generation. Our
  framework introduces a general representation of audio, called language of
  audio (LOA). Any audio can be translated into LOA based on AudioMAE, a
  self-supervised pre-trained representation learning model. In the generation
  process, we translate any modalities into LOA by using a GPT-2 model, and we
  perform self-supervised audio generation learning with a latent diffusion model
  conditioned on LOA. The proposed framework naturally brings advantages such as
  in-context learning abilities and reusable self-supervised pretrained AudioMAE
  and latent diffusion models. Experiments on the major benchmarks of
  text-to-audio, text-to-music, and text-to-speech demonstrate new
  state-of-the-art or competitive performance to previous approaches. Our demo
  and code are available at https://audioldm.github.io/audioldm2.
paper_authors._id: 64d59c3254bb9eb704f0738d
paper_authors.name: Haohe Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-08-11T13:30:27.575Z
paper_authors._id: 6155245d1c762d4d61b51d5d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1677233708065-6155245d1c762d4d61b51d5d.png
paper_authors.fullname: haoheliu
paper_authors.type: user
paper_authors.user: haoheliu
paper_authors._id: 64d59c3254bb9eb704f0738e
paper_authors.name: Qiao Tian
paper_authors._id: 64d59c3254bb9eb704f0738f
paper_authors.name: Yi Yuan
paper_authors._id: 64d59c3254bb9eb704f07390
paper_authors.name: Xubo Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-11T13:42:25.305Z
paper_authors._id: 64c260105f9e98008d498081
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c260105f9e98008d498081/TwJKuPtGMn9WL7T-54WGg.jpeg
paper_authors.fullname: Xubo Liu
paper_authors.type: user
paper_authors.user: Xubo-Liu
paper_authors._id: 64d59c3254bb9eb704f07391
paper_authors.name: Xinhao Mei
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-11T13:43:40.742Z
paper_authors._id: 626c5f8af451470f862534c6
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/626c5f8af451470f862534c6/qL5ubYstO0Won5VOFHviF.jpeg
paper_authors.fullname: Xinhao Mei
paper_authors.type: user
paper_authors.user: Xinhao
paper_authors._id: 64d59c3254bb9eb704f07392
paper_authors.name: Qiuqiang Kong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-08-11T13:44:33.368Z
paper_authors._id: 64311ee3288c97756736a5ca
paper_authors.avatarUrl: /avatars/e6dc01880149450ebc2b9a07c19920cc.svg
paper_authors.fullname: Qiuqiang Kong
paper_authors.type: user
paper_authors.user: qiuqiangkong
paper_authors._id: 64d59c3254bb9eb704f07393
paper_authors.name: Yuping Wang
paper_authors._id: 64d59c3254bb9eb704f07394
paper_authors.name: Wenwu Wang
paper_authors._id: 64d59c3254bb9eb704f07395
paper_authors.name: Yuxuan Wang
paper_authors._id: 64d59c3254bb9eb704f07396
paper_authors.name: Mark D. Plumbley
paper_title: AudioLDM 2: Learning Holistic Audio Generation with Self-supervised
    Pretraining
paper_summary: Although audio generation shares commonalities across different types of
  audio, such as speech, music, and sound effects, designing models for each type
  requires careful consideration of specific objectives and biases that can
  significantly differ from those of other types. To bring us closer to a unified
  perspective of audio generation, this paper proposes a framework that utilizes
  the same learning method for speech, music, and sound effect generation. Our
  framework introduces a general representation of audio, called language of
  audio (LOA). Any audio can be translated into LOA based on AudioMAE, a
  self-supervised pre-trained representation learning model. In the generation
  process, we translate any modalities into LOA by using a GPT-2 model, and we
  perform self-supervised audio generation learning with a latent diffusion model
  conditioned on LOA. The proposed framework naturally brings advantages such as
  in-context learning abilities and reusable self-supervised pretrained AudioMAE
  and latent diffusion models. Experiments on the major benchmarks of
  text-to-audio, text-to-music, and text-to-speech demonstrate new
  state-of-the-art or competitive performance to previous approaches. Our demo
  and code are available at https://audioldm.github.io/audioldm2.
paper_ai_summary: A unified audio generation framework using a language of audio representation, AudioMAE, and latent diffusion model achieves state-of-the-art performance across different audio types.
paper_ai_keywords: language of audio AudioMAE self-supervised pre-trained GPT-2 latent diffusion model in-context learning