title: Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation
summary: Large diffusion models have been successful in text-to-audio (T2A) synthesis
  tasks, but they often suffer from common issues such as semantic misalignment
  and poor temporal consistency due to limited natural language understanding and
  data scarcity. Additionally, 2D spatial structures widely used in T2A works
  lead to unsatisfactory audio quality when generating variable-length audio
  samples since they do not adequately prioritize temporal information. To
  address these challenges, we propose Make-an-Audio 2, a latent diffusion-based
  T2A method that builds on the success of Make-an-Audio. Our approach includes
  several techniques to improve semantic alignment and temporal consistency:
  Firstly, we use pre-trained large language models (LLMs) to parse the text into
  structured <event & order> pairs for better temporal information capture. We
  also introduce another structured-text encoder to aid in learning semantic
  alignment during the diffusion denoising process. To improve the performance of
  variable length generation and enhance the temporal information extraction, we
  design a feed-forward Transformer-based diffusion denoiser. Finally, we use
  LLMs to augment and transform a large amount of audio-label data into
  audio-text datasets to alleviate the problem of scarcity of temporal data.
  Extensive experiments show that our method outperforms baseline models in both
  objective and subjective metrics, and achieves significant gains in temporal
  information understanding, semantic consistency, and sound quality.
paper_authors._id: 6476b818e9b568d69fc12a95
paper_authors.name: Jiawei Huang
paper_authors._id: 6476b818e9b568d69fc12a96
paper_authors.name: Yi Ren
paper_authors._id: 6476b818e9b568d69fc12a97
paper_authors.name: Rongjie Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-31T07:24:58.620Z
paper_authors._id: 62f6556e04e5e02f82ac8531
paper_authors.avatarUrl: /avatars/01024815aeaaf8c4721e937b3db68903.svg
paper_authors.fullname: Rongjiehuang
paper_authors.type: user
paper_authors.user: Rongjiehuang
paper_authors._id: 6476b818e9b568d69fc12a98
paper_authors.name: Dongchao Yang
paper_authors._id: 6476b818e9b568d69fc12a99
paper_authors.name: Zhenhui Ye
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-05-31T12:01:13.350Z
paper_authors._id: 6277fcd8c5c547c31b80cc77
paper_authors.avatarUrl: /avatars/7dab3f5c879b5c5eb86f822119a9b5bb.svg
paper_authors.fullname: Zhenhui Ye
paper_authors.type: user
paper_authors.user: yerfor
paper_authors._id: 6476b818e9b568d69fc12a9a
paper_authors.name: Chen Zhang
paper_authors._id: 6476b818e9b568d69fc12a9b
paper_authors.name: Jinglin Liu
paper_authors._id: 6476b818e9b568d69fc12a9c
paper_authors.name: Xiang Yin
paper_authors._id: 6476b818e9b568d69fc12a9d
paper_authors.name: Zejun Ma
paper_authors._id: 6476b818e9b568d69fc12a9e
paper_authors.name: Zhou Zhao
paper_title: Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation
paper_summary: Large diffusion models have been successful in text-to-audio (T2A) synthesis
  tasks, but they often suffer from common issues such as semantic misalignment
  and poor temporal consistency due to limited natural language understanding and
  data scarcity. Additionally, 2D spatial structures widely used in T2A works
  lead to unsatisfactory audio quality when generating variable-length audio
  samples since they do not adequately prioritize temporal information. To
  address these challenges, we propose Make-an-Audio 2, a latent diffusion-based
  T2A method that builds on the success of Make-an-Audio. Our approach includes
  several techniques to improve semantic alignment and temporal consistency:
  Firstly, we use pre-trained large language models (LLMs) to parse the text into
  structured <event & order> pairs for better temporal information capture. We
  also introduce another structured-text encoder to aid in learning semantic
  alignment during the diffusion denoising process. To improve the performance of
  variable length generation and enhance the temporal information extraction, we
  design a feed-forward Transformer-based diffusion denoiser. Finally, we use
  LLMs to augment and transform a large amount of audio-label data into
  audio-text datasets to alleviate the problem of scarcity of temporal data.
  Extensive experiments show that our method outperforms baseline models in both
  objective and subjective metrics, and achieves significant gains in temporal
  information understanding, semantic consistency, and sound quality.
paper_ai_keywords: latent diffusion-based Structured &lt;event &amp; order&gt; pairs structured-text encoder feed-forward Transformer-based diffusion denoiser pre-trained large language models (LLMs) audio-label data audio-text datasets