Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation
Large diffusion models have been successful in text-to-audio (T2A) synthesis
tasks, but they often suffer from common issues such as semantic misalignment
and poor temporal consistency due to limited natural language understanding and
data scarcity. Additionally, 2D spatial structures widely used in T2A works
lead to unsatisfactory audio quality when generating variable-length audio
samples since they do not adequately prioritize temporal information. To
address these challenges, we propose Make-an-Audio 2, a latent diffusion-based
T2A method that builds on the success of Make-an-Audio. Our approach includes
several techniques to improve semantic alignment and temporal consistency:
Firstly, we use pre-trained large language models (LLMs) to parse the text into
structured <event & order> pairs for better temporal information capture. We
also introduce another structured-text encoder to aid in learning semantic
alignment during the diffusion denoising process. To improve the performance of
variable length generation and enhance the temporal information extraction, we
design a feed-forward Transformer-based diffusion denoiser. Finally, we use
LLMs to augment and transform a large amount of audio-label data into
audio-text datasets to alleviate the problem of scarcity of temporal data.
Extensive experiments show that our method outperforms baseline models in both
objective and subjective metrics, and achieves significant gains in temporal
information understanding, semantic consistency, and sound quality.
6476b818e9b568d69fc12a95
Jiawei Huang
6476b818e9b568d69fc12a96
Yi Ren
6476b818e9b568d69fc12a97
Rongjie Huang
claimed_verified
2023-05-31T07:24:58.620Z
62f6556e04e5e02f82ac8531
/avatars/01024815aeaaf8c4721e937b3db68903.svg
Rongjiehuang
user
Rongjiehuang
6476b818e9b568d69fc12a98
Dongchao Yang
6476b818e9b568d69fc12a99
Zhenhui Ye
claimed_verified
2023-05-31T12:01:13.350Z
6277fcd8c5c547c31b80cc77
/avatars/7dab3f5c879b5c5eb86f822119a9b5bb.svg
Zhenhui Ye
user
yerfor
6476b818e9b568d69fc12a9a
Chen Zhang
6476b818e9b568d69fc12a9b
Jinglin Liu
6476b818e9b568d69fc12a9c
Xiang Yin
6476b818e9b568d69fc12a9d
Zejun Ma
6476b818e9b568d69fc12a9e
Zhou Zhao
Make-An-Audio 2: Temporal-Enhanced Text-to-Audio Generation
Large diffusion models have been successful in text-to-audio (T2A) synthesis
tasks, but they often suffer from common issues such as semantic misalignment
and poor temporal consistency due to limited natural language understanding and
data scarcity. Additionally, 2D spatial structures widely used in T2A works
lead to unsatisfactory audio quality when generating variable-length audio
samples since they do not adequately prioritize temporal information. To
address these challenges, we propose Make-an-Audio 2, a latent diffusion-based
T2A method that builds on the success of Make-an-Audio. Our approach includes
several techniques to improve semantic alignment and temporal consistency:
Firstly, we use pre-trained large language models (LLMs) to parse the text into
structured <event & order> pairs for better temporal information capture. We
also introduce another structured-text encoder to aid in learning semantic
alignment during the diffusion denoising process. To improve the performance of
variable length generation and enhance the temporal information extraction, we
design a feed-forward Transformer-based diffusion denoiser. Finally, we use
LLMs to augment and transform a large amount of audio-label data into
audio-text datasets to alleviate the problem of scarcity of temporal data.
Extensive experiments show that our method outperforms baseline models in both
objective and subjective metrics, and achieves significant gains in temporal
information understanding, semantic consistency, and sound quality.
Make-an-Audio 2, a latent diffusion-based text-to-audio synthesis method, improves semantic alignment and temporal consistency through pre-trained language models, structured-text encoding, a Transformer-based diffusion denoiser, and data augmentation.
latent diffusion text-to-audio (T2A) pre-trained large language models (LLMs) structured-text encoder feed-forward Transformer-based diffusion denoiser