EzAudio: Enhancing Text-to-Audio Generation with Efficient Diffusion
  Transformer
Latent diffusion models have shown promising results in text-to-audio (T2A)
generation tasks, yet previous models have encountered difficulties in
generation quality, computational cost, diffusion sampling, and data
preparation. In this paper, we introduce EzAudio, a transformer-based T2A
diffusion model, to handle these challenges. Our approach includes several key
innovations: (1) We build the T2A model on the latent space of a 1D waveform
Variational Autoencoder (VAE), avoiding the complexities of handling 2D
spectrogram representations and using an additional neural vocoder. (2) We
design an optimized diffusion transformer architecture specifically tailored
for audio latent representations and diffusion modeling, which enhances
convergence speed, training stability, and memory usage, making the training
process easier and more efficient. (3) To tackle data scarcity, we adopt a
data-efficient training strategy that leverages unlabeled data for learning
acoustic dependencies, audio caption data annotated by audio-language models
for text-to-audio alignment learning, and human-labeled data for fine-tuning.
(4) We introduce a classifier-free guidance (CFG) rescaling method that
simplifies EzAudio by achieving strong prompt alignment while preserving great
audio quality when using larger CFG scores, eliminating the need to struggle
with finding the optimal CFG score to balance this trade-off. EzAudio surpasses
existing open-source models in both objective metrics and subjective
evaluations, delivering realistic listening experiences while maintaining a
streamlined model structure, low training costs, and an easy-to-follow training
pipeline. Code, data, and pre-trained models are released at:
https://haidog-yaqub.github.io/EzAudio-Page/.
66ea3ec064583d8d2f541f85
Jiarui Hai
admin_assigned
2024-09-18T08:10:16.288Z
6363e9e4969bdae89e10b5ac
https://cdn-avatars.huggingface.co/v1/production/uploads/6363e9e4969bdae89e10b5ac/uR0jz8pBJIQPmKC90OJgI.jpeg
Jiarui Hai
user
Higobeatz
66ea3ec064583d8d2f541f86
Yong Xu
66ea3ec064583d8d2f541f87
Hao Zhang
claimed_verified
2024-09-19T07:07:07.398Z
66eb130fe7ae2c45506e18ab
/avatars/296267230b201eab51837052e8662107.svg
Hao Zhang
user
GrandpaDeer
66ea3ec064583d8d2f541f88
Chenxing Li
admin_assigned
2024-09-18T08:15:29.422Z
64cc9972e60d2cddfadc5c35
/avatars/5dac96288fa93fa965f1b57ad8d0a9b7.svg
Chenxing
user
Chenxinglili
66ea3ec064583d8d2f541f89
Helin Wang
claimed_verified
2024-09-19T13:26:24.939Z
63ecfb5ec5b3c734085db9ed
/avatars/0b1d03dcd7997ad1daa764fb76f88993.svg
Helin Wang
user
westbrook
66ea3ec064583d8d2f541f8a
Mounya Elhilali
66ea3ec064583d8d2f541f8b
Dong Yu
EzAudio: Enhancing Text-to-Audio Generation with Efficient Diffusion
  Transformer
Latent diffusion models have shown promising results in text-to-audio (T2A)
generation tasks, yet previous models have encountered difficulties in
generation quality, computational cost, diffusion sampling, and data
preparation. In this paper, we introduce EzAudio, a transformer-based T2A
diffusion model, to handle these challenges. Our approach includes several key
innovations: (1) We build the T2A model on the latent space of a 1D waveform
Variational Autoencoder (VAE), avoiding the complexities of handling 2D
spectrogram representations and using an additional neural vocoder. (2) We
design an optimized diffusion transformer architecture specifically tailored
for audio latent representations and diffusion modeling, which enhances
convergence speed, training stability, and memory usage, making the training
process easier and more efficient. (3) To tackle data scarcity, we adopt a
data-efficient training strategy that leverages unlabeled data for learning
acoustic dependencies, audio caption data annotated by audio-language models
for text-to-audio alignment learning, and human-labeled data for fine-tuning.
(4) We introduce a classifier-free guidance (CFG) rescaling method that
simplifies EzAudio by achieving strong prompt alignment while preserving great
audio quality when using larger CFG scores, eliminating the need to struggle
with finding the optimal CFG score to balance this trade-off. EzAudio surpasses
existing open-source models in both objective metrics and subjective
evaluations, delivering realistic listening experiences while maintaining a
streamlined model structure, low training costs, and an easy-to-follow training
pipeline. Code, data, and pre-trained models are released at:
https://haidog-yaqub.github.io/EzAudio-Page/.
EzAudio, a transformer-based diffusion model operating in the latent space of a 1D waveform VAE, enhances text-to-audio generation by improving quality, efficiency, and data utilization through various optimizations.
latent diffusion models text-to-audio (T2A) Variational Autoencoder (VAE) diffusion transformer data-efficient training classifier-free guidance (CFG) acoustic dependencies audio caption data human-labeled data fine-tuning