title: EzAudio: Enhancing Text-to-Audio Generation with Efficient Diffusion
    Transformer
summary: Latent diffusion models have shown promising results in text-to-audio (T2A)
  generation tasks, yet previous models have encountered difficulties in
  generation quality, computational cost, diffusion sampling, and data
  preparation. In this paper, we introduce EzAudio, a transformer-based T2A
  diffusion model, to handle these challenges. Our approach includes several key
  innovations: (1) We build the T2A model on the latent space of a 1D waveform
  Variational Autoencoder (VAE), avoiding the complexities of handling 2D
  spectrogram representations and using an additional neural vocoder. (2) We
  design an optimized diffusion transformer architecture specifically tailored
  for audio latent representations and diffusion modeling, which enhances
  convergence speed, training stability, and memory usage, making the training
  process easier and more efficient. (3) To tackle data scarcity, we adopt a
  data-efficient training strategy that leverages unlabeled data for learning
  acoustic dependencies, audio caption data annotated by audio-language models
  for text-to-audio alignment learning, and human-labeled data for fine-tuning.
  (4) We introduce a classifier-free guidance (CFG) rescaling method that
  simplifies EzAudio by achieving strong prompt alignment while preserving great
  audio quality when using larger CFG scores, eliminating the need to struggle
  with finding the optimal CFG score to balance this trade-off. EzAudio surpasses
  existing open-source models in both objective metrics and subjective
  evaluations, delivering realistic listening experiences while maintaining a
  streamlined model structure, low training costs, and an easy-to-follow training
  pipeline. Code, data, and pre-trained models are released at:
  https://haidog-yaqub.github.io/EzAudio-Page/.
paper_authors._id: 66ea3ec064583d8d2f541f85
paper_authors.name: Jiarui Hai
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-18T08:10:16.288Z
paper_authors._id: 6363e9e4969bdae89e10b5ac
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6363e9e4969bdae89e10b5ac/uR0jz8pBJIQPmKC90OJgI.jpeg
paper_authors.fullname: Jiarui Hai
paper_authors.type: user
paper_authors.user: Higobeatz
paper_authors._id: 66ea3ec064583d8d2f541f86
paper_authors.name: Yong Xu
paper_authors._id: 66ea3ec064583d8d2f541f87
paper_authors.name: Hao Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-19T07:07:07.398Z
paper_authors._id: 66eb130fe7ae2c45506e18ab
paper_authors.avatarUrl: /avatars/296267230b201eab51837052e8662107.svg
paper_authors.fullname: Hao Zhang
paper_authors.type: user
paper_authors.user: GrandpaDeer
paper_authors._id: 66ea3ec064583d8d2f541f88
paper_authors.name: Chenxing Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-09-18T08:15:29.422Z
paper_authors._id: 64cc9972e60d2cddfadc5c35
paper_authors.avatarUrl: /avatars/5dac96288fa93fa965f1b57ad8d0a9b7.svg
paper_authors.fullname: Chenxing
paper_authors.type: user
paper_authors.user: Chenxinglili
paper_authors._id: 66ea3ec064583d8d2f541f89
paper_authors.name: Helin Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-09-19T13:26:24.939Z
paper_authors._id: 63ecfb5ec5b3c734085db9ed
paper_authors.avatarUrl: /avatars/0b1d03dcd7997ad1daa764fb76f88993.svg
paper_authors.fullname: Helin Wang
paper_authors.type: user
paper_authors.user: westbrook
paper_authors._id: 66ea3ec064583d8d2f541f8a
paper_authors.name: Mounya Elhilali
paper_authors._id: 66ea3ec064583d8d2f541f8b
paper_authors.name: Dong Yu
paper_title: EzAudio: Enhancing Text-to-Audio Generation with Efficient Diffusion
    Transformer
paper_summary: Latent diffusion models have shown promising results in text-to-audio (T2A)
  generation tasks, yet previous models have encountered difficulties in
  generation quality, computational cost, diffusion sampling, and data
  preparation. In this paper, we introduce EzAudio, a transformer-based T2A
  diffusion model, to handle these challenges. Our approach includes several key
  innovations: (1) We build the T2A model on the latent space of a 1D waveform
  Variational Autoencoder (VAE), avoiding the complexities of handling 2D
  spectrogram representations and using an additional neural vocoder. (2) We
  design an optimized diffusion transformer architecture specifically tailored
  for audio latent representations and diffusion modeling, which enhances
  convergence speed, training stability, and memory usage, making the training
  process easier and more efficient. (3) To tackle data scarcity, we adopt a
  data-efficient training strategy that leverages unlabeled data for learning
  acoustic dependencies, audio caption data annotated by audio-language models
  for text-to-audio alignment learning, and human-labeled data for fine-tuning.
  (4) We introduce a classifier-free guidance (CFG) rescaling method that
  simplifies EzAudio by achieving strong prompt alignment while preserving great
  audio quality when using larger CFG scores, eliminating the need to struggle
  with finding the optimal CFG score to balance this trade-off. EzAudio surpasses
  existing open-source models in both objective metrics and subjective
  evaluations, delivering realistic listening experiences while maintaining a
  streamlined model structure, low training costs, and an easy-to-follow training
  pipeline. Code, data, and pre-trained models are released at:
  https://haidog-yaqub.github.io/EzAudio-Page/.
paper_ai_summary: EzAudio, a transformer-based diffusion model operating in the latent space of a 1D waveform VAE, enhances text-to-audio generation by improving quality, efficiency, and data utilization through various optimizations.
paper_ai_keywords: latent diffusion models text-to-audio (T2A) Variational Autoencoder (VAE) diffusion transformer data-efficient training classifier-free guidance (CFG) acoustic dependencies audio caption data human-labeled data fine-tuning