title: Retrieval-Augmented Text-to-Audio Generation
summary: Despite recent progress in text-to-audio (TTA) generation, we show that the
  state-of-the-art models, such as AudioLDM, trained on datasets with an
  imbalanced class distribution, such as AudioCaps, are biased in their
  generation performance. Specifically, they excel in generating common audio
  classes while underperforming in the rare ones, thus degrading the overall
  generation performance. We refer to this problem as long-tailed text-to-audio
  generation. To address this issue, we propose a simple retrieval-augmented
  approach for TTA models. Specifically, given an input text prompt, we first
  leverage a Contrastive Language Audio Pretraining (CLAP) model to retrieve
  relevant text-audio pairs. The features of the retrieved audio-text data are
  then used as additional conditions to guide the learning of TTA models. We
  enhance AudioLDM with our proposed approach and denote the resulting augmented
  system as Re-AudioLDM. On the AudioCaps dataset, Re-AudioLDM achieves a
  state-of-the-art Frechet Audio Distance (FAD) of 1.37, outperforming the
  existing approaches by a large margin. Furthermore, we show that Re-AudioLDM
  can generate realistic audio for complex scenes, rare audio classes, and even
  unseen audio types, indicating its potential in TTA tasks.
paper_authors._id: 6507e3b5c6ae3df8f2e9f42b
paper_authors.name: Yi Yuan
paper_authors._id: 6507e3b5c6ae3df8f2e9f42c
paper_authors.name: Haohe Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-18T10:25:20.182Z
paper_authors._id: 6155245d1c762d4d61b51d5d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1677233708065-6155245d1c762d4d61b51d5d.png
paper_authors.fullname: haoheliu
paper_authors.type: user
paper_authors.user: haoheliu
paper_authors._id: 6507e3b5c6ae3df8f2e9f42d
paper_authors.name: Xubo Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-18T10:25:28.629Z
paper_authors._id: 64c260105f9e98008d498081
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64c260105f9e98008d498081/TwJKuPtGMn9WL7T-54WGg.jpeg
paper_authors.fullname: Xubo Liu
paper_authors.type: user
paper_authors.user: Xubo-Liu
paper_authors._id: 6507e3b5c6ae3df8f2e9f42e
paper_authors.name: Qiushi Huang
paper_authors._id: 6507e3b5c6ae3df8f2e9f42f
paper_authors.name: Mark D. Plumbley
paper_authors._id: 6507e3b5c6ae3df8f2e9f430
paper_authors.name: Wenwu Wang
paper_title: Retrieval-Augmented Text-to-Audio Generation
paper_summary: Despite recent progress in text-to-audio (TTA) generation, we show that the
  state-of-the-art models, such as AudioLDM, trained on datasets with an
  imbalanced class distribution, such as AudioCaps, are biased in their
  generation performance. Specifically, they excel in generating common audio
  classes while underperforming in the rare ones, thus degrading the overall
  generation performance. We refer to this problem as long-tailed text-to-audio
  generation. To address this issue, we propose a simple retrieval-augmented
  approach for TTA models. Specifically, given an input text prompt, we first
  leverage a Contrastive Language Audio Pretraining (CLAP) model to retrieve
  relevant text-audio pairs. The features of the retrieved audio-text data are
  then used as additional conditions to guide the learning of TTA models. We
  enhance AudioLDM with our proposed approach and denote the resulting augmented
  system as Re-AudioLDM. On the AudioCaps dataset, Re-AudioLDM achieves a
  state-of-the-art Frechet Audio Distance (FAD) of 1.37, outperforming the
  existing approaches by a large margin. Furthermore, we show that Re-AudioLDM
  can generate realistic audio for complex scenes, rare audio classes, and even
  unseen audio types, indicating its potential in TTA tasks.
paper_ai_summary: A retrieval-augmented approach enhances text-to-audio generation by addressing class imbalance and improving generation performance for rare audio classes.
paper_ai_keywords: Contrastive Language Audio Pretraining (CLAP) Frechet Audio Distance (FAD) text-to-audio generation long-tailed text-to-audio generation retrieval-augmented approach