title: Towards Diverse and Efficient Audio Captioning via Diffusion Models
summary: We introduce Diffusion-based Audio Captioning (DAC), a non-autoregressive
  diffusion model tailored for diverse and efficient audio captioning. Although
  existing captioning models relying on language backbones have achieved
  remarkable success in various captioning tasks, their insufficient performance
  in terms of generation speed and diversity impede progress in audio
  understanding and multimedia applications. Our diffusion-based framework offers
  unique advantages stemming from its inherent stochasticity and holistic context
  modeling in captioning. Through rigorous evaluation, we demonstrate that DAC
  not only achieves SOTA performance levels compared to existing benchmarks in
  the caption quality, but also significantly outperforms them in terms of
  generation speed and diversity. The success of DAC illustrates that text
  generation can also be seamlessly integrated with audio and visual generation
  tasks using a diffusion backbone, paving the way for a unified, audio-related
  generative model across different modalities.
paper_authors._id: 66eb89cf88f1d02146196ce7
paper_authors.name: Manjie Xu
paper_authors._id: 66eb89cf88f1d02146196ce8
paper_authors.name: Chenxing Li
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-09-24T06:44:00.159Z
paper_authors._id: 64cc9972e60d2cddfadc5c35
paper_authors.avatarUrl: /avatars/5dac96288fa93fa965f1b57ad8d0a9b7.svg
paper_authors.fullname: Chenxing
paper_authors.type: user
paper_authors.user: Chenxinglili
paper_authors._id: 66eb89cf88f1d02146196ce9
paper_authors.name: Xinyi Tu
paper_authors._id: 66eb89cf88f1d02146196cea
paper_authors.name: Yong Ren
paper_authors._id: 66eb89cf88f1d02146196ceb
paper_authors.name: Ruibo Fu
paper_authors._id: 66eb89cf88f1d02146196cec
paper_authors.name: Wei Liang
paper_authors._id: 66eb89cf88f1d02146196ced
paper_authors.name: Dong Yu
paper_title: Towards Diverse and Efficient Audio Captioning via Diffusion Models
paper_summary: We introduce Diffusion-based Audio Captioning (DAC), a non-autoregressive
  diffusion model tailored for diverse and efficient audio captioning. Although
  existing captioning models relying on language backbones have achieved
  remarkable success in various captioning tasks, their insufficient performance
  in terms of generation speed and diversity impede progress in audio
  understanding and multimedia applications. Our diffusion-based framework offers
  unique advantages stemming from its inherent stochasticity and holistic context
  modeling in captioning. Through rigorous evaluation, we demonstrate that DAC
  not only achieves SOTA performance levels compared to existing benchmarks in
  the caption quality, but also significantly outperforms them in terms of
  generation speed and diversity. The success of DAC illustrates that text
  generation can also be seamlessly integrated with audio and visual generation
  tasks using a diffusion backbone, paving the way for a unified, audio-related
  generative model across different modalities.
paper_ai_keywords: diffusion-based audio captioning DAC (Diffusion-based Audio Captioning) non-autoregressive diffusion model captioning tasks language backbones generation speed diversity audio understanding multimedia applications diffusion-based framework stochasticity holistic context modeling SOTA performance levels caption quality generation speed generated text generation unified generative model different modalities