Towards Diverse and Efficient Audio Captioning via Diffusion Models
We introduce Diffusion-based Audio Captioning (DAC), a non-autoregressive
diffusion model tailored for diverse and efficient audio captioning. Although
existing captioning models relying on language backbones have achieved
remarkable success in various captioning tasks, their insufficient performance
in terms of generation speed and diversity impede progress in audio
understanding and multimedia applications. Our diffusion-based framework offers
unique advantages stemming from its inherent stochasticity and holistic context
modeling in captioning. Through rigorous evaluation, we demonstrate that DAC
not only achieves SOTA performance levels compared to existing benchmarks in
the caption quality, but also significantly outperforms them in terms of
generation speed and diversity. The success of DAC illustrates that text
generation can also be seamlessly integrated with audio and visual generation
tasks using a diffusion backbone, paving the way for a unified, audio-related
generative model across different modalities.
66eb89cf88f1d02146196ce7
Manjie Xu
66eb89cf88f1d02146196ce8
Chenxing Li
extracted_confirmed
2024-09-24T06:44:00.159Z
64cc9972e60d2cddfadc5c35
/avatars/5dac96288fa93fa965f1b57ad8d0a9b7.svg
Chenxing
user
Chenxinglili
66eb89cf88f1d02146196ce9
Xinyi Tu
66eb89cf88f1d02146196cea
Yong Ren
66eb89cf88f1d02146196ceb
Ruibo Fu
66eb89cf88f1d02146196cec
Wei Liang
66eb89cf88f1d02146196ced
Dong Yu
Towards Diverse and Efficient Audio Captioning via Diffusion Models
We introduce Diffusion-based Audio Captioning (DAC), a non-autoregressive
diffusion model tailored for diverse and efficient audio captioning. Although
existing captioning models relying on language backbones have achieved
remarkable success in various captioning tasks, their insufficient performance
in terms of generation speed and diversity impede progress in audio
understanding and multimedia applications. Our diffusion-based framework offers
unique advantages stemming from its inherent stochasticity and holistic context
modeling in captioning. Through rigorous evaluation, we demonstrate that DAC
not only achieves SOTA performance levels compared to existing benchmarks in
the caption quality, but also significantly outperforms them in terms of
generation speed and diversity. The success of DAC illustrates that text
generation can also be seamlessly integrated with audio and visual generation
tasks using a diffusion backbone, paving the way for a unified, audio-related
generative model across different modalities.
Diffusion-based Audio Captioning achieves state-of-the-art performance in caption quality and surpasses existing models in generation speed and diversity for audio captioning tasks.
diffusion model non-autoregressive stochasticity holistic context modeling SOTA performance