FusionAudio-1.2M: Towards Fine-grained Audio Captioning with Multimodal
  Contextual Fusion
High-quality, large-scale audio captioning is crucial for advancing audio
understanding, yet current automated methods often generate captions that lack
fine-grained detail and contextual accuracy, primarily due to their reliance on
limited unimodal or superficial multimodal information. Drawing inspiration
from human auditory perception, which adeptly integrates cross-modal cues and
performs sophisticated auditory scene analysis, we introduce a novel two-stage
automated pipeline. This pipeline first employs specialized pretrained models
to extract diverse contextual cues (e.g., speech, music, general sounds, and
visual information from associated video). A large language model (LLM) then
synthesizes these rich, multimodal inputs to generate detailed and
context-aware audio captions. Key contributions of this work include: (1) the
proposed scalable method for fine-grained audio caption generation; (2)
FusionAudio, a new large-scale dataset comprising 1.2 million such detailed
captions, combined with 6 million QA pairs; and (3) enhanced audio models
developed using FusionAudio, specifically a CLAP-based audio encoder with
superior audio-text alignment and instruction following. This paper paves the
way for more nuanced and accurate automated understanding of complex audio
environments. Code and data can be found in
https://github.com/satsuki2486441738/FusionAudio.
6845b6a33ec10bdd8ab4da1b
Shunian Chen
6845b6a33ec10bdd8ab4da1c
Xinyuan Xie
claimed_verified
2025-06-09T10:11:42.759Z
66440e86bfe15e84d369cb03
/avatars/d15b3b3831bc74138206071612169f64.svg
Xinyuan Xie
user
SatsukiVie
6845b6a33ec10bdd8ab4da1d
Zheshu Chen
6845b6a33ec10bdd8ab4da1e
Liyan Zhao
6845b6a33ec10bdd8ab4da1f
Owen Lee
6845b6a33ec10bdd8ab4da20
Zhan Su
6845b6a33ec10bdd8ab4da21
Qilin Sun
6845b6a33ec10bdd8ab4da22
Benyou Wang
FusionAudio-1.2M: Towards Fine-grained Audio Captioning with Multimodal
  Contextual Fusion
High-quality, large-scale audio captioning is crucial for advancing audio
understanding, yet current automated methods often generate captions that lack
fine-grained detail and contextual accuracy, primarily due to their reliance on
limited unimodal or superficial multimodal information. Drawing inspiration
from human auditory perception, which adeptly integrates cross-modal cues and
performs sophisticated auditory scene analysis, we introduce a novel two-stage
automated pipeline. This pipeline first employs specialized pretrained models
to extract diverse contextual cues (e.g., speech, music, general sounds, and
visual information from associated video). A large language model (LLM) then
synthesizes these rich, multimodal inputs to generate detailed and
context-aware audio captions. Key contributions of this work include: (1) the
proposed scalable method for fine-grained audio caption generation; (2)
FusionAudio, a new large-scale dataset comprising 1.2 million such detailed
captions, combined with 6 million QA pairs; and (3) enhanced audio models
developed using FusionAudio, specifically a CLAP-based audio encoder with
superior audio-text alignment and instruction following. This paper paves the
way for more nuanced and accurate automated understanding of complex audio
environments. Code and data can be found in
https://github.com/satsuki2486441738/FusionAudio.
A novel two-stage pipeline using specialized pretrained models and a large language model enhances audio caption quality by integrating diverse multimodal cues and contextual information.
audio captioning auditory perception auditory scene analysis pretrained models large language model FusionAudio CLAP-based audio encoder audio-text alignment instruction following