Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding
  and Expert Reasoning Abilities
Understanding and reasoning over non-speech sounds and music are crucial for
both humans and AI agents to interact effectively with their environments. In
this paper, we introduce Audio Flamingo 2 (AF2), an Audio-Language Model (ALM)
with advanced audio understanding and reasoning capabilities. AF2 leverages (i)
a custom CLAP model, (ii) synthetic Audio QA data for fine-grained audio
reasoning, and (iii) a multi-stage curriculum learning strategy. AF2 achieves
state-of-the-art performance with only a 3B parameter small language model,
surpassing large open-source and proprietary models across over 20 benchmarks.
Next, for the first time, we extend audio understanding to long audio segments
(30 secs to 5 mins) and propose LongAudio, a large and novel dataset for
training ALMs on long audio captioning and question-answering tasks.
Fine-tuning AF2 on LongAudio leads to exceptional performance on our proposed
LongAudioBench, an expert annotated benchmark for evaluating ALMs on long audio
understanding capabilities. We conduct extensive ablation studies to confirm
the efficacy of our approach. Project Website:
https://research.nvidia.com/labs/adlr/AF2/.
67ca66c1cb7e422997cbd148
Sreyan Ghosh
admin_assigned
2025-03-07T09:40:39.324Z
627a354cc488a8ce15a2dec5
/avatars/0d99a2fea8b193993fe5b9b7e5b74f40.svg
Sreyan Ghosh
user
SreyanG-NVIDIA
67ca66c1cb7e422997cbd149
Zhifeng Kong
extracted_pending
2025-03-07T03:23:47.395Z
652a4dfc36f031c5e6f8b8a6
/avatars/9fb56b025dc25f91ca6c31136eaf74b2.svg
Zhifeng Kong
user
ZhifengKong
67ca66c1cb7e422997cbd14a
Sonal Kumar
claimed_verified
2025-03-10T08:03:25.580Z
65203f31d6dec04046139874
https://cdn-avatars.huggingface.co/v1/production/uploads/65203f31d6dec04046139874/1LDWW4KblYeIDve9F_kUA.png
Sonal Kumar
user
sonalkum
67ca66c1cb7e422997cbd14b
S Sakshi
67ca66c1cb7e422997cbd14c
Jaehyeon Kim
admin_assigned
2025-03-07T09:41:17.053Z
63fc1124a3c067e62897a73f
/avatars/aa63337a7cd73181b7c1e92decf635f4.svg
Jaehyeon Kim
user
firecomputer
67ca66c1cb7e422997cbd14d
Wei Ping
67ca66c1cb7e422997cbd14e
Rafael Valle
admin_assigned
2025-03-07T09:41:08.488Z
6440ddd65d600fb09518daa8
/avatars/ac5898afd2082d230e2ebf6fb867ad4f.svg
Rafael Valle
user
rafaelvalle
67ca66c1cb7e422997cbd14f
Dinesh Manocha
admin_assigned
2025-03-07T09:41:01.710Z
6537a569568d8be8fa096b8c
/avatars/bfda5cb252d8b5bc3ad737d99c0d7f49.svg
Dinesh Manocha
user
manocha
67ca66c1cb7e422997cbd150
Bryan Catanzaro
admin_assigned
2025-03-07T09:40:55.626Z
6311021788942700629e6247
/avatars/e7adc1632b76e80e7e4a590033d1c20a.svg
Bryan Catanzaro
user
ctnzr
Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding
  and Expert Reasoning Abilities
Understanding and reasoning over non-speech sounds and music are crucial for
both humans and AI agents to interact effectively with their environments. In
this paper, we introduce Audio Flamingo 2 (AF2), an Audio-Language Model (ALM)
with advanced audio understanding and reasoning capabilities. AF2 leverages (i)
a custom CLAP model, (ii) synthetic Audio QA data for fine-grained audio
reasoning, and (iii) a multi-stage curriculum learning strategy. AF2 achieves
state-of-the-art performance with only a 3B parameter small language model,
surpassing large open-source and proprietary models across over 20 benchmarks.
Next, for the first time, we extend audio understanding to long audio segments
(30 secs to 5 mins) and propose LongAudio, a large and novel dataset for
training ALMs on long audio captioning and question-answering tasks.
Fine-tuning AF2 on LongAudio leads to exceptional performance on our proposed
LongAudioBench, an expert annotated benchmark for evaluating ALMs on long audio
understanding capabilities. We conduct extensive ablation studies to confirm
the efficacy of our approach. Project Website:
https://research.nvidia.com/labs/adlr/AF2/.
AF2, an advanced Audio-Language Model, demonstrates state-of-the-art performance on over 20 benchmarks and extends audio understanding to long segments using the LongAudio dataset and LongAudioBench.
Audio Flamingo 2 Audio-Language Model CLAP model Audio QA data multi-stage curriculum learning LongAudio long audio captioning long audio question-answering LongAudioBench long audio understanding