title: Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding
    and Expert Reasoning Abilities
summary: Understanding and reasoning over non-speech sounds and music are crucial for
  both humans and AI agents to interact effectively with their environments. In
  this paper, we introduce Audio Flamingo 2 (AF2), an Audio-Language Model (ALM)
  with advanced audio understanding and reasoning capabilities. AF2 leverages (i)
  a custom CLAP model, (ii) synthetic Audio QA data for fine-grained audio
  reasoning, and (iii) a multi-stage curriculum learning strategy. AF2 achieves
  state-of-the-art performance with only a 3B parameter small language model,
  surpassing large open-source and proprietary models across over 20 benchmarks.
  Next, for the first time, we extend audio understanding to long audio segments
  (30 secs to 5 mins) and propose LongAudio, a large and novel dataset for
  training ALMs on long audio captioning and question-answering tasks.
  Fine-tuning AF2 on LongAudio leads to exceptional performance on our proposed
  LongAudioBench, an expert annotated benchmark for evaluating ALMs on long audio
  understanding capabilities. We conduct extensive ablation studies to confirm
  the efficacy of our approach. Project Website:
  https://research.nvidia.com/labs/adlr/AF2/.
paper_authors._id: 67ca66c1cb7e422997cbd148
paper_authors.name: Sreyan Ghosh
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:40:39.324Z
paper_authors._id: 627a354cc488a8ce15a2dec5
paper_authors.avatarUrl: /avatars/0d99a2fea8b193993fe5b9b7e5b74f40.svg
paper_authors.fullname: Sreyan Ghosh
paper_authors.type: user
paper_authors.user: SreyanG-NVIDIA
paper_authors._id: 67ca66c1cb7e422997cbd149
paper_authors.name: Zhifeng Kong
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-03-07T03:23:47.395Z
paper_authors._id: 652a4dfc36f031c5e6f8b8a6
paper_authors.avatarUrl: /avatars/9fb56b025dc25f91ca6c31136eaf74b2.svg
paper_authors.fullname: Zhifeng Kong
paper_authors.type: user
paper_authors.user: ZhifengKong
paper_authors._id: 67ca66c1cb7e422997cbd14a
paper_authors.name: Sonal Kumar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-10T08:03:25.580Z
paper_authors._id: 65203f31d6dec04046139874
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65203f31d6dec04046139874/1LDWW4KblYeIDve9F_kUA.png
paper_authors.fullname: Sonal Kumar
paper_authors.type: user
paper_authors.user: sonalkum
paper_authors._id: 67ca66c1cb7e422997cbd14b
paper_authors.name: S Sakshi
paper_authors._id: 67ca66c1cb7e422997cbd14c
paper_authors.name: Jaehyeon Kim
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:41:17.053Z
paper_authors._id: 63fc1124a3c067e62897a73f
paper_authors.avatarUrl: /avatars/aa63337a7cd73181b7c1e92decf635f4.svg
paper_authors.fullname: Jaehyeon Kim
paper_authors.type: user
paper_authors.user: firecomputer
paper_authors._id: 67ca66c1cb7e422997cbd14d
paper_authors.name: Wei Ping
paper_authors._id: 67ca66c1cb7e422997cbd14e
paper_authors.name: Rafael Valle
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:41:08.488Z
paper_authors._id: 6440ddd65d600fb09518daa8
paper_authors.avatarUrl: /avatars/ac5898afd2082d230e2ebf6fb867ad4f.svg
paper_authors.fullname: Rafael Valle
paper_authors.type: user
paper_authors.user: rafaelvalle
paper_authors._id: 67ca66c1cb7e422997cbd14f
paper_authors.name: Dinesh Manocha
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:41:01.710Z
paper_authors._id: 6537a569568d8be8fa096b8c
paper_authors.avatarUrl: /avatars/bfda5cb252d8b5bc3ad737d99c0d7f49.svg
paper_authors.fullname: Dinesh Manocha
paper_authors.type: user
paper_authors.user: manocha
paper_authors._id: 67ca66c1cb7e422997cbd150
paper_authors.name: Bryan Catanzaro
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-07T09:40:55.626Z
paper_authors._id: 6311021788942700629e6247
paper_authors.avatarUrl: /avatars/e7adc1632b76e80e7e4a590033d1c20a.svg
paper_authors.fullname: Bryan Catanzaro
paper_authors.type: user
paper_authors.user: ctnzr
paper_title: Audio Flamingo 2: An Audio-Language Model with Long-Audio Understanding
    and Expert Reasoning Abilities
paper_summary: Understanding and reasoning over non-speech sounds and music are crucial for
  both humans and AI agents to interact effectively with their environments. In
  this paper, we introduce Audio Flamingo 2 (AF2), an Audio-Language Model (ALM)
  with advanced audio understanding and reasoning capabilities. AF2 leverages (i)
  a custom CLAP model, (ii) synthetic Audio QA data for fine-grained audio
  reasoning, and (iii) a multi-stage curriculum learning strategy. AF2 achieves
  state-of-the-art performance with only a 3B parameter small language model,
  surpassing large open-source and proprietary models across over 20 benchmarks.
  Next, for the first time, we extend audio understanding to long audio segments
  (30 secs to 5 mins) and propose LongAudio, a large and novel dataset for
  training ALMs on long audio captioning and question-answering tasks.
  Fine-tuning AF2 on LongAudio leads to exceptional performance on our proposed
  LongAudioBench, an expert annotated benchmark for evaluating ALMs on long audio
  understanding capabilities. We conduct extensive ablation studies to confirm
  the efficacy of our approach. Project Website:
  https://research.nvidia.com/labs/adlr/AF2/.
paper_ai_summary: AF2, an advanced Audio-Language Model, demonstrates state-of-the-art performance on over 20 benchmarks and extends audio understanding to long segments using the LongAudio dataset and LongAudioBench.
paper_ai_keywords: Audio Flamingo 2 Audio-Language Model CLAP model Audio QA data multi-stage curriculum learning LongAudio long audio captioning long audio question-answering LongAudioBench long audio understanding