title: GAMA: A Large Audio-Language Model with Advanced Audio Understanding and
    Complex Reasoning Abilities
summary: Perceiving and understanding non-speech sounds and non-verbal speech is
  essential to making decisions that help us interact with our surroundings. In
  this paper, we propose GAMA, a novel General-purpose Large Audio-Language Model
  (LALM) with Advanced Audio Understanding and Complex Reasoning Abilities. We
  build GAMA by integrating an LLM with multiple types of audio representations,
  including features from a custom Audio Q-Former, a multi-layer aggregator that
  aggregates features from multiple layers of an audio encoder. We fine-tune GAMA
  on a large-scale audio-language dataset, which augments it with audio
  understanding capabilities. Next, we propose CompA-R (Instruction-Tuning for
  Complex Audio Reasoning), a synthetically generated instruction-tuning (IT)
  dataset with instructions that require the model to perform complex reasoning
  on the input audio. We instruction-tune GAMA with CompA-R to endow it with
  complex reasoning abilities, where we further add a soft prompt as input with
  high-level semantic evidence by leveraging event tags of the input audio.
  Finally, we also propose CompA-R-test, a human-labeled evaluation dataset for
  evaluating the capabilities of LALMs on open-ended audio question-answering
  that requires complex reasoning. Through automated and expert human
  evaluations, we show that GAMA outperforms all other LALMs in literature on
  diverse audio understanding tasks by margins of 1%-84%. Further, GAMA IT-ed on
  CompA-R proves to be superior in its complex reasoning and instruction
  following capabilities.
paper_authors._id: 6670eaa84eee852a8b0a11aa
paper_authors.name: Sreyan Ghosh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-18T15:33:48.383Z
paper_authors._id: 62c9664eb34e600d7eaa4beb
paper_authors.avatarUrl: /avatars/ca23ecdec2d31c99ecce97d9b180ae0c.svg
paper_authors.fullname: Ghosh
paper_authors.type: user
paper_authors.user: Sreyan88
paper_authors._id: 6670eaa84eee852a8b0a11ab
paper_authors.name: Sonal Kumar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-06-21T07:07:11.970Z
paper_authors._id: 65203f31d6dec04046139874
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65203f31d6dec04046139874/1LDWW4KblYeIDve9F_kUA.png
paper_authors.fullname: Sonal Kumar
paper_authors.type: user
paper_authors.user: sonalkum
paper_authors._id: 6670eaa84eee852a8b0a11ac
paper_authors.name: Ashish Seth
paper_authors._id: 6670eaa84eee852a8b0a11ad
paper_authors.name: Chandra Kiran Reddy Evuru
paper_authors._id: 6670eaa84eee852a8b0a11ae
paper_authors.name: Utkarsh Tyagi
paper_authors._id: 6670eaa84eee852a8b0a11af
paper_authors.name: S Sakshi
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-20T10:23:57.918Z
paper_authors._id: 66607692fc751a711a6efbd9
paper_authors.avatarUrl: /avatars/f04f59822d5eaf64ccb205186aef3a50.svg
paper_authors.fullname: Sakshi
paper_authors.type: user
paper_authors.user: ssakshi
paper_authors._id: 6670eaa84eee852a8b0a11b0
paper_authors.name: Oriol Nieto
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-20T10:23:51.245Z
paper_authors._id: 657a29d7d3e458405b8dc146
paper_authors.avatarUrl: /avatars/1fe425dcf3133462fb6376e8e1550b4a.svg
paper_authors.fullname: Oriol Nieto
paper_authors.type: user
paper_authors.user: urinieto
paper_authors._id: 6670eaa84eee852a8b0a11b1
paper_authors.name: Ramani Duraiswami
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-20T10:23:45.112Z
paper_authors._id: 65b8f0339b7250e205100190
paper_authors.avatarUrl: /avatars/29df91154c5911a44ef503bb3580ac81.svg
paper_authors.fullname: Ramani Duraiswami
paper_authors.type: user
paper_authors.user: RamaniD
paper_authors._id: 6670eaa84eee852a8b0a11b2
paper_authors.name: Dinesh Manocha
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-06-20T10:23:39.175Z
paper_authors._id: 6537a569568d8be8fa096b8c
paper_authors.avatarUrl: /avatars/bfda5cb252d8b5bc3ad737d99c0d7f49.svg
paper_authors.fullname: Dinesh Manocha
paper_authors.type: user
paper_authors.user: manocha
paper_title: GAMA: A Large Audio-Language Model with Advanced Audio Understanding and
    Complex Reasoning Abilities
paper_summary: Perceiving and understanding non-speech sounds and non-verbal speech is
  essential to making decisions that help us interact with our surroundings. In
  this paper, we propose GAMA, a novel General-purpose Large Audio-Language Model
  (LALM) with Advanced Audio Understanding and Complex Reasoning Abilities. We
  build GAMA by integrating an LLM with multiple types of audio representations,
  including features from a custom Audio Q-Former, a multi-layer aggregator that
  aggregates features from multiple layers of an audio encoder. We fine-tune GAMA
  on a large-scale audio-language dataset, which augments it with audio
  understanding capabilities. Next, we propose CompA-R (Instruction-Tuning for
  Complex Audio Reasoning), a synthetically generated instruction-tuning (IT)
  dataset with instructions that require the model to perform complex reasoning
  on the input audio. We instruction-tune GAMA with CompA-R to endow it with
  complex reasoning abilities, where we further add a soft prompt as input with
  high-level semantic evidence by leveraging event tags of the input audio.
  Finally, we also propose CompA-R-test, a human-labeled evaluation dataset for
  evaluating the capabilities of LALMs on open-ended audio question-answering
  that requires complex reasoning. Through automated and expert human
  evaluations, we show that GAMA outperforms all other LALMs in literature on
  diverse audio understanding tasks by margins of 1%-84%. Further, GAMA IT-ed on
  CompA-R proves to be superior in its complex reasoning and instruction
  following capabilities.
paper_ai_keywords: Audio Q-Former multi-layer aggregator General-purpose Large Audio-Language Model (LALM) audio encoder instruction-tuning (IT) CompA-R (Instruction-Tuning for Complex Audio Reasoning) CompA-R-test open-ended audio question-answering complex reasoning instruction following LALMs