GAMA: A Large Audio-Language Model with Advanced Audio Understanding and
  Complex Reasoning Abilities
Perceiving and understanding non-speech sounds and non-verbal speech is
essential to making decisions that help us interact with our surroundings. In
this paper, we propose GAMA, a novel General-purpose Large Audio-Language Model
(LALM) with Advanced Audio Understanding and Complex Reasoning Abilities. We
build GAMA by integrating an LLM with multiple types of audio representations,
including features from a custom Audio Q-Former, a multi-layer aggregator that
aggregates features from multiple layers of an audio encoder. We fine-tune GAMA
on a large-scale audio-language dataset, which augments it with audio
understanding capabilities. Next, we propose CompA-R (Instruction-Tuning for
Complex Audio Reasoning), a synthetically generated instruction-tuning (IT)
dataset with instructions that require the model to perform complex reasoning
on the input audio. We instruction-tune GAMA with CompA-R to endow it with
complex reasoning abilities, where we further add a soft prompt as input with
high-level semantic evidence by leveraging event tags of the input audio.
Finally, we also propose CompA-R-test, a human-labeled evaluation dataset for
evaluating the capabilities of LALMs on open-ended audio question-answering
that requires complex reasoning. Through automated and expert human
evaluations, we show that GAMA outperforms all other LALMs in literature on
diverse audio understanding tasks by margins of 1%-84%. Further, GAMA IT-ed on
CompA-R proves to be superior in its complex reasoning and instruction
following capabilities.
6670eaa84eee852a8b0a11aa
Sreyan Ghosh
claimed_verified
2024-06-18T15:33:48.383Z
62c9664eb34e600d7eaa4beb
/avatars/ca23ecdec2d31c99ecce97d9b180ae0c.svg
Ghosh
user
Sreyan88
6670eaa84eee852a8b0a11ab
Sonal Kumar
claimed_verified
2024-06-21T07:07:11.970Z
65203f31d6dec04046139874
https://cdn-avatars.huggingface.co/v1/production/uploads/65203f31d6dec04046139874/1LDWW4KblYeIDve9F_kUA.png
Sonal Kumar
user
sonalkum
6670eaa84eee852a8b0a11ac
Ashish Seth
6670eaa84eee852a8b0a11ad
Chandra Kiran Reddy Evuru
6670eaa84eee852a8b0a11ae
Utkarsh Tyagi
6670eaa84eee852a8b0a11af
S Sakshi
admin_assigned
2024-06-20T10:23:57.918Z
66607692fc751a711a6efbd9
/avatars/f04f59822d5eaf64ccb205186aef3a50.svg
Sakshi
user
ssakshi
6670eaa84eee852a8b0a11b0
Oriol Nieto
admin_assigned
2024-06-20T10:23:51.245Z
657a29d7d3e458405b8dc146
/avatars/1fe425dcf3133462fb6376e8e1550b4a.svg
Oriol Nieto
user
urinieto
6670eaa84eee852a8b0a11b1
Ramani Duraiswami
admin_assigned
2024-06-20T10:23:45.112Z
65b8f0339b7250e205100190
/avatars/29df91154c5911a44ef503bb3580ac81.svg
Ramani Duraiswami
user
RamaniD
6670eaa84eee852a8b0a11b2
Dinesh Manocha
admin_assigned
2024-06-20T10:23:39.175Z
6537a569568d8be8fa096b8c
/avatars/bfda5cb252d8b5bc3ad737d99c0d7f49.svg
Dinesh Manocha
user
manocha
GAMA: A Large Audio-Language Model with Advanced Audio Understanding and
  Complex Reasoning Abilities
Perceiving and understanding non-speech sounds and non-verbal speech is
essential to making decisions that help us interact with our surroundings. In
this paper, we propose GAMA, a novel General-purpose Large Audio-Language Model
(LALM) with Advanced Audio Understanding and Complex Reasoning Abilities. We
build GAMA by integrating an LLM with multiple types of audio representations,
including features from a custom Audio Q-Former, a multi-layer aggregator that
aggregates features from multiple layers of an audio encoder. We fine-tune GAMA
on a large-scale audio-language dataset, which augments it with audio
understanding capabilities. Next, we propose CompA-R (Instruction-Tuning for
Complex Audio Reasoning), a synthetically generated instruction-tuning (IT)
dataset with instructions that require the model to perform complex reasoning
on the input audio. We instruction-tune GAMA with CompA-R to endow it with
complex reasoning abilities, where we further add a soft prompt as input with
high-level semantic evidence by leveraging event tags of the input audio.
Finally, we also propose CompA-R-test, a human-labeled evaluation dataset for
evaluating the capabilities of LALMs on open-ended audio question-answering
that requires complex reasoning. Through automated and expert human
evaluations, we show that GAMA outperforms all other LALMs in literature on
diverse audio understanding tasks by margins of 1%-84%. Further, GAMA IT-ed on
CompA-R proves to be superior in its complex reasoning and instruction
following capabilities.
GAMA, a General-purpose Large Audio-Language Model, excels in diverse audio understanding and complex reasoning tasks through advanced audio processing and specialized instruction tuning.
General-purpose Large Audio-Language Model LALM LLM Audio Q-Former multi-layer aggregator audio encoder auto-language dataset CompA-R Instruction-Tuning complex audio reasoning soft prompt high-level semantic evidence CompA-R-test audio question-answering