title: MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark
summary: The ability to comprehend audio--which includes speech, non-speech sounds,
  and music--is crucial for AI agents to interact effectively with the world. We
  present MMAU, a novel benchmark designed to evaluate multimodal audio
  understanding models on tasks requiring expert-level knowledge and complex
  reasoning. MMAU comprises 10k carefully curated audio clips paired with
  human-annotated natural language questions and answers spanning speech,
  environmental sounds, and music. It includes information extraction and
  reasoning questions, requiring models to demonstrate 27 distinct skills across
  unique and challenging tasks. Unlike existing benchmarks, MMAU emphasizes
  advanced perception and reasoning with domain-specific knowledge, challenging
  models to tackle tasks akin to those faced by experts. We assess 18 open-source
  and proprietary (Large) Audio-Language Models, demonstrating the significant
  challenges posed by MMAU. Notably, even the most advanced Gemini Pro v1.5
  achieves only 52.97% accuracy, and the state-of-the-art open-source Qwen2-Audio
  achieves only 52.50%, highlighting considerable room for improvement. We
  believe MMAU will drive the audio and multimodal research community to develop
  more advanced audio understanding models capable of solving complex audio
  tasks.
paper_authors._id: 671ef25002737c628d38c1a3
paper_authors.name: S Sakshi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-30T09:54:36.298Z
paper_authors._id: 66607692fc751a711a6efbd9
paper_authors.avatarUrl: /avatars/f04f59822d5eaf64ccb205186aef3a50.svg
paper_authors.fullname: Sakshi
paper_authors.type: user
paper_authors.user: ssakshi
paper_authors._id: 671ef25002737c628d38c1a4
paper_authors.name: Utkarsh Tyagi
paper_authors._id: 671ef25002737c628d38c1a5
paper_authors.name: Sonal Kumar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-28T15:40:32.700Z
paper_authors._id: 65203f31d6dec04046139874
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/65203f31d6dec04046139874/1LDWW4KblYeIDve9F_kUA.png
paper_authors.fullname: Sonal Kumar
paper_authors.type: user
paper_authors.user: sonalkum
paper_authors._id: 671ef25002737c628d38c1a6
paper_authors.name: Ashish Seth
paper_authors._id: 671ef25002737c628d38c1a7
paper_authors.name: Ramaneswaran Selvakumar
paper_authors._id: 671ef25002737c628d38c1a8
paper_authors.name: Oriol Nieto
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-28T15:28:06.888Z
paper_authors._id: 657a29d7d3e458405b8dc146
paper_authors.avatarUrl: /avatars/1fe425dcf3133462fb6376e8e1550b4a.svg
paper_authors.fullname: Oriol Nieto
paper_authors.type: user
paper_authors.user: urinieto
paper_authors._id: 671ef25002737c628d38c1a9
paper_authors.name: Ramani Duraiswami
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-28T15:28:01.454Z
paper_authors._id: 65b8f0339b7250e205100190
paper_authors.avatarUrl: /avatars/29df91154c5911a44ef503bb3580ac81.svg
paper_authors.fullname: Ramani Duraiswami
paper_authors.type: user
paper_authors.user: RamaniD
paper_authors._id: 671ef25002737c628d38c1aa
paper_authors.name: Sreyan Ghosh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-28T15:36:42.210Z
paper_authors._id: 62c9664eb34e600d7eaa4beb
paper_authors.avatarUrl: /avatars/ca23ecdec2d31c99ecce97d9b180ae0c.svg
paper_authors.fullname: Ghosh
paper_authors.type: user
paper_authors.user: Sreyan88
paper_authors._id: 671ef25002737c628d38c1ab
paper_authors.name: Dinesh Manocha
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-28T15:27:27.197Z
paper_authors._id: 6537a569568d8be8fa096b8c
paper_authors.avatarUrl: /avatars/bfda5cb252d8b5bc3ad737d99c0d7f49.svg
paper_authors.fullname: Dinesh Manocha
paper_authors.type: user
paper_authors.user: manocha
paper_title: MMAU: A Massive Multi-Task Audio Understanding and Reasoning Benchmark
paper_summary: The ability to comprehend audio--which includes speech, non-speech sounds,
  and music--is crucial for AI agents to interact effectively with the world. We
  present MMAU, a novel benchmark designed to evaluate multimodal audio
  understanding models on tasks requiring expert-level knowledge and complex
  reasoning. MMAU comprises 10k carefully curated audio clips paired with
  human-annotated natural language questions and answers spanning speech,
  environmental sounds, and music. It includes information extraction and
  reasoning questions, requiring models to demonstrate 27 distinct skills across
  unique and challenging tasks. Unlike existing benchmarks, MMAU emphasizes
  advanced perception and reasoning with domain-specific knowledge, challenging
  models to tackle tasks akin to those faced by experts. We assess 18 open-source
  and proprietary (Large) Audio-Language Models, demonstrating the significant
  challenges posed by MMAU. Notably, even the most advanced Gemini Pro v1.5
  achieves only 52.97% accuracy, and the state-of-the-art open-source Qwen2-Audio
  achieves only 52.50%, highlighting considerable room for improvement. We
  believe MMAU will drive the audio and multimodal research community to develop
  more advanced audio understanding models capable of solving complex audio
  tasks.
paper_ai_keywords: multimodal audio understanding models information extraction and reasoning domain-specific knowledge Large Audio-Language Models Gemini Pro v1.5 Qwen2-Audio