title: Towards Holistic Evaluation of Large Audio-Language Models: A
    Comprehensive Survey
summary: With advancements in large audio-language models (LALMs), which enhance large
  language models (LLMs) with auditory capabilities, these models are expected to
  demonstrate universal proficiency across various auditory tasks. While numerous
  benchmarks have emerged to assess LALMs' performance, they remain fragmented
  and lack a structured taxonomy. To bridge this gap, we conduct a comprehensive
  survey and propose a systematic taxonomy for LALM evaluations, categorizing
  them into four dimensions based on their objectives: (1) General Auditory
  Awareness and Processing, (2) Knowledge and Reasoning, (3) Dialogue-oriented
  Ability, and (4) Fairness, Safety, and Trustworthiness. We provide detailed
  overviews within each category and highlight challenges in this field, offering
  insights into promising future directions. To the best of our knowledge, this
  is the first survey specifically focused on the evaluations of LALMs, providing
  clear guidelines for the community. We will release the collection of the
  surveyed papers and actively maintain it to support ongoing advancements in the
  field.
paper_authors._id: 6830745670e219f5de8ad360
paper_authors.name: Chih-Kai Yang
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2025-05-23T13:21:28.397Z
paper_authors._id: 646fa3016441111304fec68d
paper_authors.avatarUrl: /avatars/923629340f3785ae8c6e52cf3674d5c2.svg
paper_authors.fullname: Chih-Kai Yang
paper_authors.type: user
paper_authors.user: zenyn
paper_authors._id: 6830745670e219f5de8ad361
paper_authors.name: Neo S. Ho
paper_authors._id: 6830745670e219f5de8ad362
paper_authors.name: Hung-yi Lee
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-05-27T12:40:58.251Z
paper_authors._id: 629b6eb3ad498ab19282ae6f
paper_authors.avatarUrl: /avatars/adade8c512122120b06e5c50724a178d.svg
paper_authors.fullname: Hung-yi Lee
paper_authors.type: user
paper_authors.user: hungyilee
paper_title: Towards Holistic Evaluation of Large Audio-Language Models: A
    Comprehensive Survey
paper_summary: With advancements in large audio-language models (LALMs), which enhance large
  language models (LLMs) with auditory capabilities, these models are expected to
  demonstrate universal proficiency across various auditory tasks. While numerous
  benchmarks have emerged to assess LALMs' performance, they remain fragmented
  and lack a structured taxonomy. To bridge this gap, we conduct a comprehensive
  survey and propose a systematic taxonomy for LALM evaluations, categorizing
  them into four dimensions based on their objectives: (1) General Auditory
  Awareness and Processing, (2) Knowledge and Reasoning, (3) Dialogue-oriented
  Ability, and (4) Fairness, Safety, and Trustworthiness. We provide detailed
  overviews within each category and highlight challenges in this field, offering
  insights into promising future directions. To the best of our knowledge, this
  is the first survey specifically focused on the evaluations of LALMs, providing
  clear guidelines for the community. We will release the collection of the
  surveyed papers and actively maintain it to support ongoing advancements in the
  field.
paper_ai_summary: A survey proposes a systematic taxonomy for evaluating large audio-language models across dimensions including auditory awareness, knowledge reasoning, dialogue ability, and fairness, to address fragmented benchmarks in the field.
paper_ai_keywords: large audio-language models LALMs large language models LLMs auditory capabilities general auditory awareness processing knowledge and reasoning dialogue-oriented ability fairness safety trustworthiness taxonomy evaluations benchmark survey guidelines