title: AudioTrust: Benchmarking the Multifaceted Trustworthiness of Audio Large
    Language Models
summary: The rapid advancement and expanding applications of Audio Large Language
  Models (ALLMs) demand a rigorous understanding of their trustworthiness.
  However, systematic research on evaluating these models, particularly
  concerning risks unique to the audio modality, remains largely unexplored.
  Existing evaluation frameworks primarily focus on the text modality or address
  only a restricted set of safety dimensions, failing to adequately account for
  the unique characteristics and application scenarios inherent to the audio
  modality. We introduce AudioTrust-the first multifaceted trustworthiness
  evaluation framework and benchmark specifically designed for ALLMs. AudioTrust
  facilitates assessments across six key dimensions: fairness, hallucination,
  safety, privacy, robustness, and authentication. To comprehensively evaluate
  these dimensions, AudioTrust is structured around 18 distinct experimental
  setups. Its core is a meticulously constructed dataset of over 4,420 audio/text
  samples, drawn from real-world scenarios (e.g., daily conversations, emergency
  calls, voice assistant interactions), specifically designed to probe the
  multifaceted trustworthiness of ALLMs. For assessment, the benchmark carefully
  designs 9 audio-specific evaluation metrics, and we employ a large-scale
  automated pipeline for objective and scalable scoring of model outputs.
  Experimental results reveal the trustworthiness boundaries and limitations of
  current state-of-the-art open-source and closed-source ALLMs when confronted
  with various high-risk audio scenarios, offering valuable insights for the
  secure and trustworthy deployment of future audio models. Our platform and
  benchmark are available at https://github.com/JusperLee/AudioTrust.
paper_authors._id: 6833d9cfdf7cbb5c087cb9cd
paper_authors.name: Kai Li
paper_authors._id: 6833d9cfdf7cbb5c087cb9ce
paper_authors.name: Can Shen
paper_authors._id: 6833d9cfdf7cbb5c087cb9cf
paper_authors.name: Yile Liu
paper_authors._id: 6833d9cfdf7cbb5c087cb9d0
paper_authors.name: Jirui Han
paper_authors._id: 6833d9cfdf7cbb5c087cb9d1
paper_authors.name: Kelong Zheng
paper_authors._id: 6833d9cfdf7cbb5c087cb9d2
paper_authors.name: Xuechao Zou
paper_authors._id: 6833d9cfdf7cbb5c087cb9d3
paper_authors.name: Zhe Wang
paper_authors._id: 6833d9cfdf7cbb5c087cb9d4
paper_authors.name: Xingjian Du
paper_authors._id: 6833d9cfdf7cbb5c087cb9d5
paper_authors.name: Shun Zhang
paper_authors._id: 6833d9cfdf7cbb5c087cb9d6
paper_authors.name: Hanjun Luo
paper_authors._id: 6833d9cfdf7cbb5c087cb9d7
paper_authors.name: Yingbin Jin
paper_authors._id: 6833d9cfdf7cbb5c087cb9d8
paper_authors.name: Xinxin Xing
paper_authors._id: 6833d9cfdf7cbb5c087cb9d9
paper_authors.name: Ziyang Ma
paper_authors._id: 6833d9cfdf7cbb5c087cb9da
paper_authors.name: Yue Liu
paper_authors._id: 6833d9cfdf7cbb5c087cb9db
paper_authors.name: Xiaojun Jia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:09:37.847Z
paper_authors._id: 64c6627d5671d42e0adfad56
paper_authors.avatarUrl: /avatars/8b98054b2911b86dcc4856a15306e60f.svg
paper_authors.fullname: jiaxiaojunQAQ
paper_authors.type: user
paper_authors.user: jiaxiaojunQAQ
paper_authors._id: 6833d9cfdf7cbb5c087cb9dc
paper_authors.name: Yifan Zhang
paper_authors._id: 6833d9cfdf7cbb5c087cb9dd
paper_authors.name: Junfeng Fang
paper_authors._id: 6833d9cfdf7cbb5c087cb9de
paper_authors.name: Kun Wang
paper_authors._id: 6833d9cfdf7cbb5c087cb9df
paper_authors.name: Yibo Yan
paper_authors._id: 6833d9cfdf7cbb5c087cb9e0
paper_authors.name: Haoyang Li
paper_authors._id: 6833d9cfdf7cbb5c087cb9e1
paper_authors.name: Yiming Li
paper_authors._id: 6833d9cfdf7cbb5c087cb9e2
paper_authors.name: Xiaobin Zhuang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-03T08:51:08.135Z
paper_authors._id: 63774ca43a63a2983ffc12f9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63774ca43a63a2983ffc12f9/jUJasGf5_2rwiEun_eYl7.png
paper_authors.fullname: xiaobin zhuang
paper_authors.type: user
paper_authors.user: xiaobinzhuang
paper_authors._id: 6833d9cfdf7cbb5c087cb9e3
paper_authors.name: Yang Liu
paper_authors._id: 6833d9cfdf7cbb5c087cb9e4
paper_authors.name: Haibo Hu
paper_authors._id: 6833d9cfdf7cbb5c087cb9e5
paper_authors.name: Zhuo Chen
paper_authors._id: 6833d9cfdf7cbb5c087cb9e6
paper_authors.name: Zhizheng Wu
paper_authors._id: 6833d9cfdf7cbb5c087cb9e7
paper_authors.name: Xiaolin Hu
paper_authors._id: 6833d9cfdf7cbb5c087cb9e8
paper_authors.name: Eng-Siong Chng
paper_authors._id: 6833d9cfdf7cbb5c087cb9e9
paper_authors.name: XiaoFeng Wang
paper_authors._id: 6833d9cfdf7cbb5c087cb9ea
paper_authors.name: Wenyuan Xu
paper_authors._id: 6833d9cfdf7cbb5c087cb9eb
paper_authors.name: Wei Dong
paper_authors._id: 6833d9cfdf7cbb5c087cb9ec
paper_authors.name: Xinfeng Li
paper_title: AudioTrust: Benchmarking the Multifaceted Trustworthiness of Audio Large
    Language Models
paper_summary: The rapid advancement and expanding applications of Audio Large Language
  Models (ALLMs) demand a rigorous understanding of their trustworthiness.
  However, systematic research on evaluating these models, particularly
  concerning risks unique to the audio modality, remains largely unexplored.
  Existing evaluation frameworks primarily focus on the text modality or address
  only a restricted set of safety dimensions, failing to adequately account for
  the unique characteristics and application scenarios inherent to the audio
  modality. We introduce AudioTrust-the first multifaceted trustworthiness
  evaluation framework and benchmark specifically designed for ALLMs. AudioTrust
  facilitates assessments across six key dimensions: fairness, hallucination,
  safety, privacy, robustness, and authentication. To comprehensively evaluate
  these dimensions, AudioTrust is structured around 18 distinct experimental
  setups. Its core is a meticulously constructed dataset of over 4,420 audio/text
  samples, drawn from real-world scenarios (e.g., daily conversations, emergency
  calls, voice assistant interactions), specifically designed to probe the
  multifaceted trustworthiness of ALLMs. For assessment, the benchmark carefully
  designs 9 audio-specific evaluation metrics, and we employ a large-scale
  automated pipeline for objective and scalable scoring of model outputs.
  Experimental results reveal the trustworthiness boundaries and limitations of
  current state-of-the-art open-source and closed-source ALLMs when confronted
  with various high-risk audio scenarios, offering valuable insights for the
  secure and trustworthy deployment of future audio models. Our platform and
  benchmark are available at https://github.com/JusperLee/AudioTrust.
paper_ai_summary: AudioTrust evaluates the trustworthiness of Audio Large Language Models across multifaceted dimensions, using a comprehensive dataset and specific metrics to assess their performance in real-world audio scenarios.
paper_ai_keywords: Audio Large Language Models ALLMs trustworthiness fairness hallucination safety privacy robustness authentication AudioTrust experimental setups audio-specific evaluation metrics automated pipeline