title: Benchmarking Trustworthiness of Multimodal Large Language Models: A
    Comprehensive Study
summary: Despite the superior capabilities of Multimodal Large Language Models (MLLMs)
  across diverse tasks, they still face significant trustworthiness challenges.
  Yet, current literature on the assessment of trustworthy MLLMs remains limited,
  lacking a holistic evaluation to offer thorough insights into future
  improvements. In this work, we establish MultiTrust, the first comprehensive
  and unified benchmark on the trustworthiness of MLLMs across five primary
  aspects: truthfulness, safety, robustness, fairness, and privacy. Our benchmark
  employs a rigorous evaluation strategy that addresses both multimodal risks and
  cross-modal impacts, encompassing 32 diverse tasks with self-curated datasets.
  Extensive experiments with 21 modern MLLMs reveal some previously unexplored
  trustworthiness issues and risks, highlighting the complexities introduced by
  the multimodality and underscoring the necessity for advanced methodologies to
  enhance their reliability. For instance, typical proprietary models still
  struggle with the perception of visually confusing images and are vulnerable to
  multimodal jailbreaking and adversarial attacks; MLLMs are more inclined to
  disclose privacy in text and reveal ideological and cultural biases even when
  paired with irrelevant images in inference, indicating that the multimodality
  amplifies the internal risks from base LLMs. Additionally, we release a
  scalable toolbox for standardized trustworthiness research, aiming to
  facilitate future advancements in this important field. Code and resources are
  publicly available at: https://multi-trust.github.io/.
paper_authors._id: 6699faa6aa8e2bb22e4ffba9
paper_authors.name: Yichi Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-19T11:22:57.701Z
paper_authors._id: 647068933601bb7b0664e09c
paper_authors.avatarUrl: /avatars/991299300500fbe88a654d9d455c2454.svg
paper_authors.fullname: Yichi Zhang
paper_authors.type: user
paper_authors.user: zycheiheihei
paper_authors._id: 6699faa6aa8e2bb22e4ffbaa
paper_authors.name: Yao Huang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-07-23T08:46:04.564Z
paper_authors._id: 6547219cc1949364695a2913
paper_authors.avatarUrl: /avatars/84d4ce48646f791f4506cc86d9792e1d.svg
paper_authors.fullname: Yao Huang
paper_authors.type: user
paper_authors.user: aries-iai
paper_authors._id: 6699faa6aa8e2bb22e4ffbab
paper_authors.name: Yitong Sun
paper_authors._id: 6699faa6aa8e2bb22e4ffbac
paper_authors.name: Chang Liu
paper_authors._id: 6699faa6aa8e2bb22e4ffbad
paper_authors.name: Zhe Zhao
paper_authors._id: 6699faa6aa8e2bb22e4ffbae
paper_authors.name: Zhengwei Fang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-07-22T14:59:27.500Z
paper_authors._id: 662dec002d4c0e85dac07185
paper_authors.avatarUrl: /avatars/108a8e4e44ebc93f9c091e9ffdab4f26.svg
paper_authors.fullname: zhengwei fang
paper_authors.type: user
paper_authors.user: stmrvv
paper_authors._id: 6699faa6aa8e2bb22e4ffbaf
paper_authors.name: Yifan Wang
paper_authors._id: 6699faa6aa8e2bb22e4ffbb0
paper_authors.name: Huanran Chen
paper_authors._id: 6699faa6aa8e2bb22e4ffbb1
paper_authors.name: Xiao Yang
paper_authors._id: 6699faa6aa8e2bb22e4ffbb2
paper_authors.name: Xingxing Wei
paper_authors._id: 6699faa6aa8e2bb22e4ffbb3
paper_authors.name: Hang Su
paper_authors._id: 6699faa6aa8e2bb22e4ffbb4
paper_authors.name: Yinpeng Dong
paper_authors._id: 6699faa6aa8e2bb22e4ffbb5
paper_authors.name: Jun Zhu
paper_title: Benchmarking Trustworthiness of Multimodal Large Language Models: A
    Comprehensive Study
paper_summary: Despite the superior capabilities of Multimodal Large Language Models (MLLMs)
  across diverse tasks, they still face significant trustworthiness challenges.
  Yet, current literature on the assessment of trustworthy MLLMs remains limited,
  lacking a holistic evaluation to offer thorough insights into future
  improvements. In this work, we establish MultiTrust, the first comprehensive
  and unified benchmark on the trustworthiness of MLLMs across five primary
  aspects: truthfulness, safety, robustness, fairness, and privacy. Our benchmark
  employs a rigorous evaluation strategy that addresses both multimodal risks and
  cross-modal impacts, encompassing 32 diverse tasks with self-curated datasets.
  Extensive experiments with 21 modern MLLMs reveal some previously unexplored
  trustworthiness issues and risks, highlighting the complexities introduced by
  the multimodality and underscoring the necessity for advanced methodologies to
  enhance their reliability. For instance, typical proprietary models still
  struggle with the perception of visually confusing images and are vulnerable to
  multimodal jailbreaking and adversarial attacks; MLLMs are more inclined to
  disclose privacy in text and reveal ideological and cultural biases even when
  paired with irrelevant images in inference, indicating that the multimodality
  amplifies the internal risks from base LLMs. Additionally, we release a
  scalable toolbox for standardized trustworthiness research, aiming to
  facilitate future advancements in this important field. Code and resources are
  publicly available at: https://multi-trust.github.io/.
paper_ai_summary: MultiTrust is a comprehensive benchmark assessing the trustworthiness of MLLMs across truthfulness, safety, robustness, fairness, and privacy, revealing multimodal challenges and the need for advanced methodologies.
paper_ai_keywords: Multimodal Large Language Models MLLMs MultiTrust truthfulness safety robustness fairness privacy multimodal risks cross-modal impacts self-curated datasets multimodal jailbreaking adversarial attacks privacy disclosure ideological biases cultural biases