Benchmarking Trustworthiness of Multimodal Large Language Models: A
  Comprehensive Study
Despite the superior capabilities of Multimodal Large Language Models (MLLMs)
across diverse tasks, they still face significant trustworthiness challenges.
Yet, current literature on the assessment of trustworthy MLLMs remains limited,
lacking a holistic evaluation to offer thorough insights into future
improvements. In this work, we establish MultiTrust, the first comprehensive
and unified benchmark on the trustworthiness of MLLMs across five primary
aspects: truthfulness, safety, robustness, fairness, and privacy. Our benchmark
employs a rigorous evaluation strategy that addresses both multimodal risks and
cross-modal impacts, encompassing 32 diverse tasks with self-curated datasets.
Extensive experiments with 21 modern MLLMs reveal some previously unexplored
trustworthiness issues and risks, highlighting the complexities introduced by
the multimodality and underscoring the necessity for advanced methodologies to
enhance their reliability. For instance, typical proprietary models still
struggle with the perception of visually confusing images and are vulnerable to
multimodal jailbreaking and adversarial attacks; MLLMs are more inclined to
disclose privacy in text and reveal ideological and cultural biases even when
paired with irrelevant images in inference, indicating that the multimodality
amplifies the internal risks from base LLMs. Additionally, we release a
scalable toolbox for standardized trustworthiness research, aiming to
facilitate future advancements in this important field. Code and resources are
publicly available at: https://multi-trust.github.io/.
6699faa6aa8e2bb22e4ffba9
Yichi Zhang
claimed_verified
2024-07-19T11:22:57.701Z
647068933601bb7b0664e09c
/avatars/991299300500fbe88a654d9d455c2454.svg
Yichi Zhang
user
zycheiheihei
6699faa6aa8e2bb22e4ffbaa
Yao Huang
claimed_verified
2024-07-23T08:46:04.564Z
6547219cc1949364695a2913
/avatars/84d4ce48646f791f4506cc86d9792e1d.svg
Yao Huang
user
aries-iai
6699faa6aa8e2bb22e4ffbab
Yitong Sun
6699faa6aa8e2bb22e4ffbac
Chang Liu
6699faa6aa8e2bb22e4ffbad
Zhe Zhao
6699faa6aa8e2bb22e4ffbae
Zhengwei Fang
admin_assigned
2024-07-22T14:59:27.500Z
662dec002d4c0e85dac07185
/avatars/108a8e4e44ebc93f9c091e9ffdab4f26.svg
zhengwei fang
user
stmrvv
6699faa6aa8e2bb22e4ffbaf
Yifan Wang
6699faa6aa8e2bb22e4ffbb0
Huanran Chen
6699faa6aa8e2bb22e4ffbb1
Xiao Yang
6699faa6aa8e2bb22e4ffbb2
Xingxing Wei
6699faa6aa8e2bb22e4ffbb3
Hang Su
6699faa6aa8e2bb22e4ffbb4
Yinpeng Dong
6699faa6aa8e2bb22e4ffbb5
Jun Zhu
Benchmarking Trustworthiness of Multimodal Large Language Models: A
  Comprehensive Study
Despite the superior capabilities of Multimodal Large Language Models (MLLMs)
across diverse tasks, they still face significant trustworthiness challenges.
Yet, current literature on the assessment of trustworthy MLLMs remains limited,
lacking a holistic evaluation to offer thorough insights into future
improvements. In this work, we establish MultiTrust, the first comprehensive
and unified benchmark on the trustworthiness of MLLMs across five primary
aspects: truthfulness, safety, robustness, fairness, and privacy. Our benchmark
employs a rigorous evaluation strategy that addresses both multimodal risks and
cross-modal impacts, encompassing 32 diverse tasks with self-curated datasets.
Extensive experiments with 21 modern MLLMs reveal some previously unexplored
trustworthiness issues and risks, highlighting the complexities introduced by
the multimodality and underscoring the necessity for advanced methodologies to
enhance their reliability. For instance, typical proprietary models still
struggle with the perception of visually confusing images and are vulnerable to
multimodal jailbreaking and adversarial attacks; MLLMs are more inclined to
disclose privacy in text and reveal ideological and cultural biases even when
paired with irrelevant images in inference, indicating that the multimodality
amplifies the internal risks from base LLMs. Additionally, we release a
scalable toolbox for standardized trustworthiness research, aiming to
facilitate future advancements in this important field. Code and resources are
publicly available at: https://multi-trust.github.io/.
MultiTrust is a comprehensive benchmark assessing the trustworthiness of MLLMs across truthfulness, safety, robustness, fairness, and privacy, revealing multimodal challenges and the need for advanced methodologies.
Multimodal Large Language Models MLLMs MultiTrust truthfulness safety robustness fairness privacy multimodal risks cross-modal impacts self-curated datasets multimodal jailbreaking adversarial attacks privacy disclosure ideological biases cultural biases