MoHoBench: Assessing Honesty of Multimodal Large Language Models via
  Unanswerable Visual Questions
Recently Multimodal Large Language Models (MLLMs) have achieved considerable
advancements in vision-language tasks, yet produce potentially harmful or
untrustworthy content. Despite substantial work investigating the
trustworthiness of language models, MMLMs' capability to act honestly,
especially when faced with visually unanswerable questions, remains largely
underexplored. This work presents the first systematic assessment of honesty
behaviors across various MLLMs. We ground honesty in models' response behaviors
to unanswerable visual questions, define four representative types of such
questions, and construct MoHoBench, a large-scale MMLM honest benchmark,
consisting of 12k+ visual question samples, whose quality is guaranteed by
multi-stage filtering and human verification. Using MoHoBench, we benchmarked
the honesty of 28 popular MMLMs and conducted a comprehensive analysis. Our
findings show that: (1) most models fail to appropriately refuse to answer when
necessary, and (2) MMLMs' honesty is not solely a language modeling issue, but
is deeply influenced by visual information, necessitating the development of
dedicated methods for multimodal honesty alignment. Therefore, we implemented
initial alignment methods using supervised and preference learning to improve
honesty behavior, providing a foundation for future work on trustworthy MLLMs.
Our data and code can be found at https://github.com/DSTTSD/MoHoBench.
6889fecc6b57cb050413053b
Yanxu Zhu
claimed_verified
2025-07-30T14:18:09.637Z
6344df7aee1504dbcd5c87f9
/avatars/474e2fe189e558e2ae73011a78615edf.svg
yanxuzhu
user
yanxuzhu
6889fecc6b57cb050413053c
Shitong Duan
6889fecc6b57cb050413053d
Xiangxu Zhang
6889fecc6b57cb050413053e
Jitao Sang
6889fecc6b57cb050413053f
Peng Zhang
6889fecc6b57cb0504130540
Tun Lu
6889fecc6b57cb0504130541
Xiao Zhou
6889fecc6b57cb0504130542
Jing Yao
6889fecc6b57cb0504130543
Xiaoyuan Yi
6889fecc6b57cb0504130544
Xing Xie
MoHoBench: Assessing Honesty of Multimodal Large Language Models via
  Unanswerable Visual Questions
Recently Multimodal Large Language Models (MLLMs) have achieved considerable
advancements in vision-language tasks, yet produce potentially harmful or
untrustworthy content. Despite substantial work investigating the
trustworthiness of language models, MMLMs' capability to act honestly,
especially when faced with visually unanswerable questions, remains largely
underexplored. This work presents the first systematic assessment of honesty
behaviors across various MLLMs. We ground honesty in models' response behaviors
to unanswerable visual questions, define four representative types of such
questions, and construct MoHoBench, a large-scale MMLM honest benchmark,
consisting of 12k+ visual question samples, whose quality is guaranteed by
multi-stage filtering and human verification. Using MoHoBench, we benchmarked
the honesty of 28 popular MMLMs and conducted a comprehensive analysis. Our
findings show that: (1) most models fail to appropriately refuse to answer when
necessary, and (2) MMLMs' honesty is not solely a language modeling issue, but
is deeply influenced by visual information, necessitating the development of
dedicated methods for multimodal honesty alignment. Therefore, we implemented
initial alignment methods using supervised and preference learning to improve
honesty behavior, providing a foundation for future work on trustworthy MLLMs.
Our data and code can be found at https://github.com/DSTTSD/MoHoBench.
A systematic assessment of honesty in Multimodal Large Language Models (MLLMs) using a large-scale benchmark reveals that models often fail to appropriately refuse unanswerable visual questions, highlighting the need for multimodal honesty alignment methods.
Multimodal Large Language Models MLLMs MoHoBench visual question samples honesty behaviors supervised learning preference learning