HallusionBench: You See What You Think? Or You Think What You See? An
  Image-Context Reasoning Benchmark Challenging for GPT-4V(ision), LLaVA-1.5,
  and Other Multi-modality Models
Large language models (LLMs), after being aligned with vision models and
integrated into vision-language models (VLMs), can bring impressive improvement
in image reasoning tasks. This was shown by the recently released GPT-4V(ison),
LLaVA-1.5, etc. However, the strong language prior in these SOTA LVLMs can be a
double-edged sword: they may ignore the image context and solely rely on the
(even contradictory) language prior for reasoning. In contrast, the vision
modules in VLMs are weaker than LLMs and may result in misleading visual
representations, which are then translated to confident mistakes by LLMs. To
study these two types of VLM mistakes, i.e., language hallucination and visual
illusion, we curated HallusionBench, an image-context reasoning benchmark that
is still challenging to even GPT-4V and LLaVA-1.5. We provide a detailed
analysis of examples in HallusionBench, which sheds novel insights on the
illusion or hallucination of VLMs and how to improve them in the future. The
benchmark and codebase will be released at
https://github.com/tianyi-lab/HallusionBench.
65372d83019de94263b27af3
Fuxiao Liu
admin_assigned
2023-10-24T08:24:13.063Z
64618b9496259bec21d44704
https://cdn-avatars.huggingface.co/v1/production/uploads/64618b9496259bec21d44704/FCzN0p7Elh8MeUTuM-hya.jpeg
Fuxiao Liu
user
Fuxiao
65372d83019de94263b27af4
Tianrui Guan
admin_assigned
2023-10-24T08:24:19.792Z
64baeea1d1c7b6d84883e7fb
https://cdn-avatars.huggingface.co/v1/production/uploads/64baeea1d1c7b6d84883e7fb/gYEYt5T5ee50-dL1UbEzd.jpeg
Tianrui Guan
user
rayguan
65372d83019de94263b27af5
Zongxia Li
admin_assigned
2023-10-24T08:24:28.027Z
642447e873f7a0d40b30d677
https://cdn-avatars.huggingface.co/v1/production/uploads/642447e873f7a0d40b30d677/hbUnfIhKZCeSSPv9wGAzk.png
LZX
user
zli12321
65372d83019de94263b27af6
Lichang Chen
claimed_verified
2023-10-24T07:29:22.944Z
62323bb408bcea92917e42ee
https://cdn-avatars.huggingface.co/v1/production/uploads/62323bb408bcea92917e42ee/2vHxkv-oSROtLteOnqa8P.jpeg
Lichang Chen
user
Lichang-Chen
65372d83019de94263b27af7
Yaser Yacoob
65372d83019de94263b27af8
Dinesh Manocha
claimed_verified
2023-10-24T11:47:12.151Z
6537a569568d8be8fa096b8c
/avatars/bfda5cb252d8b5bc3ad737d99c0d7f49.svg
Dinesh Manocha
user
manocha
65372d83019de94263b27af9
Tianyi Zhou
claimed_verified
2023-10-24T07:29:26.043Z
647f5af5b0e96764589f3b2a
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
Tianyi Zhou
user
zhoutianyi
HallusionBench: You See What You Think? Or You Think What You See? An
  Image-Context Reasoning Benchmark Challenging for GPT-4V(ision), LLaVA-1.5,
  and Other Multi-modality Models
Large language models (LLMs), after being aligned with vision models and
integrated into vision-language models (VLMs), can bring impressive improvement
in image reasoning tasks. This was shown by the recently released GPT-4V(ison),
LLaVA-1.5, etc. However, the strong language prior in these SOTA LVLMs can be a
double-edged sword: they may ignore the image context and solely rely on the
(even contradictory) language prior for reasoning. In contrast, the vision
modules in VLMs are weaker than LLMs and may result in misleading visual
representations, which are then translated to confident mistakes by LLMs. To
study these two types of VLM mistakes, i.e., language hallucination and visual
illusion, we curated HallusionBench, an image-context reasoning benchmark that
is still challenging to even GPT-4V and LLaVA-1.5. We provide a detailed
analysis of examples in HallusionBench, which sheds novel insights on the
illusion or hallucination of VLMs and how to improve them in the future. The
benchmark and codebase will be released at
https://github.com/tianyi-lab/HallusionBench.
HallusionBench is a benchmark that highlights language hallucination and visual illusion issues in vision-language models (VLMs), showcasing the limitations of current state-of-the-art models like GPT-4V and LLaVA-1.5.
Large language models vision models vision-language models GPT-4V LLaVA-1.5 image reasoning language prior image context misleading visual representations language hallucination visual illusion HallusionBench