title: HallusionBench: You See What You Think? Or You Think What You See? An
    Image-Context Reasoning Benchmark Challenging for GPT-4V(ision), LLaVA-1.5,
    and Other Multi-modality Models
summary: Large language models (LLMs), after being aligned with vision models and
  integrated into vision-language models (VLMs), can bring impressive improvement
  in image reasoning tasks. This was shown by the recently released GPT-4V(ison),
  LLaVA-1.5, etc. However, the strong language prior in these SOTA LVLMs can be a
  double-edged sword: they may ignore the image context and solely rely on the
  (even contradictory) language prior for reasoning. In contrast, the vision
  modules in VLMs are weaker than LLMs and may result in misleading visual
  representations, which are then translated to confident mistakes by LLMs. To
  study these two types of VLM mistakes, i.e., language hallucination and visual
  illusion, we curated HallusionBench, an image-context reasoning benchmark that
  is still challenging to even GPT-4V and LLaVA-1.5. We provide a detailed
  analysis of examples in HallusionBench, which sheds novel insights on the
  illusion or hallucination of VLMs and how to improve them in the future. The
  benchmark and codebase will be released at
  https://github.com/tianyi-lab/HallusionBench.
paper_authors._id: 65372d83019de94263b27af3
paper_authors.name: Fuxiao Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:24:13.063Z
paper_authors._id: 64618b9496259bec21d44704
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64618b9496259bec21d44704/FCzN0p7Elh8MeUTuM-hya.jpeg
paper_authors.fullname: Fuxiao Liu
paper_authors.type: user
paper_authors.user: Fuxiao
paper_authors._id: 65372d83019de94263b27af4
paper_authors.name: Tianrui Guan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:24:19.792Z
paper_authors._id: 64baeea1d1c7b6d84883e7fb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64baeea1d1c7b6d84883e7fb/gYEYt5T5ee50-dL1UbEzd.jpeg
paper_authors.fullname: Tianrui Guan
paper_authors.type: user
paper_authors.user: rayguan
paper_authors._id: 65372d83019de94263b27af5
paper_authors.name: Zongxia Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-24T08:24:28.027Z
paper_authors._id: 642447e873f7a0d40b30d677
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/642447e873f7a0d40b30d677/U8eTvyLJPfXyr8GNWhyOT.jpeg
paper_authors.fullname: LZX
paper_authors.type: user
paper_authors.user: zli12321
paper_authors._id: 65372d83019de94263b27af6
paper_authors.name: Lichang Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-24T07:29:22.944Z
paper_authors._id: 62323bb408bcea92917e42ee
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62323bb408bcea92917e42ee/2vHxkv-oSROtLteOnqa8P.jpeg
paper_authors.fullname: Lichang Chen
paper_authors.type: user
paper_authors.user: Lichang-Chen
paper_authors._id: 65372d83019de94263b27af7
paper_authors.name: Yaser Yacoob
paper_authors._id: 65372d83019de94263b27af8
paper_authors.name: Dinesh Manocha
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-24T11:47:12.151Z
paper_authors._id: 6537a569568d8be8fa096b8c
paper_authors.avatarUrl: /avatars/bfda5cb252d8b5bc3ad737d99c0d7f49.svg
paper_authors.fullname: Dinesh Manocha
paper_authors.type: user
paper_authors.user: manocha
paper_authors._id: 65372d83019de94263b27af9
paper_authors.name: Tianyi Zhou
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2023-10-24T07:29:26.043Z
paper_authors._id: 647f5af5b0e96764589f3b2a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VJ4cDyjp5M3V5WmI5gPIU.jpeg
paper_authors.fullname: Tianyi Zhou
paper_authors.type: user
paper_authors.user: zhoutianyi
paper_title: HallusionBench: You See What You Think? Or You Think What You See? An
    Image-Context Reasoning Benchmark Challenging for GPT-4V(ision), LLaVA-1.5,
    and Other Multi-modality Models
paper_summary: Large language models (LLMs), after being aligned with vision models and
  integrated into vision-language models (VLMs), can bring impressive improvement
  in image reasoning tasks. This was shown by the recently released GPT-4V(ison),
  LLaVA-1.5, etc. However, the strong language prior in these SOTA LVLMs can be a
  double-edged sword: they may ignore the image context and solely rely on the
  (even contradictory) language prior for reasoning. In contrast, the vision
  modules in VLMs are weaker than LLMs and may result in misleading visual
  representations, which are then translated to confident mistakes by LLMs. To
  study these two types of VLM mistakes, i.e., language hallucination and visual
  illusion, we curated HallusionBench, an image-context reasoning benchmark that
  is still challenging to even GPT-4V and LLaVA-1.5. We provide a detailed
  analysis of examples in HallusionBench, which sheds novel insights on the
  illusion or hallucination of VLMs and how to improve them in the future. The
  benchmark and codebase will be released at
  https://github.com/tianyi-lab/HallusionBench.
paper_ai_keywords: vision models vision-language models (VLMs) GPT-4V LLaVA-1.5 language prior image context vision modules image reasoning tasks HallusionBench image-context reasoning benchmark language hallucination visual illusion visual representations confident mistakes LLMs large language models semi insights