title: NaturalBench: Evaluating Vision-Language Models on Natural Adversarial
    Samples
summary: Vision-language models (VLMs) have made significant progress in recent
  visual-question-answering (VQA) benchmarks that evaluate complex
  visio-linguistic reasoning. However, are these models truly effective? In this
  work, we show that VLMs still struggle with natural images and questions that
  humans can easily answer, which we term natural adversarial samples. We also
  find it surprisingly easy to generate these VQA samples from natural image-text
  corpora using off-the-shelf models like CLIP and ChatGPT. We propose a
  semi-automated approach to collect a new benchmark, NaturalBench, for reliably
  evaluating VLMs with 10,000 human-verified VQA samples. Crucially, we adopt a
  vision-centric design by pairing each question with two images that
  yield different answers, preventing blind solutions from answering without
  using the images. This makes NaturalBench more challenging than previous
  benchmarks that can be solved with commonsense priors. We evaluate 53
  state-of-the-art VLMs on NaturalBench, showing that models like
  LLaVA-OneVision, Cambrian-1, Llama3.2-Vision, Molmo, Qwen2-VL, and even GPT-4o
  lag 50%-70% behind human performance (over 90%). We analyze why NaturalBench is
  hard from two angles: (1) Compositionality: Solving NaturalBench requires
  diverse visio-linguistic skills, including understanding attribute bindings,
  object relationships, and advanced reasoning like logic and counting. To this
  end, unlike prior work that uses a single tag per sample, we tag each
  NaturalBench sample with 1 to 8 skill tags for fine-grained evaluation. (2)
  Biases: NaturalBench exposes severe biases in VLMs, as models often choose the
  same answer regardless of the image. Lastly, we apply our benchmark curation
  method to diverse data sources, including long captions (over 100 words) and
  non-English languages like Chinese and Hindi, highlighting its potential for
  dynamic evaluations of VLMs.
paper_authors._id: 6715c4518b7afb7724fa6aae
paper_authors.name: Baiqi Li
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-21T07:40:56.291Z
paper_authors._id: 65ea393b1032135623fe8037
paper_authors.avatarUrl: /avatars/c5844e9cd43092cfb67543f5b4d7c0bf.svg
paper_authors.fullname: BaiqiLi
paper_authors.type: user
paper_authors.user: BaiqiL
paper_authors._id: 6715c4518b7afb7724fa6aaf
paper_authors.name: Zhiqiu Lin
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-21T12:06:52.741Z
paper_authors._id: 64c170190bfb901b04399295
paper_authors.avatarUrl: /avatars/c30ce7566ae3497ddc989ec8918d37cc.svg
paper_authors.fullname: Zhiqiu Lin
paper_authors.type: user
paper_authors.user: zhiqiulin
paper_authors._id: 6715c4518b7afb7724fa6ab0
paper_authors.name: Wenxuan Peng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-21T12:07:22.881Z
paper_authors._id: 63b91e47f270ad02f613237b
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1673076442939-63b91e47f270ad02f613237b.jpeg
paper_authors.fullname: WENXUAN PENG
paper_authors.type: user
paper_authors.user: Lilymelon7
paper_authors._id: 6715c4518b7afb7724fa6ab1
paper_authors.name: Jean de Dieu Nyandwi
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-21T11:42:11.775Z
paper_authors._id: 62989cd743e990340bead77c
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1654168730944-noauth.png
paper_authors.fullname: Jean de Dieu Nyandwi
paper_authors.type: user
paper_authors.user: Nyandwi
paper_authors._id: 6715c4518b7afb7724fa6ab2
paper_authors.name: Daniel Jiang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-10-21T20:01:33.369Z
paper_authors._id: 66277092cee7ea5738cd04b6
paper_authors.avatarUrl: /avatars/7894d5b8260dc426c89ffa237c6074e9.svg
paper_authors.fullname: Daniel Jiang
paper_authors.type: user
paper_authors.user: djiang04
paper_authors._id: 6715c4518b7afb7724fa6ab3
paper_authors.name: Zixian Ma
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-21T12:07:32.845Z
paper_authors._id: 6442cce0e255a338677e59ff
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/VheGtt37p7aYIodlt9N3n.jpeg
paper_authors.fullname: Zixian Ma
paper_authors.type: user
paper_authors.user: zixianma
paper_authors._id: 6715c4518b7afb7724fa6ab4
paper_authors.name: Simran Khanuja
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-21T12:08:00.335Z
paper_authors._id: 63a4d079658851481f7e4394
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63a4d079658851481f7e4394/yptj_hJiaqHvEaqxLs0jT.jpeg
paper_authors.fullname: Simran Khanuja
paper_authors.type: user
paper_authors.user: skhanuja
paper_authors._id: 6715c4518b7afb7724fa6ab5
paper_authors.name: Ranjay Krishna
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-21T12:08:07.227Z
paper_authors._id: 66429868ab89e3a3a85668b0
paper_authors.avatarUrl: /avatars/170e0daa454838deee2bf946f7118651.svg
paper_authors.fullname: Ranjay Krishna
paper_authors.type: user
paper_authors.user: ranjaykrishna
paper_authors._id: 6715c4518b7afb7724fa6ab6
paper_authors.name: Graham Neubig
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-21T12:08:13.199Z
paper_authors._id: 60de14638bedd2315529d43f
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1625166923504-noauth.png
paper_authors.fullname: Graham Neubig
paper_authors.type: user
paper_authors.user: gneubig
paper_authors._id: 6715c4518b7afb7724fa6ab7
paper_authors.name: Deva Ramanan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-21T12:08:21.627Z
paper_authors._id: 6337151b0267ebcf02640eb6
paper_authors.avatarUrl: /avatars/14a723cafc5587043bdfb19304fc202d.svg
paper_authors.fullname: Deva Ramanan
paper_authors.type: user
paper_authors.user: devakramanan
paper_title: NaturalBench: Evaluating Vision-Language Models on Natural Adversarial
    Samples
paper_summary: Vision-language models (VLMs) have made significant progress in recent
  visual-question-answering (VQA) benchmarks that evaluate complex
  visio-linguistic reasoning. However, are these models truly effective? In this
  work, we show that VLMs still struggle with natural images and questions that
  humans can easily answer, which we term natural adversarial samples. We also
  find it surprisingly easy to generate these VQA samples from natural image-text
  corpora using off-the-shelf models like CLIP and ChatGPT. We propose a
  semi-automated approach to collect a new benchmark, NaturalBench, for reliably
  evaluating VLMs with 10,000 human-verified VQA samples. Crucially, we adopt a
  vision-centric design by pairing each question with two images that
  yield different answers, preventing blind solutions from answering without
  using the images. This makes NaturalBench more challenging than previous
  benchmarks that can be solved with commonsense priors. We evaluate 53
  state-of-the-art VLMs on NaturalBench, showing that models like
  LLaVA-OneVision, Cambrian-1, Llama3.2-Vision, Molmo, Qwen2-VL, and even GPT-4o
  lag 50%-70% behind human performance (over 90%). We analyze why NaturalBench is
  hard from two angles: (1) Compositionality: Solving NaturalBench requires
  diverse visio-linguistic skills, including understanding attribute bindings,
  object relationships, and advanced reasoning like logic and counting. To this
  end, unlike prior work that uses a single tag per sample, we tag each
  NaturalBench sample with 1 to 8 skill tags for fine-grained evaluation. (2)
  Biases: NaturalBench exposes severe biases in VLMs, as models often choose the
  same answer regardless of the image. Lastly, we apply our benchmark curation
  method to diverse data sources, including long captions (over 100 words) and
  non-English languages like Chinese and Hindi, highlighting its potential for
  dynamic evaluations of VLMs.
paper_ai_keywords: vision-centric vision-language models (VLMs) visual-question-answering (VQA) adversarial samples CLIP ChatGPT robust evaluation human-verified benchmark NaturalBench visio-linguistic reasoning vision-centric design commonsense priors attribute bindings object relationships logic reasoning counting multi-modal evaluation skill tags dynamic evaluations long captions non-English languages