title: VisualSimpleQA: A Benchmark for Decoupled Evaluation of Large
    Vision-Language Models in Fact-Seeking Question Answering
summary: Large vision-language models (LVLMs) have demonstrated remarkable
  achievements, yet the generation of non-factual responses remains prevalent in
  fact-seeking question answering (QA). Current multimodal fact-seeking
  benchmarks primarily focus on comparing model outputs to ground truth answers,
  providing limited insights into the performance of modality-specific modules.
  To bridge this gap, we introduce VisualSimpleQA, a multimodal fact-seeking
  benchmark with two key features. First, it enables streamlined and decoupled
  evaluation of LVLMs in visual and linguistic modalities. Second, it
  incorporates well-defined difficulty criteria to guide human annotation and
  facilitates the extraction of a challenging subset, VisualSimpleQA-hard.
  Experiments on 15 LVLMs show that even state-of-the-art models such as GPT-4o
  achieve merely 60%+ correctness in multimodal fact-seeking QA on VisualSimpleQA
  and 30%+ on VisualSimpleQA-hard. Furthermore, the decoupled evaluation across
  these models highlights substantial opportunities for improvement in both
  visual and linguistic modules. The dataset is available at
  https://huggingface.co/datasets/WYLing/VisualSimpleQA.
paper_authors._id: 67cfe557ad91643b5cb7d2c6
paper_authors.name: Yanling Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-11T16:09:28.071Z
paper_authors._id: 67cd327432668b04f4555270
paper_authors.avatarUrl: /avatars/15e2cef976cbe05c4c5858c88dccf4af.svg
paper_authors.fullname: Yanling Wang
paper_authors.type: user
paper_authors.user: WYLing
paper_authors._id: 67cfe557ad91643b5cb7d2c7
paper_authors.name: Yihan Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-13T08:24:35.205Z
paper_authors._id: 6762d3aa4e22f0dd279df94e
paper_authors.avatarUrl: /avatars/71f113a0d75da9d5af79ca440a5bce15.svg
paper_authors.fullname: Yihan Zhao
paper_authors.type: user
paper_authors.user: xhansome
paper_authors._id: 67cfe557ad91643b5cb7d2c8
paper_authors.name: Xiaodong Chen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-13T08:24:37.528Z
paper_authors._id: 67be7cd21616162fc336cb44
paper_authors.avatarUrl: /avatars/e58cc3c2d1484419222a5ccfc11f5c48.svg
paper_authors.fullname: Xiaodong Chen
paper_authors.type: user
paper_authors.user: XiaodongChen
paper_authors._id: 67cfe557ad91643b5cb7d2c9
paper_authors.name: Shasha Guo
paper_authors._id: 67cfe557ad91643b5cb7d2ca
paper_authors.name: Lixin Liu
paper_authors._id: 67cfe557ad91643b5cb7d2cb
paper_authors.name: Haoyang Li
paper_authors._id: 67cfe557ad91643b5cb7d2cc
paper_authors.name: Yong Xiao
paper_authors._id: 67cfe557ad91643b5cb7d2cd
paper_authors.name: Jing Zhang
paper_authors._id: 67cfe557ad91643b5cb7d2ce
paper_authors.name: Qi Li
paper_authors._id: 67cfe557ad91643b5cb7d2cf
paper_authors.name: Ke Xu
paper_title: VisualSimpleQA: A Benchmark for Decoupled Evaluation of Large
    Vision-Language Models in Fact-Seeking Question Answering
paper_summary: Large vision-language models (LVLMs) have demonstrated remarkable
  achievements, yet the generation of non-factual responses remains prevalent in
  fact-seeking question answering (QA). Current multimodal fact-seeking
  benchmarks primarily focus on comparing model outputs to ground truth answers,
  providing limited insights into the performance of modality-specific modules.
  To bridge this gap, we introduce VisualSimpleQA, a multimodal fact-seeking
  benchmark with two key features. First, it enables streamlined and decoupled
  evaluation of LVLMs in visual and linguistic modalities. Second, it
  incorporates well-defined difficulty criteria to guide human annotation and
  facilitates the extraction of a challenging subset, VisualSimpleQA-hard.
  Experiments on 15 LVLMs show that even state-of-the-art models such as GPT-4o
  achieve merely 60%+ correctness in multimodal fact-seeking QA on VisualSimpleQA
  and 30%+ on VisualSimpleQA-hard. Furthermore, the decoupled evaluation across
  these models highlights substantial opportunities for improvement in both
  visual and linguistic modules. The dataset is available at
  https://huggingface.co/datasets/WYLing/VisualSimpleQA.
paper_ai_keywords: Large vision-language models fact-seeking question answering multimodal benchmarks visual modality linguistic modality VisualSimpleQA VisualSimpleQA-hard GPT-4 multimodal fact-seeking QA decoupled evaluation