VisualSimpleQA: A Benchmark for Decoupled Evaluation of Large
  Vision-Language Models in Fact-Seeking Question Answering
Large vision-language models (LVLMs) have demonstrated remarkable
achievements, yet the generation of non-factual responses remains prevalent in
fact-seeking question answering (QA). Current multimodal fact-seeking
benchmarks primarily focus on comparing model outputs to ground truth answers,
providing limited insights into the performance of modality-specific modules.
To bridge this gap, we introduce VisualSimpleQA, a multimodal fact-seeking
benchmark with two key features. First, it enables streamlined and decoupled
evaluation of LVLMs in visual and linguistic modalities. Second, it
incorporates well-defined difficulty criteria to guide human annotation and
facilitates the extraction of a challenging subset, VisualSimpleQA-hard.
Experiments on 15 LVLMs show that even state-of-the-art models such as GPT-4o
achieve merely 60%+ correctness in multimodal fact-seeking QA on VisualSimpleQA
and 30%+ on VisualSimpleQA-hard. Furthermore, the decoupled evaluation across
these models highlights substantial opportunities for improvement in both
visual and linguistic modules. The dataset is available at
https://huggingface.co/datasets/WYLing/VisualSimpleQA.
67cfe557ad91643b5cb7d2c6
Yanling Wang
claimed_verified
2025-03-11T16:09:28.071Z
67cd327432668b04f4555270
/avatars/15e2cef976cbe05c4c5858c88dccf4af.svg
Yanling Wang
user
WYLing
67cfe557ad91643b5cb7d2c7
Yihan Zhao
claimed_verified
2025-03-13T08:24:35.205Z
6762d3aa4e22f0dd279df94e
/avatars/71f113a0d75da9d5af79ca440a5bce15.svg
Yihan Zhao
user
xhansome
67cfe557ad91643b5cb7d2c8
Xiaodong Chen
claimed_verified
2025-03-13T08:24:37.528Z
67be7cd21616162fc336cb44
/avatars/e58cc3c2d1484419222a5ccfc11f5c48.svg
Xiaodong Chen
user
XiaodongChen
67cfe557ad91643b5cb7d2c9
Shasha Guo
67cfe557ad91643b5cb7d2ca
Lixin Liu
67cfe557ad91643b5cb7d2cb
Haoyang Li
67cfe557ad91643b5cb7d2cc
Yong Xiao
67cfe557ad91643b5cb7d2cd
Jing Zhang
67cfe557ad91643b5cb7d2ce
Qi Li
67cfe557ad91643b5cb7d2cf
Ke Xu
VisualSimpleQA: A Benchmark for Decoupled Evaluation of Large
  Vision-Language Models in Fact-Seeking Question Answering
Large vision-language models (LVLMs) have demonstrated remarkable
achievements, yet the generation of non-factual responses remains prevalent in
fact-seeking question answering (QA). Current multimodal fact-seeking
benchmarks primarily focus on comparing model outputs to ground truth answers,
providing limited insights into the performance of modality-specific modules.
To bridge this gap, we introduce VisualSimpleQA, a multimodal fact-seeking
benchmark with two key features. First, it enables streamlined and decoupled
evaluation of LVLMs in visual and linguistic modalities. Second, it
incorporates well-defined difficulty criteria to guide human annotation and
facilitates the extraction of a challenging subset, VisualSimpleQA-hard.
Experiments on 15 LVLMs show that even state-of-the-art models such as GPT-4o
achieve merely 60%+ correctness in multimodal fact-seeking QA on VisualSimpleQA
and 30%+ on VisualSimpleQA-hard. Furthermore, the decoupled evaluation across
these models highlights substantial opportunities for improvement in both
visual and linguistic modules. The dataset is available at
https://huggingface.co/datasets/WYLing/VisualSimpleQA.
VisualSimpleQA is a multimodal benchmark that decouples the evaluation of visual and linguistic components in large vision-language models, revealing low accuracy and opportunities for improvement.
large vision-language models fact-seeking question answering multimodal fact-seeking benchmarks visualSimpleQA visual modality linguistic modality decoupled evaluation difficulty criteria multimodal fact-seeking QA VisualSimpleQA-hard