VisualPuzzles: Decoupling Multimodal Reasoning Evaluation from Domain
  Knowledge
Current multimodal benchmarks often conflate reasoning with domain-specific
knowledge, making it difficult to isolate and evaluate general reasoning
abilities in non-expert settings. To address this, we introduce VisualPuzzles,
a benchmark that targets visual reasoning while deliberately minimizing
reliance on specialized knowledge. VisualPuzzles consists of diverse questions
spanning five categories: algorithmic, analogical, deductive, inductive, and
spatial reasoning. One major source of our questions is manually translated
logical reasoning questions from the Chinese Civil Service Examination.
Experiments show that VisualPuzzles requires significantly less intensive
domain-specific knowledge and more complex reasoning compared to benchmarks
like MMMU, enabling us to better evaluate genuine multimodal reasoning.
Evaluations show that state-of-the-art multimodal large language models
consistently lag behind human performance on VisualPuzzles, and that strong
performance on knowledge-intensive benchmarks does not necessarily translate to
success on reasoning-focused, knowledge-light tasks. Additionally, reasoning
enhancements such as scaling up inference compute (with "thinking" modes) yield
inconsistent gains across models and task types, and we observe no clear
correlation between model size and performance. We also found that models
exhibit different reasoning and answering patterns on VisualPuzzles compared to
benchmarks with heavier emphasis on knowledge. VisualPuzzles offers a clearer
lens through which to evaluate reasoning capabilities beyond factual recall and
domain knowledge.
67fe51cda72a71760da6d239
Yueqi Song
claimed_verified
2025-04-17T08:05:16.792Z
63e7bf7be02ee67e8e53f78d
https://cdn-avatars.huggingface.co/v1/production/uploads/63e7bf7be02ee67e8e53f78d/z9j1PWrurEyIA2JXzpz0i.png
Yueqi Song
user
yueqis
67fe51cda72a71760da6d23a
Tianyue Ou
67fe51cda72a71760da6d23b
Yibo Kong
67fe51cda72a71760da6d23c
Zecheng Li
67fe51cda72a71760da6d23d
Graham Neubig
67fe51cda72a71760da6d23e
Xiang Yue
claimed_verified
2025-07-02T16:08:11.465Z
6230d750d93e84e233882dbc
https://cdn-avatars.huggingface.co/v1/production/uploads/6230d750d93e84e233882dbc/4MGEekLW3oWzqeFWDWvIK.jpeg
Xiang Yue
user
yuexiang96
VisualPuzzles: Decoupling Multimodal Reasoning Evaluation from Domain
  Knowledge
Current multimodal benchmarks often conflate reasoning with domain-specific
knowledge, making it difficult to isolate and evaluate general reasoning
abilities in non-expert settings. To address this, we introduce VisualPuzzles,
a benchmark that targets visual reasoning while deliberately minimizing
reliance on specialized knowledge. VisualPuzzles consists of diverse questions
spanning five categories: algorithmic, analogical, deductive, inductive, and
spatial reasoning. One major source of our questions is manually translated
logical reasoning questions from the Chinese Civil Service Examination.
Experiments show that VisualPuzzles requires significantly less intensive
domain-specific knowledge and more complex reasoning compared to benchmarks
like MMMU, enabling us to better evaluate genuine multimodal reasoning.
Evaluations show that state-of-the-art multimodal large language models
consistently lag behind human performance on VisualPuzzles, and that strong
performance on knowledge-intensive benchmarks does not necessarily translate to
success on reasoning-focused, knowledge-light tasks. Additionally, reasoning
enhancements such as scaling up inference compute (with "thinking" modes) yield
inconsistent gains across models and task types, and we observe no clear
correlation between model size and performance. We also found that models
exhibit different reasoning and answering patterns on VisualPuzzles compared to
benchmarks with heavier emphasis on knowledge. VisualPuzzles offers a clearer
lens through which to evaluate reasoning capabilities beyond factual recall and
domain knowledge.
VisualPuzzles is a benchmark designed to evaluate general visual reasoning abilities by minimizing reliance on domain-specific knowledge, revealing that current multimodal models lag behind humans in these tasks.
visual reasoning domain-specific knowledge logical reasoning multimodal benchmarks algorithmic reasoning analogical reasoning deductive reasoning inductive reasoning spatial reasoning multimodal large language models reasoning enhancements knowledge-intensive benchmarks