title: VisualPuzzles: Decoupling Multimodal Reasoning Evaluation from Domain
    Knowledge
summary: Current multimodal benchmarks often conflate reasoning with domain-specific
  knowledge, making it difficult to isolate and evaluate general reasoning
  abilities in non-expert settings. To address this, we introduce VisualPuzzles,
  a benchmark that targets visual reasoning while deliberately minimizing
  reliance on specialized knowledge. VisualPuzzles consists of diverse questions
  spanning five categories: algorithmic, analogical, deductive, inductive, and
  spatial reasoning. One major source of our questions is manually translated
  logical reasoning questions from the Chinese Civil Service Examination.
  Experiments show that VisualPuzzles requires significantly less intensive
  domain-specific knowledge and more complex reasoning compared to benchmarks
  like MMMU, enabling us to better evaluate genuine multimodal reasoning.
  Evaluations show that state-of-the-art multimodal large language models
  consistently lag behind human performance on VisualPuzzles, and that strong
  performance on knowledge-intensive benchmarks does not necessarily translate to
  success on reasoning-focused, knowledge-light tasks. Additionally, reasoning
  enhancements such as scaling up inference compute (with "thinking" modes) yield
  inconsistent gains across models and task types, and we observe no clear
  correlation between model size and performance. We also found that models
  exhibit different reasoning and answering patterns on VisualPuzzles compared to
  benchmarks with heavier emphasis on knowledge. VisualPuzzles offers a clearer
  lens through which to evaluate reasoning capabilities beyond factual recall and
  domain knowledge.
paper_authors._id: 67fe51cda72a71760da6d239
paper_authors.name: Yueqi Song
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-17T08:05:16.792Z
paper_authors._id: 63e7bf7be02ee67e8e53f78d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/63e7bf7be02ee67e8e53f78d/z9j1PWrurEyIA2JXzpz0i.png
paper_authors.fullname: Yueqi Song
paper_authors.type: user
paper_authors.user: yueqis
paper_authors._id: 67fe51cda72a71760da6d23a
paper_authors.name: Tianyue Ou
paper_authors._id: 67fe51cda72a71760da6d23b
paper_authors.name: Yibo Kong
paper_authors._id: 67fe51cda72a71760da6d23c
paper_authors.name: Zecheng Li
paper_authors._id: 67fe51cda72a71760da6d23d
paper_authors.name: Graham Neubig
paper_authors._id: 67fe51cda72a71760da6d23e
paper_authors.name: Xiang Yue
paper_title: VisualPuzzles: Decoupling Multimodal Reasoning Evaluation from Domain
    Knowledge
paper_summary: Current multimodal benchmarks often conflate reasoning with domain-specific
  knowledge, making it difficult to isolate and evaluate general reasoning
  abilities in non-expert settings. To address this, we introduce VisualPuzzles,
  a benchmark that targets visual reasoning while deliberately minimizing
  reliance on specialized knowledge. VisualPuzzles consists of diverse questions
  spanning five categories: algorithmic, analogical, deductive, inductive, and
  spatial reasoning. One major source of our questions is manually translated
  logical reasoning questions from the Chinese Civil Service Examination.
  Experiments show that VisualPuzzles requires significantly less intensive
  domain-specific knowledge and more complex reasoning compared to benchmarks
  like MMMU, enabling us to better evaluate genuine multimodal reasoning.
  Evaluations show that state-of-the-art multimodal large language models
  consistently lag behind human performance on VisualPuzzles, and that strong
  performance on knowledge-intensive benchmarks does not necessarily translate to
  success on reasoning-focused, knowledge-light tasks. Additionally, reasoning
  enhancements such as scaling up inference compute (with "thinking" modes) yield
  inconsistent gains across models and task types, and we observe no clear
  correlation between model size and performance. We also found that models
  exhibit different reasoning and answering patterns on VisualPuzzles compared to
  benchmarks with heavier emphasis on knowledge. VisualPuzzles offers a clearer
  lens through which to evaluate reasoning capabilities beyond factual recall and
  domain knowledge.
paper_ai_summary: VisualPuzzles is a benchmark designed to evaluate general visual reasoning abilities by minimizing reliance on domain-specific knowledge, revealing that current multimodal models lag behind humans in these tasks.
paper_ai_keywords: visual reasoning domain-specific knowledge logical reasoning multimodal benchmarks algorithmic reasoning analogical reasoning deductive reasoning inductive reasoning spatial reasoning multimodal large language models reasoning enhancements knowledge-intensive benchmarks