title: IQBench: How "Smart'' Are Vision-Language Models? A Study with Human IQ
    Tests
summary: Although large Vision-Language Models (VLMs) have demonstrated remarkable
  performance in a wide range of multimodal tasks, their true reasoning
  capabilities on human IQ tests remain underexplored. To advance research on the
  fluid intelligence of VLMs, we introduce **IQBench**, a new benchmark designed
  to evaluate VLMs on standardized visual IQ tests. We focus on evaluating the
  reasoning capabilities of VLMs, which we argue are more important than the
  accuracy of the final prediction. **Our benchmark is visually centric,
  minimizing the dependence on unnecessary textual content**, thus encouraging
  models to derive answers primarily from image-based information rather than
  learned textual knowledge. To this end, we manually collected and annotated 500
  visual IQ questions to **prevent unintentional data leakage during training**.
  Unlike prior work that focuses primarily on the accuracy of the final answer,
  we evaluate the reasoning ability of the models by assessing their explanations
  and the patterns used to solve each problem, along with the accuracy of the
  final prediction and human evaluation. Our experiments show that there are
  substantial performance disparities between tasks, with models such as
  `o4-mini`, `gemini-2.5-flash`, and `claude-3.7-sonnet` achieving the highest
  average accuracies of 0.615, 0.578, and 0.548, respectively. However, all
  models struggle with 3D spatial and anagram reasoning tasks, highlighting
  significant limitations in current VLMs' general reasoning abilities. In terms
  of reasoning scores, `o4-mini`, `gemini-2.5-flash`, and `claude-3.7-sonnet`
  achieved top averages of 0.696, 0.586, and 0.516, respectively. These results
  highlight inconsistencies between the reasoning processes of the models and
  their final answers, emphasizing the importance of evaluating the accuracy of
  the reasoning in addition to the final predictions.
paper_authors._id: 6838fb5e9deef11aa62251ee
paper_authors.name: Tan-Hanh Pham
paper_authors._id: 6838fb5e9deef11aa62251ef
paper_authors.name: Phu-Vinh Nguyen
paper_authors._id: 6838fb5e9deef11aa62251f0
paper_authors.name: Dang The Hung
paper_authors._id: 6838fb5e9deef11aa62251f1
paper_authors.name: Bui Trong Duong
paper_authors._id: 6838fb5e9deef11aa62251f2
paper_authors.name: Vu Nguyen Thanh
paper_authors._id: 6838fb5e9deef11aa62251f3
paper_authors.name: Chris Ngo
paper_authors._id: 6838fb5e9deef11aa62251f4
paper_authors.name: Tri Quang Truong
paper_authors._id: 6838fb5e9deef11aa62251f5
paper_authors.name: Truong-Son Hy
paper_title: IQBench: How "Smart'' Are Vision-Language Models? A Study with Human IQ
    Tests
paper_summary: Although large Vision-Language Models (VLMs) have demonstrated remarkable
  performance in a wide range of multimodal tasks, their true reasoning
  capabilities on human IQ tests remain underexplored. To advance research on the
  fluid intelligence of VLMs, we introduce **IQBench**, a new benchmark designed
  to evaluate VLMs on standardized visual IQ tests. We focus on evaluating the
  reasoning capabilities of VLMs, which we argue are more important than the
  accuracy of the final prediction. **Our benchmark is visually centric,
  minimizing the dependence on unnecessary textual content**, thus encouraging
  models to derive answers primarily from image-based information rather than
  learned textual knowledge. To this end, we manually collected and annotated 500
  visual IQ questions to **prevent unintentional data leakage during training**.
  Unlike prior work that focuses primarily on the accuracy of the final answer,
  we evaluate the reasoning ability of the models by assessing their explanations
  and the patterns used to solve each problem, along with the accuracy of the
  final prediction and human evaluation. Our experiments show that there are
  substantial performance disparities between tasks, with models such as
  `o4-mini`, `gemini-2.5-flash`, and `claude-3.7-sonnet` achieving the highest
  average accuracies of 0.615, 0.578, and 0.548, respectively. However, all
  models struggle with 3D spatial and anagram reasoning tasks, highlighting
  significant limitations in current VLMs' general reasoning abilities. In terms
  of reasoning scores, `o4-mini`, `gemini-2.5-flash`, and `claude-3.7-sonnet`
  achieved top averages of 0.696, 0.586, and 0.516, respectively. These results
  highlight inconsistencies between the reasoning processes of the models and
  their final answers, emphasizing the importance of evaluating the accuracy of
  the reasoning in addition to the final predictions.
paper_ai_summary: IQBench evaluates the reasoning capabilities of Vision-Language Models on visual IQ tests, focusing on pattern recognition and explanation quality rather than final accuracy alone.
paper_ai_keywords: IQBench Vision-Language Models VLMs visual IQ tests reasoning capabilities pattern recognition explanation quality