title: How Far Are We from Intelligent Visual Deductive Reasoning?
summary: Vision-Language Models (VLMs) such as GPT-4V have recently demonstrated
  incredible strides on diverse vision language tasks. We dig into vision-based
  deductive reasoning, a more sophisticated but less explored realm, and find
  previously unexposed blindspots in the current SOTA VLMs. Specifically, we
  leverage Raven's Progressive Matrices (RPMs), to assess VLMs' abilities to
  perform multi-hop relational and deductive reasoning relying solely on visual
  clues. We perform comprehensive evaluations of several popular VLMs employing
  standard strategies such as in-context learning, self-consistency, and
  Chain-of-thoughts (CoT) on three diverse datasets, including the Mensa IQ test,
  IntelligenceTest, and RAVEN. The results reveal that despite the impressive
  capabilities of LLMs in text-based reasoning, we are still far from achieving
  comparable proficiency in visual deductive reasoning. We found that certain
  standard strategies that are effective when applied to LLMs do not seamlessly
  translate to the challenges presented by visual reasoning tasks. Moreover, a
  detailed analysis reveals that VLMs struggle to solve these tasks mainly
  because they are unable to perceive and comprehend multiple, confounding
  abstract patterns in RPM examples.
paper_authors._id: 65ea7a22859bf22ab4521798
paper_authors.name: Yizhe Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-08T09:46:24.870Z
paper_authors._id: 5e64858c87403103f9f1055d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/5dd96eb166059660ed1ee413/96-oNO2OLVkAFOojMaV0T.jpeg
paper_authors.fullname: Yizhe Zhang
paper_authors.type: user
paper_authors.user: yizhezhang
paper_authors._id: 65ea7a22859bf22ab4521799
paper_authors.name: He Bai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-03-08T08:33:17.432Z
paper_authors._id: 5f024bd8ff69163f6f59e775
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1652111948107-5f024bd8ff69163f6f59e775.png
paper_authors.fullname: Richard He Bai
paper_authors.type: user
paper_authors.user: richardbaihe
paper_authors._id: 65ea7a22859bf22ab452179a
paper_authors.name: Ruixiang Zhang
paper_authors._id: 65ea7a22859bf22ab452179b
paper_authors.name: Jiatao Gu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-08T09:47:22.141Z
paper_authors._id: 6164e72d73996c363c52e66d
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1634002684894-noauth.png
paper_authors.fullname: Jiatao Gu
paper_authors.type: user
paper_authors.user: thomagram
paper_authors._id: 65ea7a22859bf22ab452179c
paper_authors.name: Shuangfei Zhai
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-13T15:46:05.132Z
paper_authors._id: 675bf04240726aac79ae8e7b
paper_authors.avatarUrl: /avatars/8ae95255dfce91a6f5e76f14eae9e9e5.svg
paper_authors.fullname: Shuangfei Zhai
paper_authors.type: user
paper_authors.user: shuangfei
paper_authors._id: 65ea7a22859bf22ab452179d
paper_authors.name: Josh Susskind
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-11-22T16:30:49.961Z
paper_authors._id: 6470d2247fd7ecdbd0ec3cc9
paper_authors.avatarUrl: /avatars/52c5eca12499a1aa9bd49c43d4f20685.svg
paper_authors.fullname: Joshua M. Susskind
paper_authors.type: user
paper_authors.user: jsusskind
paper_authors._id: 65ea7a22859bf22ab452179e
paper_authors.name: Navdeep Jaitly
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-03-08T09:47:40.093Z
paper_authors._id: 631119df88942700629f4aab
paper_authors.avatarUrl: /avatars/251c20a8fd48831a8c0cc0d198a42455.svg
paper_authors.fullname: Navdeep Jaitly
paper_authors.type: user
paper_authors.user: ndjaitly
paper_title: How Far Are We from Intelligent Visual Deductive Reasoning?
paper_summary: Vision-Language Models (VLMs) such as GPT-4V have recently demonstrated
  incredible strides on diverse vision language tasks. We dig into vision-based
  deductive reasoning, a more sophisticated but less explored realm, and find
  previously unexposed blindspots in the current SOTA VLMs. Specifically, we
  leverage Raven's Progressive Matrices (RPMs), to assess VLMs' abilities to
  perform multi-hop relational and deductive reasoning relying solely on visual
  clues. We perform comprehensive evaluations of several popular VLMs employing
  standard strategies such as in-context learning, self-consistency, and
  Chain-of-thoughts (CoT) on three diverse datasets, including the Mensa IQ test,
  IntelligenceTest, and RAVEN. The results reveal that despite the impressive
  capabilities of LLMs in text-based reasoning, we are still far from achieving
  comparable proficiency in visual deductive reasoning. We found that certain
  standard strategies that are effective when applied to LLMs do not seamlessly
  translate to the challenges presented by visual reasoning tasks. Moreover, a
  detailed analysis reveals that VLMs struggle to solve these tasks mainly
  because they are unable to perceive and comprehend multiple, confounding
  abstract patterns in RPM examples.
paper_ai_summary: VLMs show deficiencies in visual deductive reasoning, particularly in multi-hop relational tasks, despite strong performance in text-based reasoning.
paper_ai_keywords: Vision-Language Models VLMs GPT-4V Raven's Progressive Matrices RPMs in-context learning self-consistency Chain-of-thoughts CoT visual deductive reasoning multi-hop relational reasoning Mensa IQ test IntelligenceTest RAVEN