title: Can Large Reasoning Models do Analogical Reasoning under Perceptual
    Uncertainty?
summary: This work presents a first evaluation of two state-of-the-art Large Reasoning
  Models (LRMs), OpenAI's o3-mini and DeepSeek R1, on analogical reasoning,
  focusing on well-established nonverbal human IQ tests based on Raven's
  progressive matrices. We benchmark with the I-RAVEN dataset and its more
  difficult extension, I-RAVEN-X, which tests the ability to generalize to longer
  reasoning rules and ranges of the attribute values. To assess the influence of
  visual uncertainties on these nonverbal analogical reasoning tests, we extend
  the I-RAVEN-X dataset, which otherwise assumes an oracle perception. We adopt a
  two-fold strategy to simulate this imperfect visual perception: 1) we introduce
  confounding attributes which, being sampled at random, do not contribute to the
  prediction of the correct answer of the puzzles and 2) smoothen the
  distributions of the input attributes' values. We observe a sharp decline in
  OpenAI's o3-mini task accuracy, dropping from 86.6% on the original I-RAVEN to
  just 17.0% -- approaching random chance -- on the more challenging I-RAVEN-X,
  which increases input length and range and emulates perceptual uncertainty.
  This drop occurred despite spending 3.4x more reasoning tokens. A similar trend
  is also observed for DeepSeek R1: from 80.6% to 23.2%. On the other hand, a
  neuro-symbolic probabilistic abductive model, ARLC, that achieves
  state-of-the-art performances on I-RAVEN, can robustly reason under all these
  out-of-distribution tests, maintaining strong accuracy with only a modest
  reduction from 98.6% to 88.0%. Our code is available at
  https://github.com/IBM/raven-large-language-models.
paper_authors._id: 67d82c6e6d6baf84fba32ea1
paper_authors.name: Giacomo Camposampiero
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-17T15:07:03.399Z
paper_authors._id: 6395beb3b5af054482d87456
paper_authors.avatarUrl: /avatars/b24ae0d2e3fea8ad716d1b7225d9e722.svg
paper_authors.fullname: Giacomo Camposampiero
paper_authors.type: user
paper_authors.user: gcamposampie
paper_authors._id: 67d82c6e6d6baf84fba32ea2
paper_authors.name: Michael Hersche
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-03-17T14:06:39.558Z
paper_authors._id: 668f96ca35cddb688ad9f2cf
paper_authors.avatarUrl: /avatars/eb4febcf47b424732ee0a73a81c3b6dd.svg
paper_authors.fullname: Michael Hersche
paper_authors.type: user
paper_authors.user: mihoesche
paper_authors._id: 67d82c6e6d6baf84fba32ea3
paper_authors.name: Roger Wattenhofer
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-03-17T14:06:39.558Z
paper_authors._id: 64afd0c109d6573c1d89dbf7
paper_authors.avatarUrl: /avatars/ed6bd7133517590f46847a0930d7a049.svg
paper_authors.fullname: Roger Wattenhofer
paper_authors.type: user
paper_authors.user: Rogrrr
paper_authors._id: 67d82c6e6d6baf84fba32ea4
paper_authors.name: Abu Sebastian
paper_authors._id: 67d82c6e6d6baf84fba32ea5
paper_authors.name: Abbas Rahimi
paper_title: Can Large Reasoning Models do Analogical Reasoning under Perceptual
    Uncertainty?
paper_summary: This work presents a first evaluation of two state-of-the-art Large Reasoning
  Models (LRMs), OpenAI's o3-mini and DeepSeek R1, on analogical reasoning,
  focusing on well-established nonverbal human IQ tests based on Raven's
  progressive matrices. We benchmark with the I-RAVEN dataset and its more
  difficult extension, I-RAVEN-X, which tests the ability to generalize to longer
  reasoning rules and ranges of the attribute values. To assess the influence of
  visual uncertainties on these nonverbal analogical reasoning tests, we extend
  the I-RAVEN-X dataset, which otherwise assumes an oracle perception. We adopt a
  two-fold strategy to simulate this imperfect visual perception: 1) we introduce
  confounding attributes which, being sampled at random, do not contribute to the
  prediction of the correct answer of the puzzles and 2) smoothen the
  distributions of the input attributes' values. We observe a sharp decline in
  OpenAI's o3-mini task accuracy, dropping from 86.6% on the original I-RAVEN to
  just 17.0% -- approaching random chance -- on the more challenging I-RAVEN-X,
  which increases input length and range and emulates perceptual uncertainty.
  This drop occurred despite spending 3.4x more reasoning tokens. A similar trend
  is also observed for DeepSeek R1: from 80.6% to 23.2%. On the other hand, a
  neuro-symbolic probabilistic abductive model, ARLC, that achieves
  state-of-the-art performances on I-RAVEN, can robustly reason under all these
  out-of-distribution tests, maintaining strong accuracy with only a modest
  reduction from 98.6% to 88.0%. Our code is available at
  https://github.com/IBM/raven-large-language-models.
paper_ai_keywords: Large Reasoning Models (LRMs) OpenAI's o3-mini DeepSeek R1 analogical reasoning Raven's progressive matrices I-RAVEN dataset I-RAVEN-X visual uncertainties oracle perception confounding attributes input attributes reasoning tokens neuro-symbolic probabilistic abductive model ARLC state-of-the-art performances out-of-distribution tests