title: S1-Bench: A Simple Benchmark for Evaluating System 1 Thinking Capability
    of Large Reasoning Models
summary: We introduce S1-Bench, a novel benchmark designed to evaluate Large Reasoning
  Models' (LRMs) performance on simple tasks that favor intuitive system 1
  thinking rather than deliberative system 2 reasoning. While LRMs have achieved
  significant breakthroughs in complex reasoning tasks through explicit chains of
  thought, their reliance on deep analytical thinking may limit their system 1
  thinking capabilities. Moreover, a lack of benchmark currently exists to
  evaluate LRMs' performance in tasks that require such capabilities. To fill
  this gap, S1-Bench presents a set of simple, diverse, and naturally clear
  questions across multiple domains and languages, specifically designed to
  assess LRMs' performance in such tasks. Our comprehensive evaluation of 22 LRMs
  reveals significant lower efficiency tendencies, with outputs averaging 15.5
  times longer than those of traditional small LLMs. Additionally, LRMs often
  identify correct answers early but continue unnecessary deliberation, with some
  models even producing numerous errors. These findings highlight the rigid
  reasoning patterns of current LRMs and underscore the substantial development
  needed to achieve balanced dual-system thinking capabilities that can adapt
  appropriately to task complexity.
paper_authors._id: 67fdd3a917e86592095a3ab7
paper_authors.name: Wenyuan Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-15T07:54:05.425Z
paper_authors._id: 6617c98901ad3a0642a2a08f
paper_authors.avatarUrl: /avatars/cf52fb511f2f31de7940f9c13d19b8e7.svg
paper_authors.fullname: Wenyuan Zhang
paper_authors.type: user
paper_authors.user: WYRipple
paper_authors._id: 67fdd3a917e86592095a3ab8
paper_authors.name: Shuaiyi Nie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-04-15T08:23:41.149Z
paper_authors._id: 665e84f6152658fe8d478b1f
paper_authors.avatarUrl: /avatars/9f08ce6aa78d7576c97e4feaddf77c1e.svg
paper_authors.fullname: Shuaiyi Nie
paper_authors.type: user
paper_authors.user: ShuaiyiNie
paper_authors._id: 67fdd3a917e86592095a3ab9
paper_authors.name: Xinghua Zhang
paper_authors._id: 67fdd3a917e86592095a3aba
paper_authors.name: Zefeng Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-15T07:54:01.660Z
paper_authors._id: 642c2dcec3694d2b74565c48
paper_authors.avatarUrl: /avatars/31243bb505f8c511ebd7492eaf3ea1a9.svg
paper_authors.fullname: zhangzef
paper_authors.type: user
paper_authors.user: Starrrrrry
paper_authors._id: 67fdd3a917e86592095a3abb
paper_authors.name: Tingwen Liu
paper_title: S1-Bench: A Simple Benchmark for Evaluating System 1 Thinking Capability
    of Large Reasoning Models
paper_summary: We introduce S1-Bench, a novel benchmark designed to evaluate Large Reasoning
  Models' (LRMs) performance on simple tasks that favor intuitive system 1
  thinking rather than deliberative system 2 reasoning. While LRMs have achieved
  significant breakthroughs in complex reasoning tasks through explicit chains of
  thought, their reliance on deep analytical thinking may limit their system 1
  thinking capabilities. Moreover, a lack of benchmark currently exists to
  evaluate LRMs' performance in tasks that require such capabilities. To fill
  this gap, S1-Bench presents a set of simple, diverse, and naturally clear
  questions across multiple domains and languages, specifically designed to
  assess LRMs' performance in such tasks. Our comprehensive evaluation of 22 LRMs
  reveals significant lower efficiency tendencies, with outputs averaging 15.5
  times longer than those of traditional small LLMs. Additionally, LRMs often
  identify correct answers early but continue unnecessary deliberation, with some
  models even producing numerous errors. These findings highlight the rigid
  reasoning patterns of current LRMs and underscore the substantial development
  needed to achieve balanced dual-system thinking capabilities that can adapt
  appropriately to task complexity.
paper_ai_summary: S1-Bench evaluates the efficiency of Large Reasoning Models in simple tasks requiring intuitive thinking, revealing significant inefficiencies and a tendency for unnecessary deliberation.
paper_ai_keywords: Large Reasoning Models system 1 thinking system 2 reasoning benchmark chains of thought complex reasoning tasks LRMs LLMs evaluation reasoning patterns dual-system thinking