S1-Bench: A Simple Benchmark for Evaluating System 1 Thinking Capability
  of Large Reasoning Models
We introduce S1-Bench, a novel benchmark designed to evaluate Large Reasoning
Models' (LRMs) performance on simple tasks that favor intuitive system 1
thinking rather than deliberative system 2 reasoning. While LRMs have achieved
significant breakthroughs in complex reasoning tasks through explicit chains of
thought, their reliance on deep analytical thinking may limit their system 1
thinking capabilities. Moreover, a lack of benchmark currently exists to
evaluate LRMs' performance in tasks that require such capabilities. To fill
this gap, S1-Bench presents a set of simple, diverse, and naturally clear
questions across multiple domains and languages, specifically designed to
assess LRMs' performance in such tasks. Our comprehensive evaluation of 22 LRMs
reveals significant lower efficiency tendencies, with outputs averaging 15.5
times longer than those of traditional small LLMs. Additionally, LRMs often
identify correct answers early but continue unnecessary deliberation, with some
models even producing numerous errors. These findings highlight the rigid
reasoning patterns of current LRMs and underscore the substantial development
needed to achieve balanced dual-system thinking capabilities that can adapt
appropriately to task complexity.
67fdd3a917e86592095a3ab7
Wenyuan Zhang
claimed_verified
2025-04-15T07:54:05.425Z
6617c98901ad3a0642a2a08f
/avatars/cf52fb511f2f31de7940f9c13d19b8e7.svg
Wenyuan Zhang
user
WYRipple
67fdd3a917e86592095a3ab8
Shuaiyi Nie
admin_assigned
2025-04-15T08:23:41.149Z
665e84f6152658fe8d478b1f
/avatars/9f08ce6aa78d7576c97e4feaddf77c1e.svg
Shuaiyi Nie
user
ShuaiyiNie
67fdd3a917e86592095a3ab9
Xinghua Zhang
67fdd3a917e86592095a3aba
Zefeng Zhang
claimed_verified
2025-04-15T07:54:01.660Z
642c2dcec3694d2b74565c48
/avatars/31243bb505f8c511ebd7492eaf3ea1a9.svg
zhangzef
user
Starrrrrry
67fdd3a917e86592095a3abb
Tingwen Liu
S1-Bench: A Simple Benchmark for Evaluating System 1 Thinking Capability
  of Large Reasoning Models
We introduce S1-Bench, a novel benchmark designed to evaluate Large Reasoning
Models' (LRMs) performance on simple tasks that favor intuitive system 1
thinking rather than deliberative system 2 reasoning. While LRMs have achieved
significant breakthroughs in complex reasoning tasks through explicit chains of
thought, their reliance on deep analytical thinking may limit their system 1
thinking capabilities. Moreover, a lack of benchmark currently exists to
evaluate LRMs' performance in tasks that require such capabilities. To fill
this gap, S1-Bench presents a set of simple, diverse, and naturally clear
questions across multiple domains and languages, specifically designed to
assess LRMs' performance in such tasks. Our comprehensive evaluation of 22 LRMs
reveals significant lower efficiency tendencies, with outputs averaging 15.5
times longer than those of traditional small LLMs. Additionally, LRMs often
identify correct answers early but continue unnecessary deliberation, with some
models even producing numerous errors. These findings highlight the rigid
reasoning patterns of current LRMs and underscore the substantial development
needed to achieve balanced dual-system thinking capabilities that can adapt
appropriately to task complexity.
S1-Bench evaluates the efficiency of Large Reasoning Models in simple tasks requiring intuitive thinking, revealing significant inefficiencies and a tendency for unnecessary deliberation.
Large Reasoning Models system 1 thinking system 2 reasoning benchmark chains of thought complex reasoning tasks LRMs LLMs evaluation reasoning patterns dual-system thinking