FINEREASON: Evaluating and Improving LLMs' Deliberate Reasoning through
  Reflective Puzzle Solving
Many challenging reasoning tasks require not just rapid, intuitive responses,
but a more deliberate, multi-step approach. Recent progress in large language
models (LLMs) highlights an important shift from the "System 1" way of quick
reactions to the "System 2" style of reflection-and-correction problem solving.
However, current benchmarks heavily rely on the final-answer accuracy, leaving
much of a model's intermediate reasoning steps unexamined. This fails to assess
the model's ability to reflect and rectify mistakes within the reasoning
process. To bridge this gap, we introduce FINEREASON, a logic-puzzle benchmark
for fine-grained evaluation of LLMs' reasoning capabilities. Each puzzle can be
decomposed into atomic steps, making it ideal for rigorous validation of
intermediate correctness. Building on this, we introduce two tasks: state
checking, and state transition, for a comprehensive evaluation of how models
assess the current situation and plan the next move. To support broader
research, we also provide a puzzle training set aimed at enhancing performance
on general mathematical tasks. We show that models trained on our state
checking and transition data demonstrate gains in math reasoning by up to 5.1%
on GSM8K.
67c15306333e2f71f01c8e35
Guizhen Chen
admin_assigned
2025-02-28T12:28:41.974Z
64e85b3edb3767299865e0e3
/avatars/fdbe121535dea940edd2766161393485.svg
Chen
user
Guizhen
67c15306333e2f71f01c8e36
Weiwen Xu
admin_assigned
2025-02-28T12:28:47.949Z
67627b97fc88502751bfd2b8
/avatars/4b1f5c333f9255181d7b9078c5d4eb32.svg
Wei
user
weiwenxu
67c15306333e2f71f01c8e37
Hao Zhang
claimed_verified
2025-06-10T11:21:10.845Z
64b7cd74ff6d81ae297feded
/avatars/880fbc96cc093f5e901ce84f32a1d21d.svg
ZHANG HAO
user
26hzhang
67c15306333e2f71f01c8e38
Hou Pong Chan
67c15306333e2f71f01c8e39
Chaoqun Liu
admin_assigned
2025-02-28T12:28:58.424Z
61657b0b20606e5e73f611cc
https://cdn-avatars.huggingface.co/v1/production/uploads/61657b0b20606e5e73f611cc/6ZPne2GYlWkxrx35ND1P8.png
CHAOQUN LIU
user
lukecq
67c15306333e2f71f01c8e3a
Lidong Bing
admin_assigned
2025-02-28T12:29:04.845Z
6454685a548f22be598414c4
https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/eMjMWKJ-AouF7eY1-RzGF.jpeg
Lidong Bing
user
LidongBing
67c15306333e2f71f01c8e3b
Deli Zhao
67c15306333e2f71f01c8e3c
Anh Tuan Luu
admin_assigned
2025-02-28T12:29:21.748Z
655722e80438e0854fae7554
/avatars/b93a74f7c7880f9fe0f3ffb47e2aef5e.svg
Luu Anh Tuan
user
anhtuanluu36
67c15306333e2f71f01c8e3d
Yu Rong
claimed_verified
2025-06-10T09:30:57.622Z
642eecbf9b2484d7d8526781
https://cdn-avatars.huggingface.co/v1/production/uploads/642eecbf9b2484d7d8526781/4IvGbd66s49Wx5pZyZGHA.png
Yu Rong
user
Swrooy
FINEREASON: Evaluating and Improving LLMs' Deliberate Reasoning through
  Reflective Puzzle Solving
Many challenging reasoning tasks require not just rapid, intuitive responses,
but a more deliberate, multi-step approach. Recent progress in large language
models (LLMs) highlights an important shift from the "System 1" way of quick
reactions to the "System 2" style of reflection-and-correction problem solving.
However, current benchmarks heavily rely on the final-answer accuracy, leaving
much of a model's intermediate reasoning steps unexamined. This fails to assess
the model's ability to reflect and rectify mistakes within the reasoning
process. To bridge this gap, we introduce FINEREASON, a logic-puzzle benchmark
for fine-grained evaluation of LLMs' reasoning capabilities. Each puzzle can be
decomposed into atomic steps, making it ideal for rigorous validation of
intermediate correctness. Building on this, we introduce two tasks: state
checking, and state transition, for a comprehensive evaluation of how models
assess the current situation and plan the next move. To support broader
research, we also provide a puzzle training set aimed at enhancing performance
on general mathematical tasks. We show that models trained on our state
checking and transition data demonstrate gains in math reasoning by up to 5.1%
on GSM8K.
A new logic-puzzle benchmark, FINEREASON, evaluates LLMs' intermediate reasoning steps and demonstrates improved math reasoning capabilities.
large language models LLMs System 1 System 2 reflection-and-correction reasoning tasks logic-puzzle benchmark intermediate reasoning state checking state transition GSM8K