title: FINEREASON: Evaluating and Improving LLMs' Deliberate Reasoning through
    Reflective Puzzle Solving
summary: Many challenging reasoning tasks require not just rapid, intuitive responses,
  but a more deliberate, multi-step approach. Recent progress in large language
  models (LLMs) highlights an important shift from the "System 1" way of quick
  reactions to the "System 2" style of reflection-and-correction problem solving.
  However, current benchmarks heavily rely on the final-answer accuracy, leaving
  much of a model's intermediate reasoning steps unexamined. This fails to assess
  the model's ability to reflect and rectify mistakes within the reasoning
  process. To bridge this gap, we introduce FINEREASON, a logic-puzzle benchmark
  for fine-grained evaluation of LLMs' reasoning capabilities. Each puzzle can be
  decomposed into atomic steps, making it ideal for rigorous validation of
  intermediate correctness. Building on this, we introduce two tasks: state
  checking, and state transition, for a comprehensive evaluation of how models
  assess the current situation and plan the next move. To support broader
  research, we also provide a puzzle training set aimed at enhancing performance
  on general mathematical tasks. We show that models trained on our state
  checking and transition data demonstrate gains in math reasoning by up to 5.1%
  on GSM8K.
paper_authors._id: 67c15306333e2f71f01c8e35
paper_authors.name: Guizhen Chen
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:28:41.974Z
paper_authors._id: 64e85b3edb3767299865e0e3
paper_authors.avatarUrl: /avatars/fdbe121535dea940edd2766161393485.svg
paper_authors.fullname: Chen
paper_authors.type: user
paper_authors.user: Guizhen
paper_authors._id: 67c15306333e2f71f01c8e36
paper_authors.name: Weiwen Xu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:28:47.949Z
paper_authors._id: 67627b97fc88502751bfd2b8
paper_authors.avatarUrl: /avatars/4b1f5c333f9255181d7b9078c5d4eb32.svg
paper_authors.fullname: Wei
paper_authors.type: user
paper_authors.user: weiwenxu
paper_authors._id: 67c15306333e2f71f01c8e37
paper_authors.name: Hao Zhang
paper_authors._id: 67c15306333e2f71f01c8e38
paper_authors.name: Hou Pong Chan
paper_authors._id: 67c15306333e2f71f01c8e39
paper_authors.name: Chaoqun Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:28:58.424Z
paper_authors._id: 61657b0b20606e5e73f611cc
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/61657b0b20606e5e73f611cc/6ZPne2GYlWkxrx35ND1P8.png
paper_authors.fullname: CHAOQUN LIU
paper_authors.type: user
paper_authors.user: lukecq
paper_authors._id: 67c15306333e2f71f01c8e3a
paper_authors.name: Lidong Bing
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:29:04.845Z
paper_authors._id: 6454685a548f22be598414c4
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/eMjMWKJ-AouF7eY1-RzGF.jpeg
paper_authors.fullname: Lidong Bing
paper_authors.type: user
paper_authors.user: LidongBing
paper_authors._id: 67c15306333e2f71f01c8e3b
paper_authors.name: Deli Zhao
paper_authors._id: 67c15306333e2f71f01c8e3c
paper_authors.name: Anh Tuan Luu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-02-28T12:29:21.748Z
paper_authors._id: 655722e80438e0854fae7554
paper_authors.avatarUrl: /avatars/b93a74f7c7880f9fe0f3ffb47e2aef5e.svg
paper_authors.fullname: Luu Anh Tuan
paper_authors.type: user
paper_authors.user: anhtuanluu36
paper_authors._id: 67c15306333e2f71f01c8e3d
paper_authors.name: Yu Rong
paper_title: FINEREASON: Evaluating and Improving LLMs' Deliberate Reasoning through
    Reflective Puzzle Solving
paper_summary: Many challenging reasoning tasks require not just rapid, intuitive responses,
  but a more deliberate, multi-step approach. Recent progress in large language
  models (LLMs) highlights an important shift from the "System 1" way of quick
  reactions to the "System 2" style of reflection-and-correction problem solving.
  However, current benchmarks heavily rely on the final-answer accuracy, leaving
  much of a model's intermediate reasoning steps unexamined. This fails to assess
  the model's ability to reflect and rectify mistakes within the reasoning
  process. To bridge this gap, we introduce FINEREASON, a logic-puzzle benchmark
  for fine-grained evaluation of LLMs' reasoning capabilities. Each puzzle can be
  decomposed into atomic steps, making it ideal for rigorous validation of
  intermediate correctness. Building on this, we introduce two tasks: state
  checking, and state transition, for a comprehensive evaluation of how models
  assess the current situation and plan the next move. To support broader
  research, we also provide a puzzle training set aimed at enhancing performance
  on general mathematical tasks. We show that models trained on our state
  checking and transition data demonstrate gains in math reasoning by up to 5.1%
  on GSM8K.
paper_ai_keywords: large language models (LLMs) System 1 System 2 reflection-and-correction problem solving multimodal reasoning intermediate reasoning steps logic-puzzle benchmark FINEREASON fine-grained evaluation optimal checking state transition puzzle training set math reasoning GSM8K