title: Are Your LLMs Capable of Stable Reasoning?
summary: The rapid advancement of Large Language Models (LLMs) has demonstrated
  remarkable progress in complex reasoning tasks. However, a significant
  discrepancy persists between benchmark performances and real-world
  applications. We identify this gap as primarily stemming from current
  evaluation protocols and metrics, which inadequately capture the full spectrum
  of LLM capabilities, particularly in complex reasoning tasks where both
  accuracy and consistency are crucial. This work makes two key contributions.
  First, we introduce G-Pass@k, a novel evaluation metric that provides a
  continuous assessment of model performance across multiple sampling attempts,
  quantifying both the model's peak performance potential and its stability.
  Second, we present LiveMathBench, a dynamic benchmark comprising challenging,
  contemporary mathematical problems designed to minimize data leakage risks
  during evaluation. Through extensive experiments using G-Pass@k on
  state-of-the-art LLMs with LiveMathBench, we provide comprehensive insights
  into both their maximum capabilities and operational consistency. Our findings
  reveal substantial room for improvement in LLMs' "realistic" reasoning
  capabilities, highlighting the need for more robust evaluation methods. The
  benchmark and detailed results are available at:
  https://github.com/open-compass/GPassK.
paper_authors._id: 67623bb9d44ba09e9119fe12
paper_authors.name: Junnan Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T12:55:23.904Z
paper_authors._id: 643d26979347842571bc9613
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/noauth/3heFf7h3jbhhJWJ4JfGfh.jpeg
paper_authors.fullname: Junnan Liu
paper_authors.type: user
paper_authors.user: jnanliu
paper_authors._id: 67623bb9d44ba09e9119fe13
paper_authors.name: Hongwei Liu
paper_authors._id: 67623bb9d44ba09e9119fe14
paper_authors.name: Linchen Xiao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T12:55:51.678Z
paper_authors._id: 64f58f279eaf9d8fb746cd0e
paper_authors.avatarUrl: /avatars/227db72e089026012694ff16cdf102d0.svg
paper_authors.fullname: LinchenXiao
paper_authors.type: user
paper_authors.user: LinchenXiao
paper_authors._id: 67623bb9d44ba09e9119fe15
paper_authors.name: Ziyi Wang
paper_authors._id: 67623bb9d44ba09e9119fe16
paper_authors.name: Kuikun Liu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-18T09:52:14.336Z
paper_authors._id: 63fd691794cc8f815d50c112
paper_authors.avatarUrl: /avatars/87305d1cbfcc717e910ccdfaf0568f80.svg
paper_authors.fullname: liu
paper_authors.type: user
paper_authors.user: Harold-lkk
paper_authors._id: 67623bb9d44ba09e9119fe17
paper_authors.name: Songyang Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T12:56:48.903Z
paper_authors._id: 650ab54e23196fb2d86b486b
paper_authors.avatarUrl: /avatars/e0506393589695b553ec9ee3fe99b93a.svg
paper_authors.fullname: SongYang Gao
paper_authors.type: user
paper_authors.user: Wizardcoast
paper_authors._id: 67623bb9d44ba09e9119fe18
paper_authors.name: Wenwei Zhang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-18T12:56:56.862Z
paper_authors._id: 64e8505321540e1da3226b54
paper_authors.avatarUrl: /avatars/18958b8406d1ce492b54c1c839f18c54.svg
paper_authors.fullname: Wenwei Zhang
paper_authors.type: user
paper_authors.user: ZwwWayne
paper_authors._id: 67623bb9d44ba09e9119fe19
paper_authors.name: Songyang Zhang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-18T09:52:35.610Z
paper_authors._id: 630716d11801ecc7d2595021
paper_authors.avatarUrl: /avatars/2d36a880ce4a3cf7efc5ff3987dbeaf3.svg
paper_authors.fullname: Songyang Zhang
paper_authors.type: user
paper_authors.user: zsytony
paper_authors._id: 67623bb9d44ba09e9119fe1a
paper_authors.name: Kai Chen
paper_title: Are Your LLMs Capable of Stable Reasoning?
paper_summary: The rapid advancement of Large Language Models (LLMs) has demonstrated
  remarkable progress in complex reasoning tasks. However, a significant
  discrepancy persists between benchmark performances and real-world
  applications. We identify this gap as primarily stemming from current
  evaluation protocols and metrics, which inadequately capture the full spectrum
  of LLM capabilities, particularly in complex reasoning tasks where both
  accuracy and consistency are crucial. This work makes two key contributions.
  First, we introduce G-Pass@k, a novel evaluation metric that provides a
  continuous assessment of model performance across multiple sampling attempts,
  quantifying both the model's peak performance potential and its stability.
  Second, we present LiveMathBench, a dynamic benchmark comprising challenging,
  contemporary mathematical problems designed to minimize data leakage risks
  during evaluation. Through extensive experiments using G-Pass@k on
  state-of-the-art LLMs with LiveMathBench, we provide comprehensive insights
  into both their maximum capabilities and operational consistency. Our findings
  reveal substantial room for improvement in LLMs' "realistic" reasoning
  capabilities, highlighting the need for more robust evaluation methods. The
  benchmark and detailed results are available at:
  https://github.com/open-compass/GPassK.
paper_ai_summary: G-Pass@k and LiveMathBench assess Large Language Models' reasoning capabilities and consistency using complex mathematical problems.
paper_ai_keywords: Large Language Models G-Pass@k LiveMathBench complex reasoning tasks sampling attempts data leakage evaluation metrics benchmark reasoning capabilities