title: Evaluation is All You Need: Strategic Overclaiming of LLM Reasoning
    Capabilities Through Evaluation Design
summary: Reasoning models represented by the Deepseek-R1-Distill series have been
  widely adopted by the open-source community due to their strong performance in
  mathematics, science, programming, and other domains. However, our study
  reveals that their benchmark evaluation results are subject to significant
  fluctuations caused by various factors. Subtle differences in evaluation
  conditions can lead to substantial variations in results. Similar phenomena are
  observed in other open-source inference models fine-tuned based on the
  Deepseek-R1-Distill series, as well as in the QwQ-32B model, making their
  claimed performance improvements difficult to reproduce reliably. Therefore, we
  advocate for the establishment of a more rigorous paradigm for model
  performance evaluation and present our empirical assessments of the
  Deepseek-R1-Distill series models.
paper_authors._id: 6842537f1c4f28a2031f499c
paper_authors.name: Lin Sun
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-06T07:41:08.211Z
paper_authors._id: 632c30576bcb864974cc40a8
paper_authors.avatarUrl: /avatars/96aa948ad1dd35d355e20b5765a2563a.svg
paper_authors.fullname: sunlin
paper_authors.type: user
paper_authors.user: lincharliesun
paper_authors._id: 6842537f1c4f28a2031f499d
paper_authors.name: Weihong Lin
paper_authors._id: 6842537f1c4f28a2031f499e
paper_authors.name: Jinzhu Wu
paper_authors._id: 6842537f1c4f28a2031f499f
paper_authors.name: Yongfu Zhu
paper_authors._id: 6842537f1c4f28a2031f49a0
paper_authors.name: Xiaoqi Jian
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-06-07T05:48:55.525Z
paper_authors._id: 64be2455b567ae97c34bb948
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64be2455b567ae97c34bb948/QuCdStDDGaXjDmp4V-dBj.jpeg
paper_authors.fullname: xiaoqijian
paper_authors.type: user
paper_authors.user: mx1024
paper_authors._id: 6842537f1c4f28a2031f49a1
paper_authors.name: Guangxiang Zhao
paper_authors._id: 6842537f1c4f28a2031f49a2
paper_authors.name: Change Jia
paper_authors._id: 6842537f1c4f28a2031f49a3
paper_authors.name: Linglin Zhang
paper_authors._id: 6842537f1c4f28a2031f49a4
paper_authors.name: Sai-er Hu
paper_authors._id: 6842537f1c4f28a2031f49a5
paper_authors.name: Yuhan Wu
paper_authors._id: 6842537f1c4f28a2031f49a6
paper_authors.name: Xiangzheng Zhang
paper_title: Evaluation is All You Need: Strategic Overclaiming of LLM Reasoning
    Capabilities Through Evaluation Design
paper_summary: Reasoning models represented by the Deepseek-R1-Distill series have been
  widely adopted by the open-source community due to their strong performance in
  mathematics, science, programming, and other domains. However, our study
  reveals that their benchmark evaluation results are subject to significant
  fluctuations caused by various factors. Subtle differences in evaluation
  conditions can lead to substantial variations in results. Similar phenomena are
  observed in other open-source inference models fine-tuned based on the
  Deepseek-R1-Distill series, as well as in the QwQ-32B model, making their
  claimed performance improvements difficult to reproduce reliably. Therefore, we
  advocate for the establishment of a more rigorous paradigm for model
  performance evaluation and present our empirical assessments of the
  Deepseek-R1-Distill series models.
paper_ai_summary: Empirical assessments reveal significant fluctuations in benchmark evaluation results of Deepseek-R1-Distill models, questioning the reliability of claimed performance improvements and advocating for a more rigorous evaluation paradigm.
paper_ai_keywords: reasoning models Deepseek-R1-Distill benchmark evaluation open-source inference models performance variations QwQ-32B model