title: PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoning
summary: Large language models demonstrate remarkable capabilities across various
  domains, especially mathematics and logic reasoning. However, current
  evaluations overlook physics-based reasoning - a complex task requiring physics
  theorems and constraints. We present PhysReason, a 1,200-problem benchmark
  comprising knowledge-based (25%) and reasoning-based (75%) problems, where the
  latter are divided into three difficulty levels (easy, medium, hard). Notably,
  problems require an average of 8.1 solution steps, with hard requiring 15.6,
  reflecting the complexity of physics-based reasoning. We propose the Physics
  Solution Auto Scoring Framework, incorporating efficient answer-level and
  comprehensive step-level evaluations. Top-performing models like Deepseek-R1,
  Gemini-2.0-Flash-Thinking, and o3-mini-high achieve less than 60% on
  answer-level evaluation, with performance dropping from knowledge questions
  (75.11%) to hard problems (31.95%). Through step-level evaluation, we
  identified four key bottlenecks: Physics Theorem Application, Physics Process
  Understanding, Calculation, and Physics Condition Analysis. These findings
  position PhysReason as a novel and comprehensive benchmark for evaluating
  physics-based reasoning capabilities in large language models. Our code and
  data will be published at https:/dxzxy12138.github.io/PhysReason.
paper_authors._id: 67b44a6888813676da9f8239
paper_authors.name: Xinyu Zhang
paper_authors._id: 67b44a6888813676da9f823a
paper_authors.name: Yuxuan Dong
paper_authors._id: 67b44a6888813676da9f823b
paper_authors.name: Yanrui Wu
paper_authors._id: 67b44a6888813676da9f823c
paper_authors.name: Jiaxing Huang
paper_authors._id: 67b44a6888813676da9f823d
paper_authors.name: Chengyou Jia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-18T09:30:47.313Z
paper_authors._id: 6602548a68d519ed324b47c5
paper_authors.avatarUrl: /avatars/5ab411f87440cc2a98c7a1c6a3ed5548.svg
paper_authors.fullname: ChengyouJia
paper_authors.type: user
paper_authors.user: ChengyouJia
paper_authors._id: 67b44a6888813676da9f823e
paper_authors.name: Basura Fernando
paper_authors._id: 67b44a6888813676da9f823f
paper_authors.name: Mike Zheng Shou
paper_authors._id: 67b44a6888813676da9f8240
paper_authors.name: Lingling Zhang
paper_authors._id: 67b44a6888813676da9f8241
paper_authors.name: Jun Liu
paper_title: PhysReason: A Comprehensive Benchmark towards Physics-Based Reasoning
paper_summary: Large language models demonstrate remarkable capabilities across various
  domains, especially mathematics and logic reasoning. However, current
  evaluations overlook physics-based reasoning - a complex task requiring physics
  theorems and constraints. We present PhysReason, a 1,200-problem benchmark
  comprising knowledge-based (25%) and reasoning-based (75%) problems, where the
  latter are divided into three difficulty levels (easy, medium, hard). Notably,
  problems require an average of 8.1 solution steps, with hard requiring 15.6,
  reflecting the complexity of physics-based reasoning. We propose the Physics
  Solution Auto Scoring Framework, incorporating efficient answer-level and
  comprehensive step-level evaluations. Top-performing models like Deepseek-R1,
  Gemini-2.0-Flash-Thinking, and o3-mini-high achieve less than 60% on
  answer-level evaluation, with performance dropping from knowledge questions
  (75.11%) to hard problems (31.95%). Through step-level evaluation, we
  identified four key bottlenecks: Physics Theorem Application, Physics Process
  Understanding, Calculation, and Physics Condition Analysis. These findings
  position PhysReason as a novel and comprehensive benchmark for evaluating
  physics-based reasoning capabilities in large language models. Our code and
  data will be published at https:/dxzxy12138.github.io/PhysReason.
paper_ai_keywords: PhysReason Physics Solution Auto Scoring Framework Deepseek-R1 Gemini-2.0-Flash-Thinking o3-mini-high Physics Theorem Application Physics Process Understanding Physics Condition Analysis