title: PHYBench: Holistic Evaluation of Physical Perception and Reasoning in
    Large Language Models
summary: We introduce PHYBench, a novel, high-quality benchmark designed for
  evaluating reasoning capabilities of large language models (LLMs) in physical
  contexts. PHYBench consists of 500 meticulously curated physics problems based
  on real-world physical scenarios, designed to assess the ability of models to
  understand and reason about realistic physical processes. Covering mechanics,
  electromagnetism, thermodynamics, optics, modern physics, and advanced physics,
  the benchmark spans difficulty levels from high school exercises to
  undergraduate problems and Physics Olympiad challenges. Additionally, we
  propose the Expression Edit Distance (EED) Score, a novel evaluation metric
  based on the edit distance between mathematical expressions, which effectively
  captures differences in model reasoning processes and results beyond
  traditional binary scoring methods. We evaluate various LLMs on PHYBench and
  compare their performance with human experts. Our results reveal that even
  state-of-the-art reasoning models significantly lag behind human experts,
  highlighting their limitations and the need for improvement in complex physical
  reasoning scenarios. Our benchmark results and dataset are publicly available
  at https://phybench-official.github.io/phybench-demo/.
paper_authors._id: 680a12f28ea8fd5e81332f3e
paper_authors.name: Shi Qiu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-25T08:35:06.205Z
paper_authors._id: 6644bb2c9bdbd85493074411
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6644bb2c9bdbd85493074411/iug8dkP1zjID-kXzNxDjD.jpeg
paper_authors.fullname: SHI QIU
paper_authors.type: user
paper_authors.user: StarThomas1002
paper_authors._id: 680a12f28ea8fd5e81332f3f
paper_authors.name: Shaoyang Guo
paper_authors._id: 680a12f28ea8fd5e81332f40
paper_authors.name: Zhuo-Yang Song
paper_authors._id: 680a12f28ea8fd5e81332f41
paper_authors.name: Yunbo Sun
paper_authors._id: 680a12f28ea8fd5e81332f42
paper_authors.name: Zeyu Cai
paper_authors._id: 680a12f28ea8fd5e81332f43
paper_authors.name: Jiashen Wei
paper_authors._id: 680a12f28ea8fd5e81332f44
paper_authors.name: Tianyu Luo
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-25T08:35:11.506Z
paper_authors._id: 68079e7d34496fa3eec1adeb
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/68079e7d34496fa3eec1adeb/qSpUFt6nwb8cu3GX9cd5y.jpeg
paper_authors.fullname: Tianyu Luo
paper_authors.type: user
paper_authors.user: CaCtusLuo
paper_authors._id: 680a12f28ea8fd5e81332f45
paper_authors.name: Yixuan Yin
paper_authors._id: 680a12f28ea8fd5e81332f46
paper_authors.name: Haoxu Zhang
paper_authors._id: 680a12f28ea8fd5e81332f47
paper_authors.name: Yi Hu
paper_authors._id: 680a12f28ea8fd5e81332f48
paper_authors.name: Chenyang Wang
paper_authors._id: 680a12f28ea8fd5e81332f49
paper_authors.name: Chencheng Tang
paper_authors._id: 680a12f28ea8fd5e81332f4a
paper_authors.name: Haoling Chang
paper_authors._id: 680a12f28ea8fd5e81332f4b
paper_authors.name: Qi Liu
paper_authors._id: 680a12f28ea8fd5e81332f4c
paper_authors.name: Ziheng Zhou
paper_authors._id: 680a12f28ea8fd5e81332f4d
paper_authors.name: Tianyu Zhang
paper_authors._id: 680a12f28ea8fd5e81332f4e
paper_authors.name: Jingtian Zhang
paper_authors._id: 680a12f28ea8fd5e81332f4f
paper_authors.name: Zhangyi Liu
paper_authors._id: 680a12f28ea8fd5e81332f50
paper_authors.name: Minghao Li
paper_authors._id: 680a12f28ea8fd5e81332f51
paper_authors.name: Yuku Zhang
paper_authors._id: 680a12f28ea8fd5e81332f52
paper_authors.name: Boxuan Jing
paper_authors._id: 680a12f28ea8fd5e81332f53
paper_authors.name: Xianqi Yin
paper_authors._id: 680a12f28ea8fd5e81332f54
paper_authors.name: Yutong Ren
paper_authors._id: 680a12f28ea8fd5e81332f55
paper_authors.name: Zizhuo Fu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-04-25T08:35:09.065Z
paper_authors._id: 67c7b67cbc99f666def4e1de
paper_authors.avatarUrl: /avatars/818967a977070b32a4f2037bd377a92d.svg
paper_authors.fullname: zZzZzZ
paper_authors.type: user
paper_authors.user: zZzZzZ9999999
paper_authors._id: 680a12f28ea8fd5e81332f56
paper_authors.name: Weike Wang
paper_authors._id: 680a12f28ea8fd5e81332f57
paper_authors.name: Xudong Tian
paper_authors._id: 680a12f28ea8fd5e81332f58
paper_authors.name: Anqi Lv
paper_authors._id: 680a12f28ea8fd5e81332f59
paper_authors.name: Laifu Man
paper_authors._id: 680a12f28ea8fd5e81332f5a
paper_authors.name: Jianxiang Li
paper_authors._id: 680a12f28ea8fd5e81332f5b
paper_authors.name: Feiyu Tao
paper_authors._id: 680a12f28ea8fd5e81332f5c
paper_authors.name: Qihua Sun
paper_authors._id: 680a12f28ea8fd5e81332f5d
paper_authors.name: Zhou Liang
paper_authors._id: 680a12f28ea8fd5e81332f5e
paper_authors.name: Yushu Mu
paper_authors._id: 680a12f28ea8fd5e81332f5f
paper_authors.name: Zhongxuan Li
paper_authors._id: 680a12f28ea8fd5e81332f60
paper_authors.name: Jing-Jun Zhang
paper_authors._id: 680a12f28ea8fd5e81332f61
paper_authors.name: Shutao Zhang
paper_authors._id: 680a12f28ea8fd5e81332f62
paper_authors.name: Xiaotian Li
paper_authors._id: 680a12f28ea8fd5e81332f63
paper_authors.name: Xingqi Xia
paper_authors._id: 680a12f28ea8fd5e81332f64
paper_authors.name: Jiawei Lin
paper_authors._id: 680a12f28ea8fd5e81332f65
paper_authors.name: Zheyu Shen
paper_authors._id: 680a12f28ea8fd5e81332f66
paper_authors.name: Jiahang Chen
paper_authors._id: 680a12f28ea8fd5e81332f67
paper_authors.name: Qiuhao Xiong
paper_authors._id: 680a12f28ea8fd5e81332f68
paper_authors.name: Binran Wang
paper_authors._id: 680a12f28ea8fd5e81332f69
paper_authors.name: Fengyuan Wang
paper_authors._id: 680a12f28ea8fd5e81332f6a
paper_authors.name: Ziyang Ni
paper_authors._id: 680a12f28ea8fd5e81332f6b
paper_authors.name: Bohan Zhang
paper_authors._id: 680a12f28ea8fd5e81332f6c
paper_authors.name: Fan Cui
paper_authors._id: 680a12f28ea8fd5e81332f6d
paper_authors.name: Changkun Shao
paper_authors._id: 680a12f28ea8fd5e81332f6e
paper_authors.name: Qing-Hong Cao
paper_authors._id: 680a12f28ea8fd5e81332f6f
paper_authors.name: Ming-xing Luo
paper_authors._id: 680a12f28ea8fd5e81332f70
paper_authors.name: Muhan Zhang
paper_authors._id: 680a12f28ea8fd5e81332f71
paper_authors.name: Hua Xing Zhu
paper_title: PHYBench: Holistic Evaluation of Physical Perception and Reasoning in
    Large Language Models
paper_summary: We introduce PHYBench, a novel, high-quality benchmark designed for
  evaluating reasoning capabilities of large language models (LLMs) in physical
  contexts. PHYBench consists of 500 meticulously curated physics problems based
  on real-world physical scenarios, designed to assess the ability of models to
  understand and reason about realistic physical processes. Covering mechanics,
  electromagnetism, thermodynamics, optics, modern physics, and advanced physics,
  the benchmark spans difficulty levels from high school exercises to
  undergraduate problems and Physics Olympiad challenges. Additionally, we
  propose the Expression Edit Distance (EED) Score, a novel evaluation metric
  based on the edit distance between mathematical expressions, which effectively
  captures differences in model reasoning processes and results beyond
  traditional binary scoring methods. We evaluate various LLMs on PHYBench and
  compare their performance with human experts. Our results reveal that even
  state-of-the-art reasoning models significantly lag behind human experts,
  highlighting their limitations and the need for improvement in complex physical
  reasoning scenarios. Our benchmark results and dataset are publicly available
  at https://phybench-official.github.io/phybench-demo/.
paper_ai_summary: A new benchmark evaluates large language models on physics problems with a novel Expression Edit Distance metric, revealing gaps compared to human experts.
paper_ai_keywords: PHYBench large language models LLMs physics problems mechanics electromagnetism thermodynamics optics modern physics advanced physics Expression Edit Distance EED Score