title: PhyX: Does Your Model Have the "Wits" for Physical Reasoning?
summary: Existing benchmarks fail to capture a crucial aspect of intelligence:
  physical reasoning, the integrated ability to combine domain knowledge,
  symbolic reasoning, and understanding of real-world constraints. To address
  this gap, we introduce PhyX: the first large-scale benchmark designed to assess
  models capacity for physics-grounded reasoning in visual scenarios. PhyX
  includes 3K meticulously curated multimodal questions spanning 6 reasoning
  types across 25 sub-domains and 6 core physics domains: thermodynamics,
  electromagnetism, mechanics, modern physics, optics, and wave\&acoustics. In
  our comprehensive evaluation, even state-of-the-art models struggle
  significantly with physical reasoning. GPT-4o, Claude3.7-Sonnet, and
  GPT-o4-mini achieve only 32.5\%, 42.2\%, and 45.8\% accuracy
  respectively-performance gaps exceeding 29\% compared to human experts. Our
  analysis exposes critical limitations in current models: over-reliance on
  memorized disciplinary knowledge, excessive dependence on mathematical
  formulations, and surface-level visual pattern matching rather than genuine
  physical understanding. We provide in-depth analysis through fine-grained
  statistics, detailed case studies, and multiple evaluation paradigms to
  thoroughly examine physical reasoning capabilities. To ensure reproducibility,
  we implement a compatible evaluation protocol based on widely-used toolkits
  such as VLMEvalKit, enabling one-click evaluation.
paper_authors._id: 6830404effb59afb6569273a
paper_authors.name: Hui Shen
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T07:55:15.229Z
paper_authors._id: 640feb91b0ee289c8583dd59
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/640feb91b0ee289c8583dd59/CA4UZj2mEsJO2UR0NnN5H.jpeg
paper_authors.fullname: Hui Shen
paper_authors.type: user
paper_authors.user: Cloudriver
paper_authors._id: 6830404effb59afb6569273b
paper_authors.name: Taiqiang Wu
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:14:30.851Z
paper_authors._id: 6621cea88850e38ffbb1854f
paper_authors.avatarUrl: /avatars/6d73d947046faa32260ee325069976d9.svg
paper_authors.fullname: Taki WU
paper_authors.type: user
paper_authors.user: taki555
paper_authors._id: 6830404effb59afb6569273c
paper_authors.name: Qi Han
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-27T07:55:12.818Z
paper_authors._id: 6766d2764d57021c96679b7f
paper_authors.avatarUrl: /avatars/0810011e1e0636cb80eb7c092ea13b3f.svg
paper_authors.fullname: han q
paper_authors.type: user
paper_authors.user: nrzyh
paper_authors._id: 6830404effb59afb6569273d
paper_authors.name: Yunta Hsieh
paper_authors._id: 6830404effb59afb6569273e
paper_authors.name: Jizhou Wang
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-26T08:14:33.171Z
paper_authors._id: 67fe265f9698ae4f5f4db718
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/0PLJEbUyJbM9BacFxcScP.png
paper_authors.fullname: Jizhou Wang
paper_authors.type: user
paper_authors.user: John-ai-bee
paper_authors._id: 6830404effb59afb6569273f
paper_authors.name: Yuyue Zhang
paper_authors._id: 6830404effb59afb65692740
paper_authors.name: Yuxin Cheng
paper_authors._id: 6830404effb59afb65692741
paper_authors.name: Zijian Hao
paper_authors._id: 6830404effb59afb65692742
paper_authors.name: Yuansheng Ni
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-05-28T09:01:10.443Z
paper_authors._id: 64de37ee5e192985054be575
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/64de37ee5e192985054be575/fVV7JQMtp_J3uFqszJJHH.jpeg
paper_authors.fullname: Yuansheng Ni
paper_authors.type: user
paper_authors.user: yuanshengni
paper_authors._id: 6830404effb59afb65692743
paper_authors.name: Xin Wang
paper_authors._id: 6830404effb59afb65692744
paper_authors.name: Zhongwei Wan
paper_authors._id: 6830404effb59afb65692745
paper_authors.name: Kai Zhang
paper_authors._id: 6830404effb59afb65692746
paper_authors.name: Wendong Xu
paper_authors._id: 6830404effb59afb65692747
paper_authors.name: Jing Xiong
paper_authors._id: 6830404effb59afb65692748
paper_authors.name: Ping Luo
paper_authors._id: 6830404effb59afb65692749
paper_authors.name: Wenhu Chen
paper_authors._id: 6830404effb59afb6569274a
paper_authors.name: Chaofan Tao
paper_authors._id: 6830404effb59afb6569274b
paper_authors.name: Zhuoqing Mao
paper_authors._id: 6830404effb59afb6569274c
paper_authors.name: Ngai Wong
paper_title: PhyX: Does Your Model Have the "Wits" for Physical Reasoning?
paper_summary: Existing benchmarks fail to capture a crucial aspect of intelligence:
  physical reasoning, the integrated ability to combine domain knowledge,
  symbolic reasoning, and understanding of real-world constraints. To address
  this gap, we introduce PhyX: the first large-scale benchmark designed to assess
  models capacity for physics-grounded reasoning in visual scenarios. PhyX
  includes 3K meticulously curated multimodal questions spanning 6 reasoning
  types across 25 sub-domains and 6 core physics domains: thermodynamics,
  electromagnetism, mechanics, modern physics, optics, and wave\&acoustics. In
  our comprehensive evaluation, even state-of-the-art models struggle
  significantly with physical reasoning. GPT-4o, Claude3.7-Sonnet, and
  GPT-o4-mini achieve only 32.5\%, 42.2\%, and 45.8\% accuracy
  respectively-performance gaps exceeding 29\% compared to human experts. Our
  analysis exposes critical limitations in current models: over-reliance on
  memorized disciplinary knowledge, excessive dependence on mathematical
  formulations, and surface-level visual pattern matching rather than genuine
  physical understanding. We provide in-depth analysis through fine-grained
  statistics, detailed case studies, and multiple evaluation paradigms to
  thoroughly examine physical reasoning capabilities. To ensure reproducibility,
  we implement a compatible evaluation protocol based on widely-used toolkits
  such as VLMEvalKit, enabling one-click evaluation.
paper_ai_summary: A new benchmark, PhyX, evaluates models' physics-grounded reasoning in visual scenarios, revealing significant limitations in current models' physical understanding compared to human experts.
paper_ai_keywords: multimodal questions reasoning types sub-domains core physics domains thermodynamics electromagnetism mechanics modern physics optics wave\&acoustics fine-grained statistics case studies evaluation paradigms VLMEvalKit