title: PHYSICS: Benchmarking Foundation Models on University-Level Physics
    Problem Solving
summary: We introduce PHYSICS, a comprehensive benchmark for university-level physics
  problem solving. It contains 1297 expert-annotated problems covering six core
  areas: classical mechanics, quantum mechanics, thermodynamics and statistical
  mechanics, electromagnetism, atomic physics, and optics. Each problem requires
  advanced physics knowledge and mathematical reasoning. We develop a robust
  automated evaluation system for precise and reliable validation. Our evaluation
  of leading foundation models reveals substantial limitations. Even the most
  advanced model, o3-mini, achieves only 59.9% accuracy, highlighting significant
  challenges in solving high-level scientific problems. Through comprehensive
  error analysis, exploration of diverse prompting strategies, and
  Retrieval-Augmented Generation (RAG)-based knowledge augmentation, we identify
  key areas for improvement, laying the foundation for future advancements.
paper_authors._id: 67e9ffab6887b70da56d0de5
paper_authors.name: Kaiyue Feng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-31T08:30:14.677Z
paper_authors._id: 668b6668cc2c0b4ae303bdb8
paper_authors.avatarUrl: /avatars/2a4e30c0a5ee76b66232f425d5e62747.svg
paper_authors.fullname: Kaiyue Feng
paper_authors.type: user
paper_authors.user: Carrie777
paper_authors._id: 67e9ffab6887b70da56d0de6
paper_authors.name: Yilun Zhao
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-03-31T08:11:42.629Z
paper_authors._id: 62f662bcc58915315c4eccea
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/62f662bcc58915315c4eccea/zOAQLONfMP88zr70sxHK-.jpeg
paper_authors.fullname: Yilun Zhao
paper_authors.type: user
paper_authors.user: yilunzhao
paper_authors._id: 67e9ffab6887b70da56d0de7
paper_authors.name: Yixin Liu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-31T08:30:33.598Z
paper_authors._id: 6244de1c1c560fb11edfca44
paper_authors.avatarUrl: /avatars/36558928bd04be7f49837d4c603681d7.svg
paper_authors.fullname: Yixin Liu
paper_authors.type: user
paper_authors.user: henryL7
paper_authors._id: 67e9ffab6887b70da56d0de8
paper_authors.name: Tianyu Yang
paper_authors._id: 67e9ffab6887b70da56d0de9
paper_authors.name: Chen Zhao
paper_authors._id: 67e9ffab6887b70da56d0dea
paper_authors.name: John Sous
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-31T08:30:58.653Z
paper_authors._id: 6626e136cee7ea5738a8442b
paper_authors.avatarUrl: /avatars/8c1537773e2c70f9c10b51a004380824.svg
paper_authors.fullname: John Sous
paper_authors.type: user
paper_authors.user: jsous
paper_authors._id: 67e9ffab6887b70da56d0deb
paper_authors.name: Arman Cohan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-03-31T08:31:04.921Z
paper_authors._id: 5f5ba21188f57f65f951f255
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1599840760465-noauth.png
paper_authors.fullname: Arman Cohan
paper_authors.type: user
paper_authors.user: armanc
paper_title: PHYSICS: Benchmarking Foundation Models on University-Level Physics
    Problem Solving
paper_summary: We introduce PHYSICS, a comprehensive benchmark for university-level physics
  problem solving. It contains 1297 expert-annotated problems covering six core
  areas: classical mechanics, quantum mechanics, thermodynamics and statistical
  mechanics, electromagnetism, atomic physics, and optics. Each problem requires
  advanced physics knowledge and mathematical reasoning. We develop a robust
  automated evaluation system for precise and reliable validation. Our evaluation
  of leading foundation models reveals substantial limitations. Even the most
  advanced model, o3-mini, achieves only 59.9% accuracy, highlighting significant
  challenges in solving high-level scientific problems. Through comprehensive
  error analysis, exploration of diverse prompting strategies, and
  Retrieval-Augmented Generation (RAG)-based knowledge augmentation, we identify
  key areas for improvement, laying the foundation for future advancements.
paper_ai_summary: A comprehensive physics problem-solving benchmark reveals limitations in existing foundation models, guiding future improvements through error analysis and knowledge augmentation techniques.
paper_ai_keywords: Retrieval-Augmented Generation (RAG)