title: PhysBench: Benchmarking and Enhancing Vision-Language Models for
    Physical World Understanding
summary: Understanding the physical world is a fundamental challenge in embodied AI,
  critical for enabling agents to perform complex tasks and operate safely in
  real-world environments. While Vision-Language Models (VLMs) have shown great
  promise in reasoning and task planning for embodied agents, their ability to
  comprehend physical phenomena remains extremely limited. To close this gap, we
  introduce PhysBench, a comprehensive benchmark designed to evaluate VLMs'
  physical world understanding capability across a diverse set of tasks.
  PhysBench contains 10,002 entries of interleaved video-image-text data,
  categorized into four major domains: physical object properties, physical
  object relationships, physical scene understanding, and physics-based dynamics,
  further divided into 19 subclasses and 8 distinct capability dimensions. Our
  extensive experiments, conducted on 75 representative VLMs, reveal that while
  these models excel in common-sense reasoning, they struggle with understanding
  the physical world -- likely due to the absence of physical knowledge in their
  training data and the lack of embedded physical priors. To tackle the
  shortfall, we introduce PhysAgent, a novel framework that combines the
  generalization strengths of VLMs with the specialized expertise of vision
  models, significantly enhancing VLMs' physical understanding across a variety
  of tasks, including an 18.4\% improvement on GPT-4o. Furthermore, our results
  demonstrate that enhancing VLMs' physical world understanding capabilities can
  help embodied agents such as MOKA. We believe that PhysBench and PhysAgent
  offer valuable insights and contribute to bridging the gap between VLMs and
  physical world understanding.
paper_authors._id: 679c4f344061a1ab60ebe6fa
paper_authors.name: Wei Chow
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-01-31T08:35:49.674Z
paper_authors._id: 644b71ddb2e7823a76abcf91
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/644b71ddb2e7823a76abcf91/JPF7Eqeq2jx8i79nQ962K.jpeg
paper_authors.fullname: zhou wei
paper_authors.type: user
paper_authors.user: WeiChow
paper_authors._id: 679c4f344061a1ab60ebe6fb
paper_authors.name: Jiageng Mao
paper_authors._id: 679c4f344061a1ab60ebe6fc
paper_authors.name: Boyi Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2025-01-31T08:46:09.305Z
paper_authors._id: 620dd3888528f797e88cb9b5
paper_authors.avatarUrl: /avatars/af04728788d78fe7d6375e19e32a535e.svg
paper_authors.fullname: Boyi Li
paper_authors.type: user
paper_authors.user: Boyiliee
paper_authors._id: 679c4f344061a1ab60ebe6fd
paper_authors.name: Daniel Seita
paper_authors._id: 679c4f344061a1ab60ebe6fe
paper_authors.name: Vitor Guizilini
paper_authors._id: 679c4f344061a1ab60ebe6ff
paper_authors.name: Yue Wang
paper_title: PhysBench: Benchmarking and Enhancing Vision-Language Models for
    Physical World Understanding
paper_summary: Understanding the physical world is a fundamental challenge in embodied AI,
  critical for enabling agents to perform complex tasks and operate safely in
  real-world environments. While Vision-Language Models (VLMs) have shown great
  promise in reasoning and task planning for embodied agents, their ability to
  comprehend physical phenomena remains extremely limited. To close this gap, we
  introduce PhysBench, a comprehensive benchmark designed to evaluate VLMs'
  physical world understanding capability across a diverse set of tasks.
  PhysBench contains 10,002 entries of interleaved video-image-text data,
  categorized into four major domains: physical object properties, physical
  object relationships, physical scene understanding, and physics-based dynamics,
  further divided into 19 subclasses and 8 distinct capability dimensions. Our
  extensive experiments, conducted on 75 representative VLMs, reveal that while
  these models excel in common-sense reasoning, they struggle with understanding
  the physical world -- likely due to the absence of physical knowledge in their
  training data and the lack of embedded physical priors. To tackle the
  shortfall, we introduce PhysAgent, a novel framework that combines the
  generalization strengths of VLMs with the specialized expertise of vision
  models, significantly enhancing VLMs' physical understanding across a variety
  of tasks, including an 18.4\% improvement on GPT-4o. Furthermore, our results
  demonstrate that enhancing VLMs' physical world understanding capabilities can
  help embodied agents such as MOKA. We believe that PhysBench and PhysAgent
  offer valuable insights and contribute to bridging the gap between VLMs and
  physical world understanding.
paper_ai_keywords: Vision-Language Models (VLMs) benchmark video-image-text data physical object properties physical object relationships physical scene understanding physics-based dynamics capability dimensions generalization strengths vision models PhysBench PhysAgent GPT-4o MOKA