title: Physically Grounded Vision-Language Models for Robotic Manipulation
summary: Recent advances in vision-language models (VLMs) have led to improved
  performance on tasks such as visual question answering and image captioning.
  Consequently, these models are now well-positioned to reason about the physical
  world, particularly within domains such as robotic manipulation. However,
  current VLMs are limited in their understanding of the physical concepts (e.g.,
  material, fragility) of common objects, which restricts their usefulness for
  robotic manipulation tasks that involve interaction and physical reasoning
  about such objects. To address this limitation, we propose PhysObjects, an
  object-centric dataset of 36.9K crowd-sourced and 417K automated physical
  concept annotations of common household objects. We demonstrate that
  fine-tuning a VLM on PhysObjects improves its understanding of physical object
  concepts, by capturing human priors of these concepts from visual appearance.
  We incorporate this physically-grounded VLM in an interactive framework with a
  large language model-based robotic planner, and show improved planning
  performance on tasks that require reasoning about physical object concepts,
  compared to baselines that do not leverage physically-grounded VLMs. We
  additionally illustrate the benefits of our physically-grounded VLM on a real
  robot, where it improves task success rates. We release our dataset and provide
  further details and visualizations of our results at
  https://iliad.stanford.edu/pg-vlm/.
paper_authors._id: 64f92c90fe57e7455ac3ed2d
paper_authors.name: Jensen Gao
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-07T13:30:30.738Z
paper_authors._id: 64adc155cf90fe2755648257
paper_authors.avatarUrl: /avatars/a449c4f5bc051b1e4bd0441667f4c2f8.svg
paper_authors.fullname: Jensen Gao
paper_authors.type: user
paper_authors.user: jenseng
paper_authors._id: 64f92c90fe57e7455ac3ed2e
paper_authors.name: Bidipta Sarkar
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-07T13:30:37.971Z
paper_authors._id: 63abbf74ad514ca8d14a0548
paper_authors.avatarUrl: /avatars/b1357b73b8f9a8ff9908710ad64154ef.svg
paper_authors.fullname: Bidipta Sarkar
paper_authors.type: user
paper_authors.user: bidiptas
paper_authors._id: 64f92c90fe57e7455ac3ed2f
paper_authors.name: Fei Xia
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-01-23T08:10:34.384Z
paper_authors._id: 62cfa9ee9b96f22525c36d47
paper_authors.avatarUrl: /avatars/8bfe3bf63f0609b3b6b7d7c5d7f6b3e9.svg
paper_authors.fullname: Fei Xia
paper_authors.type: user
paper_authors.user: fxia22
paper_authors._id: 64f92c90fe57e7455ac3ed30
paper_authors.name: Ted Xiao
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2025-03-06T07:26:16.159Z
paper_authors._id: 64f1835d8eaa168d21f7bac4
paper_authors.avatarUrl: /avatars/59754ba46756ecdab8d7dff740af90a6.svg
paper_authors.fullname: Ted Xiao
paper_authors.type: user
paper_authors.user: xiaoted
paper_authors._id: 64f92c90fe57e7455ac3ed31
paper_authors.name: Jiajun Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-07T13:32:03.787Z
paper_authors._id: 647b7b8345616c3de3627863
paper_authors.avatarUrl: /avatars/8196364bffed71086ba9a0a4a62027d3.svg
paper_authors.fullname: Jiajun Wu
paper_authors.type: user
paper_authors.user: jiajunwu
paper_authors._id: 64f92c90fe57e7455ac3ed32
paper_authors.name: Brian Ichter
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-09-07T13:32:11.167Z
paper_authors._id: 633a26ab474cfeb1a864dc56
paper_authors.avatarUrl: /avatars/cd40accd894fc78810fc0d5108f413e9.svg
paper_authors.fullname: Brian I
paper_authors.type: user
paper_authors.user: brianichter
paper_authors._id: 64f92c90fe57e7455ac3ed33
paper_authors.name: Anirudha Majumdar
paper_authors._id: 64f92c90fe57e7455ac3ed34
paper_authors.name: Dorsa Sadigh
paper_title: Physically Grounded Vision-Language Models for Robotic Manipulation
paper_summary: Recent advances in vision-language models (VLMs) have led to improved
  performance on tasks such as visual question answering and image captioning.
  Consequently, these models are now well-positioned to reason about the physical
  world, particularly within domains such as robotic manipulation. However,
  current VLMs are limited in their understanding of the physical concepts (e.g.,
  material, fragility) of common objects, which restricts their usefulness for
  robotic manipulation tasks that involve interaction and physical reasoning
  about such objects. To address this limitation, we propose PhysObjects, an
  object-centric dataset of 36.9K crowd-sourced and 417K automated physical
  concept annotations of common household objects. We demonstrate that
  fine-tuning a VLM on PhysObjects improves its understanding of physical object
  concepts, by capturing human priors of these concepts from visual appearance.
  We incorporate this physically-grounded VLM in an interactive framework with a
  large language model-based robotic planner, and show improved planning
  performance on tasks that require reasoning about physical object concepts,
  compared to baselines that do not leverage physically-grounded VLMs. We
  additionally illustrate the benefits of our physically-grounded VLM on a real
  robot, where it improves task success rates. We release our dataset and provide
  further details and visualizations of our results at
  https://iliad.stanford.edu/pg-vlm/.
paper_ai_keywords: vision-language models (VLMs) visual question answering image captioning robotic manipulation physical concepts material PhysObjects object-centric dataset crowd-sourced annotations automated physical concept annotations fine-tuning human priors visual appearance physically-grounded VLM large language model-based robotic manipulation interactive framework planning performance task success rates