title: GPT-4V(ision) for Robotics: Multimodal Task Planning from Human
    Demonstration
summary: We introduce a pipeline that enhances a general-purpose Vision Language
  Model, GPT-4V(ision), by integrating observations of human actions to
  facilitate robotic manipulation. This system analyzes videos of humans
  performing tasks and creates executable robot programs that incorporate
  affordance insights. The computation starts by analyzing the videos with GPT-4V
  to convert environmental and action details into text, followed by a
  GPT-4-empowered task planner. In the following analyses, vision systems
  reanalyze the video with the task plan. Object names are grounded using an
  open-vocabulary object detector, while focus on the hand-object relation helps
  to detect the moment of grasping and releasing. This spatiotemporal grounding
  allows the vision systems to further gather affordance data (e.g., grasp type,
  way points, and body postures). Experiments across various scenarios
  demonstrate this method's efficacy in achieving real robots' operations from
  human demonstrations in a zero-shot manner. The prompts of GPT-4V/GPT-4 are
  available at this project page:
  https://microsoft.github.io/GPT4Vision-Robot-Manipulation-Prompts/
paper_authors._id: 655c16ba71ed242b42677f8d
paper_authors.name: Naoki Wake
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2023-11-21T02:33:31.974Z
paper_authors._id: 650bd16424da30e773ec74ca
paper_authors.avatarUrl: /avatars/1ff8a1c9d4c6bde1893e4f80c5fd9fdb.svg
paper_authors.fullname: Wake
paper_authors.type: user
paper_authors.user: NaokiWake
paper_authors._id: 655c16ba71ed242b42677f8e
paper_authors.name: Atsushi Kanehira
paper_authors._id: 655c16ba71ed242b42677f8f
paper_authors.name: Kazuhiro Sasabuchi
paper_authors._id: 655c16ba71ed242b42677f90
paper_authors.name: Jun Takamatsu
paper_authors._id: 655c16ba71ed242b42677f91
paper_authors.name: Katsushi Ikeuchi
paper_title: GPT-4V(ision) for Robotics: Multimodal Task Planning from Human
    Demonstration
paper_summary: We introduce a pipeline that enhances a general-purpose Vision Language
  Model, GPT-4V(ision), by integrating observations of human actions to
  facilitate robotic manipulation. This system analyzes videos of humans
  performing tasks and creates executable robot programs that incorporate
  affordance insights. The computation starts by analyzing the videos with GPT-4V
  to convert environmental and action details into text, followed by a
  GPT-4-empowered task planner. In the following analyses, vision systems
  reanalyze the video with the task plan. Object names are grounded using an
  open-vocabulary object detector, while focus on the hand-object relation helps
  to detect the moment of grasping and releasing. This spatiotemporal grounding
  allows the vision systems to further gather affordance data (e.g., grasp type,
  way points, and body postures). Experiments across various scenarios
  demonstrate this method's efficacy in achieving real robots' operations from
  human demonstrations in a zero-shot manner. The prompts of GPT-4V/GPT-4 are
  available at this project page:
  https://microsoft.github.io/GPT4Vision-Robot-Manipulation-Prompts/
paper_ai_keywords: Vision Language Model GPT-4V(ision) robotic manipulation task planner open-vocabulary object detector spatiotemporal grounding affordance data zero-shot manner