title: VoxPoser: Composable 3D Value Maps for Robotic Manipulation with
    Language Models
summary: Large language models (LLMs) are shown to possess a wealth of actionable
  knowledge that can be extracted for robot manipulation in the form of reasoning
  and planning. Despite the progress, most still rely on pre-defined motion
  primitives to carry out the physical interactions with the environment, which
  remains a major bottleneck. In this work, we aim to synthesize robot
  trajectories, i.e., a dense sequence of 6-DoF end-effector waypoints, for a
  large variety of manipulation tasks given an open-set of instructions and an
  open-set of objects. We achieve this by first observing that LLMs excel at
  inferring affordances and constraints given a free-form language instruction.
  More importantly, by leveraging their code-writing capabilities, they can
  interact with a visual-language model (VLM) to compose 3D value maps to ground
  the knowledge into the observation space of the agent. The composed value maps
  are then used in a model-based planning framework to zero-shot synthesize
  closed-loop robot trajectories with robustness to dynamic perturbations. We
  further demonstrate how the proposed framework can benefit from online
  experiences by efficiently learning a dynamics model for scenes that involve
  contact-rich interactions. We present a large-scale study of the proposed
  method in both simulated and real-robot environments, showcasing the ability to
  perform a large variety of everyday manipulation tasks specified in free-form
  natural language. Project website: https://voxposer.github.io
paper_authors._id: 64af5fcd8965d8311042cc7a
paper_authors.name: Wenlong Huang
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-08-26T22:36:14.847Z
paper_authors._id: 66cd0333d94b6885fd163ff7
paper_authors.avatarUrl: /avatars/9117dd683fa19c1cb11f97355babcae2.svg
paper_authors.fullname: Wenlong Huang
paper_authors.type: user
paper_authors.user: wenlongh
paper_authors._id: 64af5fcd8965d8311042cc7b
paper_authors.name: Chen Wang
paper_authors._id: 64af5fcd8965d8311042cc7c
paper_authors.name: Ruohan Zhang
paper_authors._id: 64af5fcd8965d8311042cc7d
paper_authors.name: Yunzhu Li
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-13T09:54:51.413Z
paper_authors._id: 64a46562c641afb468a3dc04
paper_authors.avatarUrl: /avatars/bc54e5c13f657251b05bf2725086e56d.svg
paper_authors.fullname: Yunzhu Li
paper_authors.type: user
paper_authors.user: yunzhuli
paper_authors._id: 64af5fcd8965d8311042cc7e
paper_authors.name: Jiajun Wu
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-07-13T09:55:10.220Z
paper_authors._id: 647b7b8345616c3de3627863
paper_authors.avatarUrl: /avatars/8196364bffed71086ba9a0a4a62027d3.svg
paper_authors.fullname: Jiajun Wu
paper_authors.type: user
paper_authors.user: jiajunwu
paper_authors._id: 64af5fcd8965d8311042cc7f
paper_authors.name: Li Fei-Fei
paper_title: VoxPoser: Composable 3D Value Maps for Robotic Manipulation with
    Language Models
paper_summary: Large language models (LLMs) are shown to possess a wealth of actionable
  knowledge that can be extracted for robot manipulation in the form of reasoning
  and planning. Despite the progress, most still rely on pre-defined motion
  primitives to carry out the physical interactions with the environment, which
  remains a major bottleneck. In this work, we aim to synthesize robot
  trajectories, i.e., a dense sequence of 6-DoF end-effector waypoints, for a
  large variety of manipulation tasks given an open-set of instructions and an
  open-set of objects. We achieve this by first observing that LLMs excel at
  inferring affordances and constraints given a free-form language instruction.
  More importantly, by leveraging their code-writing capabilities, they can
  interact with a visual-language model (VLM) to compose 3D value maps to ground
  the knowledge into the observation space of the agent. The composed value maps
  are then used in a model-based planning framework to zero-shot synthesize
  closed-loop robot trajectories with robustness to dynamic perturbations. We
  further demonstrate how the proposed framework can benefit from online
  experiences by efficiently learning a dynamics model for scenes that involve
  contact-rich interactions. We present a large-scale study of the proposed
  method in both simulated and real-robot environments, showcasing the ability to
  perform a large variety of everyday manipulation tasks specified in free-form
  natural language. Project website: https://voxposer.github.io
paper_ai_keywords: large language models (LLMs) robot manipulation reasoning planning motion primitives robot trajectories 6-DoF end-effector waypoints manipulation tasks free-form language instruction affordances constraints code-writing capabilities visual-language model (VLM) 3D value maps observation space model-based planning framework closed-loop robot trajectories robustness to dynamic perturbations dynamics model contact-rich interactions free-form natural language