title: PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs
summary: Vision language models (VLMs) have shown impressive capabilities across a
  variety of tasks, from logical reasoning to visual understanding. This opens
  the door to richer interaction with the world, for example robotic control.
  However, VLMs produce only textual outputs, while robotic control and other
  spatial tasks require outputting continuous coordinates, actions, or
  trajectories. How can we enable VLMs to handle such settings without
  fine-tuning on task-specific data?
    In this paper, we propose a novel visual prompting approach for VLMs that we
  call Prompting with Iterative Visual Optimization (PIVOT), which casts tasks as
  iterative visual question answering. In each iteration, the image is annotated
  with a visual representation of proposals that the VLM can refer to (e.g.,
  candidate robot actions, localizations, or trajectories). The VLM then selects
  the best ones for the task. These proposals are iteratively refined, allowing
  the VLM to eventually zero in on the best available answer. We investigate
  PIVOT on real-world robotic navigation, real-world manipulation from images,
  instruction following in simulation, and additional spatial inference tasks
  such as localization. We find, perhaps surprisingly, that our approach enables
  zero-shot control of robotic systems without any robot training data,
  navigation in a variety of environments, and other capabilities. Although
  current performance is far from perfect, our work highlights potentials and
  limitations of this new regime and shows a promising approach for
  Internet-Scale VLMs in robotic and spatial reasoning domains. Website:
  pivot-prompt.github.io and HuggingFace:
  https://huggingface.co/spaces/pivot-prompt/pivot-prompt-demo.
paper_authors._id: 65cae6465495933ab081f5a5
paper_authors.name: Soroush Nasiriany
paper_authors._id: 65cae6465495933ab081f5a6
paper_authors.name: Fei Xia
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T11:41:12.984Z
paper_authors._id: 62cfa9ee9b96f22525c36d47
paper_authors.avatarUrl: /avatars/8bfe3bf63f0609b3b6b7d7c5d7f6b3e9.svg
paper_authors.fullname: Fei Xia
paper_authors.type: user
paper_authors.user: fxia22
paper_authors._id: 65cae6465495933ab081f5a7
paper_authors.name: Wenhao Yu
paper_authors._id: 65cae6465495933ab081f5a8
paper_authors.name: Ted Xiao
paper_authors._id: 65cae6465495933ab081f5a9
paper_authors.name: Jacky Liang
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T11:46:21.547Z
paper_authors._id: 63017a70792a47211c3714c1
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/1667856283106-63017a70792a47211c3714c1.jpeg
paper_authors.fullname: Jacky Liang
paper_authors.type: user
paper_authors.user: jackyliang42
paper_authors._id: 65cae6465495933ab081f5aa
paper_authors.name: Ishita Dasgupta
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T11:46:32.643Z
paper_authors._id: 64f7f2871435cc7faa39104d
paper_authors.avatarUrl: /avatars/8d42adabf6610eae5169be3ac42165e3.svg
paper_authors.fullname: Ishita Dasgupta
paper_authors.type: user
paper_authors.user: ishitadg
paper_authors._id: 65cae6465495933ab081f5ab
paper_authors.name: Annie Xie
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T11:46:43.919Z
paper_authors._id: 649e23bb3e5e75047640b3b5
paper_authors.avatarUrl: /avatars/019d008a5bd7057b8665e1ef856f13a7.svg
paper_authors.fullname: Annie Xie
paper_authors.type: user
paper_authors.user: anniexie
paper_authors._id: 65cae6465495933ab081f5ac
paper_authors.name: Danny Driess
paper_authors._id: 65cae6465495933ab081f5ad
paper_authors.name: Ayzaan Wahid
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T11:46:57.735Z
paper_authors._id: 63922fc073bf5173e3631bb7
paper_authors.avatarUrl: /avatars/d672ca1cf5a541ec51f458b88eb9acec.svg
paper_authors.fullname: Ayzaan Wahid
paper_authors.type: user
paper_authors.user: ayzaan
paper_authors._id: 65cae6465495933ab081f5ae
paper_authors.name: Zhuo Xu
paper_authors._id: 65cae6465495933ab081f5af
paper_authors.name: Quan Vuong
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T11:50:57.619Z
paper_authors._id: 63e1d6687fbb6ae4d4f4d980
paper_authors.avatarUrl: /avatars/4959319e69d69e03a31ff4be326e602a.svg
paper_authors.fullname: Quan Vuong
paper_authors.type: user
paper_authors.user: quintonx
paper_authors._id: 65cae6465495933ab081f5b0
paper_authors.name: Tingnan Zhang
paper_authors._id: 65cae6465495933ab081f5b1
paper_authors.name: Tsang-Wei Edward Lee
paper_authors._id: 65cae6465495933ab081f5b2
paper_authors.name: Kuang-Huei Lee
paper_authors._id: 65cae6465495933ab081f5b3
paper_authors.name: Peng Xu
paper_authors._id: 65cae6465495933ab081f5b4
paper_authors.name: Sean Kirmani
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T11:52:07.938Z
paper_authors._id: 648c85f33ffc11989bddcce1
paper_authors.avatarUrl: /avatars/bdde73b95a36e32cb2975656cca46022.svg
paper_authors.fullname: Sean Kirmani
paper_authors.type: user
paper_authors.user: skirmani
paper_authors._id: 65cae6465495933ab081f5b5
paper_authors.name: Yuke Zhu
paper_authors._id: 65cae6465495933ab081f5b6
paper_authors.name: Andy Zeng
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T11:52:34.206Z
paper_authors._id: 62d5f9d705461e9459d20b91
paper_authors.avatarUrl: /avatars/120917da2d8587674505163fb960e27d.svg
paper_authors.fullname: Andy Zeng
paper_authors.type: user
paper_authors.user: andyzeng
paper_authors._id: 65cae6465495933ab081f5b7
paper_authors.name: Karol Hausman
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T11:52:40.789Z
paper_authors._id: 64c93e623a3f428da67d9310
paper_authors.avatarUrl: /avatars/98f0b0f0c8ab9cd94a9f5650f197f6c3.svg
paper_authors.fullname: Karol Hausman
paper_authors.type: user
paper_authors.user: KarolHausman
paper_authors._id: 65cae6465495933ab081f5b8
paper_authors.name: Nicolas Heess
paper_authors._id: 65cae6465495933ab081f5b9
paper_authors.name: Chelsea Finn
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-13T11:52:52.411Z
paper_authors._id: 64ac22a9193f0a807deb673d
paper_authors.avatarUrl: /avatars/fcac4912678ad3cb6e817d40bdee9aea.svg
paper_authors.fullname: Chelsea Finn
paper_authors.type: user
paper_authors.user: cbfinn
paper_authors._id: 65cae6465495933ab081f5ba
paper_authors.name: Sergey Levine
paper_authors._id: 65cae6465495933ab081f5bb
paper_authors.name: Brian Ichter
paper_title: PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs
paper_summary: Vision language models (VLMs) have shown impressive capabilities across a
  variety of tasks, from logical reasoning to visual understanding. This opens
  the door to richer interaction with the world, for example robotic control.
  However, VLMs produce only textual outputs, while robotic control and other
  spatial tasks require outputting continuous coordinates, actions, or
  trajectories. How can we enable VLMs to handle such settings without
  fine-tuning on task-specific data?
    In this paper, we propose a novel visual prompting approach for VLMs that we
  call Prompting with Iterative Visual Optimization (PIVOT), which casts tasks as
  iterative visual question answering. In each iteration, the image is annotated
  with a visual representation of proposals that the VLM can refer to (e.g.,
  candidate robot actions, localizations, or trajectories). The VLM then selects
  the best ones for the task. These proposals are iteratively refined, allowing
  the VLM to eventually zero in on the best available answer. We investigate
  PIVOT on real-world robotic navigation, real-world manipulation from images,
  instruction following in simulation, and additional spatial inference tasks
  such as localization. We find, perhaps surprisingly, that our approach enables
  zero-shot control of robotic systems without any robot training data,
  navigation in a variety of environments, and other capabilities. Although
  current performance is far from perfect, our work highlights potentials and
  limitations of this new regime and shows a promising approach for
  Internet-Scale VLMs in robotic and spatial reasoning domains. Website:
  pivot-prompt.github.io and HuggingFace:
  https://huggingface.co/spaces/pivot-prompt/pivot-prompt-demo.
paper_ai_summary: A method called PIVOT enables VLMs to perform iterative visual question answering to generate continuous outputs for tasks like robotic control, enabling zero-shot control without task-specific data.
paper_ai_keywords: visual prompting prompting with iterative visual optimization PIVOT visual question answering robotic navigation image-based manipulation instruction following spatial inference localization