title: Steering Your Generalists: Improving Robotic Foundation Models via Value
    Guidance
summary: Large, general-purpose robotic policies trained on diverse demonstration
  datasets have been shown to be remarkably effective both for controlling a
  variety of robots in a range of different scenes, and for acquiring broad
  repertoires of manipulation skills. However, the data that such policies are
  trained on is generally of mixed quality -- not only are human-collected
  demonstrations unlikely to perform the task perfectly, but the larger the
  dataset is, the harder it is to curate only the highest quality examples. It
  also remains unclear how optimal data from one embodiment is for training on
  another embodiment. In this paper, we present a general and broadly applicable
  approach that enhances the performance of such generalist robot policies at
  deployment time by re-ranking their actions according to a value function
  learned via offline RL. This approach, which we call Value-Guided Policy
  Steering (V-GPS), is compatible with a wide range of different generalist
  policies, without needing to fine-tune or even access the weights of the
  policy. We show that the same value function can improve the performance of
  five different state-of-the-art policies with different architectures, even
  though they were trained on distinct datasets, attaining consistent performance
  improvement on multiple robotic platforms across a total of 12 tasks. Code and
  videos can be found at: https://nakamotoo.github.io/V-GPS
paper_authors._id: 671ae55d82e53a391ca1707e
paper_authors.name: Mitsuhiko Nakamoto
paper_authors._id: 671ae55d82e53a391ca1707f
paper_authors.name: Oier Mees
paper_authors._id: 671ae55d82e53a391ca17080
paper_authors.name: Aviral Kumar
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2025-02-11T15:34:18.785Z
paper_authors._id: 67315a324d2eab8035de786a
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/no-auth/INH5_PlcJYfYcGkyibjgP.png
paper_authors.fullname: Aviral Kumar
paper_authors.type: user
paper_authors.user: aviralku
paper_authors._id: 671ae55d82e53a391ca17081
paper_authors.name: Sergey Levine
paper_title: Steering Your Generalists: Improving Robotic Foundation Models via Value
    Guidance
paper_summary: Large, general-purpose robotic policies trained on diverse demonstration
  datasets have been shown to be remarkably effective both for controlling a
  variety of robots in a range of different scenes, and for acquiring broad
  repertoires of manipulation skills. However, the data that such policies are
  trained on is generally of mixed quality -- not only are human-collected
  demonstrations unlikely to perform the task perfectly, but the larger the
  dataset is, the harder it is to curate only the highest quality examples. It
  also remains unclear how optimal data from one embodiment is for training on
  another embodiment. In this paper, we present a general and broadly applicable
  approach that enhances the performance of such generalist robot policies at
  deployment time by re-ranking their actions according to a value function
  learned via offline RL. This approach, which we call Value-Guided Policy
  Steering (V-GPS), is compatible with a wide range of different generalist
  policies, without needing to fine-tune or even access the weights of the
  policy. We show that the same value function can improve the performance of
  five different state-of-the-art policies with different architectures, even
  though they were trained on distinct datasets, attaining consistent performance
  improvement on multiple robotic platforms across a total of 12 tasks. Code and
  videos can be found at: https://nakamotoo.github.io/V-GPS
paper_ai_summary: Value-Guided Policy Steering (V-GPS) enhances the performance of generalist robotic policies at deployment by re-ranking their actions using an offline RL learned value function, applicable across different policy architectures and robotic platforms.
paper_ai_keywords: offline RL Value-Guided Policy Steering V-GPS generalist robot policies robotic policies deployment time re-ranking value function robotic platforms manipulation skills