title: Vision-Language Models as a Source of Rewards
summary: Building generalist agents that can accomplish many goals in rich open-ended
  environments is one of the research frontiers for reinforcement learning. A key
  limiting factor for building generalist agents with RL has been the need for a
  large number of reward functions for achieving different goals. We investigate
  the feasibility of using off-the-shelf vision-language models, or VLMs, as
  sources of rewards for reinforcement learning agents. We show how rewards for
  visual achievement of a variety of language goals can be derived from the CLIP
  family of models, and used to train RL agents that can achieve a variety of
  language goals. We showcase this approach in two distinct visual domains and
  present a scaling trend showing how larger VLMs lead to more accurate rewards
  for visual goal achievement, which in turn produces more capable RL agents.
paper_authors._id: 657bcd1d7e11375c6f27ed4e
paper_authors.name: Kate Baumli
paper_authors._id: 657bcd1d7e11375c6f27ed4f
paper_authors.name: Satinder Baveja
paper_authors._id: 657bcd1d7e11375c6f27ed50
paper_authors.name: Feryal Behbahani
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-09-20T07:25:16.632Z
paper_authors._id: 64c197b7ec3c6181351b2c5d
paper_authors.avatarUrl: /avatars/f81fe98f68720b95885d4968b0e7cc0f.svg
paper_authors.fullname: Feryal Behbahani
paper_authors.type: user
paper_authors.user: ferya
paper_authors._id: 657bcd1d7e11375c6f27ed51
paper_authors.name: Harris Chan
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T10:56:04.792Z
paper_authors._id: 62d1cf9339e0332b575180ef
paper_authors.avatarUrl: /avatars/e43acba1e9d21564fb82835a189624ab.svg
paper_authors.fullname: Harris Chan
paper_authors.type: user
paper_authors.user: takonan
paper_authors._id: 657bcd1d7e11375c6f27ed52
paper_authors.name: Gheorghe Comanici
paper_authors._id: 657bcd1d7e11375c6f27ed53
paper_authors.name: Sebastian Flennerhag
paper_authors._id: 657bcd1d7e11375c6f27ed54
paper_authors.name: Maxime Gazeau
paper_authors._id: 657bcd1d7e11375c6f27ed55
paper_authors.name: Kristian Holsheimer
paper_authors._id: 657bcd1d7e11375c6f27ed56
paper_authors.name: Dan Horgan
paper_authors._id: 657bcd1d7e11375c6f27ed57
paper_authors.name: Michael Laskin
paper_authors._id: 657bcd1d7e11375c6f27ed58
paper_authors.name: Clare Lyle
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-12-15T10:57:07.252Z
paper_authors._id: 6313f69d54e6e5d9f0fb9f10
paper_authors.avatarUrl: /avatars/12114d479bb6dc394d29f988944f6d47.svg
paper_authors.fullname: Clare Lyle
paper_authors.type: user
paper_authors.user: justclarifying
paper_authors._id: 657bcd1d7e11375c6f27ed59
paper_authors.name: Hussain Masoom
paper_authors._id: 657bcd1d7e11375c6f27ed5a
paper_authors.name: Kay McKinney
paper_authors._id: 657bcd1d7e11375c6f27ed5b
paper_authors.name: Volodymyr Mnih
paper_authors._id: 657bcd1d7e11375c6f27ed5c
paper_authors.name: Alexander Neitz
paper_authors._id: 657bcd1d7e11375c6f27ed5d
paper_authors.name: Fabio Pardo
paper_authors._id: 657bcd1d7e11375c6f27ed5e
paper_authors.name: Jack Parker-Holder
paper_authors._id: 657bcd1d7e11375c6f27ed5f
paper_authors.name: John Quan
paper_authors._id: 657bcd1d7e11375c6f27ed60
paper_authors.name: Tim Rocktäschel
paper_authors._id: 657bcd1d7e11375c6f27ed61
paper_authors.name: Himanshu Sahni
paper_authors._id: 657bcd1d7e11375c6f27ed62
paper_authors.name: Tom Schaul
paper_authors._id: 657bcd1d7e11375c6f27ed63
paper_authors.name: Yannick Schroecker
paper_authors._id: 657bcd1d7e11375c6f27ed64
paper_authors.name: Stephen Spencer
paper_authors._id: 657bcd1d7e11375c6f27ed65
paper_authors.name: Richie Steigerwald
paper_authors._id: 657bcd1d7e11375c6f27ed66
paper_authors.name: Luyu Wang
paper_authors._id: 657bcd1d7e11375c6f27ed67
paper_authors.name: Lei Zhang
paper_title: Vision-Language Models as a Source of Rewards
paper_summary: Building generalist agents that can accomplish many goals in rich open-ended
  environments is one of the research frontiers for reinforcement learning. A key
  limiting factor for building generalist agents with RL has been the need for a
  large number of reward functions for achieving different goals. We investigate
  the feasibility of using off-the-shelf vision-language models, or VLMs, as
  sources of rewards for reinforcement learning agents. We show how rewards for
  visual achievement of a variety of language goals can be derived from the CLIP
  family of models, and used to train RL agents that can achieve a variety of
  language goals. We showcase this approach in two distinct visual domains and
  present a scaling trend showing how larger VLMs lead to more accurate rewards
  for visual goal achievement, which in turn produces more capable RL agents.
paper_ai_keywords: reinforcement learning generalist agents reward functions vision-language models VLMs CLIP family of models visual goals scaling trend accurate rewards capable RL agents