title: Vision-Language Models are Zero-Shot Reward Models for Reinforcement
    Learning
summary: Reinforcement learning (RL) requires either manually specifying a reward
  function, which is often infeasible, or learning a reward model from a large
  amount of human feedback, which is often very expensive. We study a more
  sample-efficient alternative: using pretrained vision-language models (VLMs) as
  zero-shot reward models (RMs) to specify tasks via natural language. We propose
  a natural and general approach to using VLMs as reward models, which we call
  VLM-RMs. We use VLM-RMs based on CLIP to train a MuJoCo humanoid to learn
  complex tasks without a manually specified reward function, such as kneeling,
  doing the splits, and sitting in a lotus position. For each of these tasks, we
  only provide a single sentence text prompt describing the desired task with
  minimal prompt engineering. We provide videos of the trained agents at:
  https://sites.google.com/view/vlm-rm. We can improve performance by providing a
  second ``baseline'' prompt and projecting out parts of the CLIP embedding space
  irrelevant to distinguish between goal and baseline. Further, we find a strong
  scaling effect for VLM-RMs: larger VLMs trained with more compute and data are
  better reward models. The failure modes of VLM-RMs we encountered are all
  related to known capability limitations of current VLMs, such as limited
  spatial reasoning ability or visually unrealistic environments that are far
  off-distribution for the VLM. We find that VLM-RMs are remarkably robust as
  long as the VLM is large enough. This suggests that future VLMs will become
  more and more useful reward models for a wide range of RL applications.
paper_authors._id: 653214f5cdc25730d4c0be20
paper_authors.name: Juan Rocamonde
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-20T08:38:28.878Z
paper_authors._id: 64b70c692862fe5b40c1f5d0
paper_authors.avatarUrl: /avatars/4bfe49ac44cfda9e9c344268a0488d82.svg
paper_authors.fullname: Juan Rocamonde
paper_authors.type: user
paper_authors.user: JuanRocamonde
paper_authors._id: 653214f5cdc25730d4c0be21
paper_authors.name: Victoriano Montesinos
paper_authors._id: 653214f5cdc25730d4c0be22
paper_authors.name: Elvis Nava
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-20T08:38:42.821Z
paper_authors._id: 63072da7cb09c0a904286762
paper_authors.avatarUrl: /avatars/fbea5f226b7f06ca81c4a717a7329cf9.svg
paper_authors.fullname: Elvis Nava
paper_authors.type: user
paper_authors.user: ElvisNava
paper_authors._id: 653214f5cdc25730d4c0be23
paper_authors.name: Ethan Perez
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-20T08:38:49.292Z
paper_authors._id: 63f91bf4aadb0790bb5fb802
paper_authors.avatarUrl: /avatars/8af702e49a639f0d1574745b0ab25c75.svg
paper_authors.fullname: Ethan Perez
paper_authors.type: user
paper_authors.user: EthanAraragi
paper_authors._id: 653214f5cdc25730d4c0be24
paper_authors.name: David Lindner
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2023-10-20T08:38:56.222Z
paper_authors._id: 64aef034710608ee1a02b1ac
paper_authors.avatarUrl: /avatars/0f8449bf7ee2a908962502ec863155be.svg
paper_authors.fullname: David Lindner
paper_authors.type: user
paper_authors.user: 1obextiopo
paper_title: Vision-Language Models are Zero-Shot Reward Models for Reinforcement
    Learning
paper_summary: Reinforcement learning (RL) requires either manually specifying a reward
  function, which is often infeasible, or learning a reward model from a large
  amount of human feedback, which is often very expensive. We study a more
  sample-efficient alternative: using pretrained vision-language models (VLMs) as
  zero-shot reward models (RMs) to specify tasks via natural language. We propose
  a natural and general approach to using VLMs as reward models, which we call
  VLM-RMs. We use VLM-RMs based on CLIP to train a MuJoCo humanoid to learn
  complex tasks without a manually specified reward function, such as kneeling,
  doing the splits, and sitting in a lotus position. For each of these tasks, we
  only provide a single sentence text prompt describing the desired task with
  minimal prompt engineering. We provide videos of the trained agents at:
  https://sites.google.com/view/vlm-rm. We can improve performance by providing a
  second ``baseline'' prompt and projecting out parts of the CLIP embedding space
  irrelevant to distinguish between goal and baseline. Further, we find a strong
  scaling effect for VLM-RMs: larger VLMs trained with more compute and data are
  better reward models. The failure modes of VLM-RMs we encountered are all
  related to known capability limitations of current VLMs, such as limited
  spatial reasoning ability or visually unrealistic environments that are far
  off-distribution for the VLM. We find that VLM-RMs are remarkably robust as
  long as the VLM is large enough. This suggests that future VLMs will become
  more and more useful reward models for a wide range of RL applications.
paper_ai_summary: Pretrained vision-language models can serve as zero-shot reward models in reinforcement learning, enabling complex task learning from minimal natural language prompts.
paper_ai_keywords: reinforcement learning reward function reward model pretrained vision-language models zero-shot reward models VLM-RMs CLIP MuJoCo task specification prompt engineering CLIP embedding space spatial reasoning visually unrealistic environments