Vision-Language Models are Zero-Shot Reward Models for Reinforcement
  Learning
Reinforcement learning (RL) requires either manually specifying a reward
function, which is often infeasible, or learning a reward model from a large
amount of human feedback, which is often very expensive. We study a more
sample-efficient alternative: using pretrained vision-language models (VLMs) as
zero-shot reward models (RMs) to specify tasks via natural language. We propose
a natural and general approach to using VLMs as reward models, which we call
VLM-RMs. We use VLM-RMs based on CLIP to train a MuJoCo humanoid to learn
complex tasks without a manually specified reward function, such as kneeling,
doing the splits, and sitting in a lotus position. For each of these tasks, we
only provide a single sentence text prompt describing the desired task with
minimal prompt engineering. We provide videos of the trained agents at:
https://sites.google.com/view/vlm-rm. We can improve performance by providing a
second ``baseline'' prompt and projecting out parts of the CLIP embedding space
irrelevant to distinguish between goal and baseline. Further, we find a strong
scaling effect for VLM-RMs: larger VLMs trained with more compute and data are
better reward models. The failure modes of VLM-RMs we encountered are all
related to known capability limitations of current VLMs, such as limited
spatial reasoning ability or visually unrealistic environments that are far
off-distribution for the VLM. We find that VLM-RMs are remarkably robust as
long as the VLM is large enough. This suggests that future VLMs will become
more and more useful reward models for a wide range of RL applications.
653214f5cdc25730d4c0be20
Juan Rocamonde
admin_assigned
2023-10-20T08:38:28.878Z
64b70c692862fe5b40c1f5d0
/avatars/4bfe49ac44cfda9e9c344268a0488d82.svg
Juan Rocamonde
user
JuanRocamonde
653214f5cdc25730d4c0be21
Victoriano Montesinos
653214f5cdc25730d4c0be22
Elvis Nava
admin_assigned
2023-10-20T08:38:42.821Z
63072da7cb09c0a904286762
/avatars/fbea5f226b7f06ca81c4a717a7329cf9.svg
Elvis Nava
user
ElvisNava
653214f5cdc25730d4c0be23
Ethan Perez
admin_assigned
2023-10-20T08:38:49.292Z
63f91bf4aadb0790bb5fb802
/avatars/8af702e49a639f0d1574745b0ab25c75.svg
Ethan Perez
user
EthanAraragi
653214f5cdc25730d4c0be24
David Lindner
admin_assigned
2023-10-20T08:38:56.222Z
64aef034710608ee1a02b1ac
/avatars/0f8449bf7ee2a908962502ec863155be.svg
David Lindner
user
1obextiopo
Vision-Language Models are Zero-Shot Reward Models for Reinforcement
  Learning
Reinforcement learning (RL) requires either manually specifying a reward
function, which is often infeasible, or learning a reward model from a large
amount of human feedback, which is often very expensive. We study a more
sample-efficient alternative: using pretrained vision-language models (VLMs) as
zero-shot reward models (RMs) to specify tasks via natural language. We propose
a natural and general approach to using VLMs as reward models, which we call
VLM-RMs. We use VLM-RMs based on CLIP to train a MuJoCo humanoid to learn
complex tasks without a manually specified reward function, such as kneeling,
doing the splits, and sitting in a lotus position. For each of these tasks, we
only provide a single sentence text prompt describing the desired task with
minimal prompt engineering. We provide videos of the trained agents at:
https://sites.google.com/view/vlm-rm. We can improve performance by providing a
second ``baseline'' prompt and projecting out parts of the CLIP embedding space
irrelevant to distinguish between goal and baseline. Further, we find a strong
scaling effect for VLM-RMs: larger VLMs trained with more compute and data are
better reward models. The failure modes of VLM-RMs we encountered are all
related to known capability limitations of current VLMs, such as limited
spatial reasoning ability or visually unrealistic environments that are far
off-distribution for the VLM. We find that VLM-RMs are remarkably robust as
long as the VLM is large enough. This suggests that future VLMs will become
more and more useful reward models for a wide range of RL applications.
Pretrained vision-language models can serve as zero-shot reward models in reinforcement learning, enabling complex task learning from minimal natural language prompts.
reinforcement learning reward function reward model pretrained vision-language models zero-shot reward models VLM-RMs CLIP MuJoCo task specification prompt engineering CLIP embedding space spatial reasoning visually unrealistic environments