RL Zero: Zero-Shot Language to Behaviors without any Supervision
Rewards remain an uninterpretable way to specify tasks for Reinforcement
Learning, as humans are often unable to predict the optimal behavior of any
given reward function, leading to poor reward design and reward hacking.
Language presents an appealing way to communicate intent to agents and bypass
reward design, but prior efforts to do so have been limited by costly and
unscalable labeling efforts. In this work, we propose a method for a completely
unsupervised alternative to grounding language instructions in a zero-shot
manner to obtain policies. We present a solution that takes the form of
imagine, project, and imitate: The agent imagines the observation sequence
corresponding to the language description of a task, projects the imagined
sequence to our target domain, and grounds it to a policy. Video-language
models allow us to imagine task descriptions that leverage knowledge of tasks
learned from internet-scale video-text mappings. The challenge remains to
ground these generations to a policy. In this work, we show that we can achieve
a zero-shot language-to-behavior policy by first grounding the imagined
sequences in real observations of an unsupervised RL agent and using a
closed-form solution to imitation learning that allows the RL agent to mimic
the grounded observations. Our method, RLZero, is the first to our knowledge to
show zero-shot language to behavior generation abilities without any
supervision on a variety of tasks on simulated domains. We further show that
RLZero can also generate policies zero-shot from cross-embodied videos such as
those scraped from YouTube.
67579ec47ed449ff067b45ff
Harshit Sikchi
extracted_pending
2024-12-10T01:52:05.311Z
6304559b1dd5d3c6248878f1
/avatars/50ad7ffcc226c9ed7c6a78634a5fc8b4.svg
Harshit Sikchi
user
hsikchi
67579ec47ed449ff067b4600
Siddhant Agarwal
67579ec47ed449ff067b4601
Pranaya Jajoo
67579ec47ed449ff067b4602
Samyak Parajuli
admin_assigned
2024-12-16T13:55:42.044Z
631ba8448de8e645af70820c
/avatars/ff5709d81010d8980f637775d92a71a5.svg
Samyak Parajuli
user
samp830
67579ec47ed449ff067b4603
Caleb Chuck
67579ec47ed449ff067b4604
Max Rudolph
admin_assigned
2024-12-16T13:56:00.824Z
6388bd0380134ba508cee34f
/avatars/910c873004d16f404e64a896cc6eff0d.svg
Max Rudolph
user
maxrudolph
67579ec47ed449ff067b4605
Peter Stone
67579ec47ed449ff067b4606
Amy Zhang
67579ec47ed449ff067b4607
Scott Niekum
admin_assigned
2024-12-16T13:56:17.015Z
653675a67f86289757f461ca
/avatars/47fe2c67305a3062c29c27aa2ae642b2.svg
Scott Niekum
user
sniekum
RL Zero: Zero-Shot Language to Behaviors without any Supervision
Rewards remain an uninterpretable way to specify tasks for Reinforcement
Learning, as humans are often unable to predict the optimal behavior of any
given reward function, leading to poor reward design and reward hacking.
Language presents an appealing way to communicate intent to agents and bypass
reward design, but prior efforts to do so have been limited by costly and
unscalable labeling efforts. In this work, we propose a method for a completely
unsupervised alternative to grounding language instructions in a zero-shot
manner to obtain policies. We present a solution that takes the form of
imagine, project, and imitate: The agent imagines the observation sequence
corresponding to the language description of a task, projects the imagined
sequence to our target domain, and grounds it to a policy. Video-language
models allow us to imagine task descriptions that leverage knowledge of tasks
learned from internet-scale video-text mappings. The challenge remains to
ground these generations to a policy. In this work, we show that we can achieve
a zero-shot language-to-behavior policy by first grounding the imagined
sequences in real observations of an unsupervised RL agent and using a
closed-form solution to imitation learning that allows the RL agent to mimic
the grounded observations. Our method, RLZero, is the first to our knowledge to
show zero-shot language to behavior generation abilities without any
supervision on a variety of tasks on simulated domains. We further show that
RLZero can also generate policies zero-shot from cross-embodied videos such as
those scraped from YouTube.
A novel unsupervised method translates language instructions into policies using imagination, projection, and imitation, enabling zero-shot language-to-behavior generation in RL.
Reinforcement Learning reward function language instructions zero-shot grounding unsupervised RL video-language models imitation learning RLZero