title: RL Zero: Zero-Shot Language to Behaviors without any Supervision
summary: Rewards remain an uninterpretable way to specify tasks for Reinforcement
  Learning, as humans are often unable to predict the optimal behavior of any
  given reward function, leading to poor reward design and reward hacking.
  Language presents an appealing way to communicate intent to agents and bypass
  reward design, but prior efforts to do so have been limited by costly and
  unscalable labeling efforts. In this work, we propose a method for a completely
  unsupervised alternative to grounding language instructions in a zero-shot
  manner to obtain policies. We present a solution that takes the form of
  imagine, project, and imitate: The agent imagines the observation sequence
  corresponding to the language description of a task, projects the imagined
  sequence to our target domain, and grounds it to a policy. Video-language
  models allow us to imagine task descriptions that leverage knowledge of tasks
  learned from internet-scale video-text mappings. The challenge remains to
  ground these generations to a policy. In this work, we show that we can achieve
  a zero-shot language-to-behavior policy by first grounding the imagined
  sequences in real observations of an unsupervised RL agent and using a
  closed-form solution to imitation learning that allows the RL agent to mimic
  the grounded observations. Our method, RLZero, is the first to our knowledge to
  show zero-shot language to behavior generation abilities without any
  supervision on a variety of tasks on simulated domains. We further show that
  RLZero can also generate policies zero-shot from cross-embodied videos such as
  those scraped from YouTube.
paper_authors._id: 67579ec47ed449ff067b45ff
paper_authors.name: Harshit Sikchi
paper_authors.status: extracted_pending
paper_authors.statusLastChangedAt: 2024-12-10T01:52:05.311Z
paper_authors._id: 6304559b1dd5d3c6248878f1
paper_authors.avatarUrl: /avatars/50ad7ffcc226c9ed7c6a78634a5fc8b4.svg
paper_authors.fullname: Harshit Sikchi
paper_authors.type: user
paper_authors.user: hsikchi
paper_authors._id: 67579ec47ed449ff067b4600
paper_authors.name: Siddhant Agarwal
paper_authors._id: 67579ec47ed449ff067b4601
paper_authors.name: Pranaya Jajoo
paper_authors._id: 67579ec47ed449ff067b4602
paper_authors.name: Samyak Parajuli
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T13:55:42.044Z
paper_authors._id: 631ba8448de8e645af70820c
paper_authors.avatarUrl: /avatars/ff5709d81010d8980f637775d92a71a5.svg
paper_authors.fullname: Samyak Parajuli
paper_authors.type: user
paper_authors.user: samp830
paper_authors._id: 67579ec47ed449ff067b4603
paper_authors.name: Caleb Chuck
paper_authors._id: 67579ec47ed449ff067b4604
paper_authors.name: Max Rudolph
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T13:56:00.824Z
paper_authors._id: 6388bd0380134ba508cee34f
paper_authors.avatarUrl: /avatars/910c873004d16f404e64a896cc6eff0d.svg
paper_authors.fullname: Max Rudolph
paper_authors.type: user
paper_authors.user: maxrudolph
paper_authors._id: 67579ec47ed449ff067b4605
paper_authors.name: Peter Stone
paper_authors._id: 67579ec47ed449ff067b4606
paper_authors.name: Amy Zhang
paper_authors._id: 67579ec47ed449ff067b4607
paper_authors.name: Scott Niekum
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-12-16T13:56:17.015Z
paper_authors._id: 653675a67f86289757f461ca
paper_authors.avatarUrl: /avatars/47fe2c67305a3062c29c27aa2ae642b2.svg
paper_authors.fullname: Scott Niekum
paper_authors.type: user
paper_authors.user: sniekum
paper_title: RL Zero: Zero-Shot Language to Behaviors without any Supervision
paper_summary: Rewards remain an uninterpretable way to specify tasks for Reinforcement
  Learning, as humans are often unable to predict the optimal behavior of any
  given reward function, leading to poor reward design and reward hacking.
  Language presents an appealing way to communicate intent to agents and bypass
  reward design, but prior efforts to do so have been limited by costly and
  unscalable labeling efforts. In this work, we propose a method for a completely
  unsupervised alternative to grounding language instructions in a zero-shot
  manner to obtain policies. We present a solution that takes the form of
  imagine, project, and imitate: The agent imagines the observation sequence
  corresponding to the language description of a task, projects the imagined
  sequence to our target domain, and grounds it to a policy. Video-language
  models allow us to imagine task descriptions that leverage knowledge of tasks
  learned from internet-scale video-text mappings. The challenge remains to
  ground these generations to a policy. In this work, we show that we can achieve
  a zero-shot language-to-behavior policy by first grounding the imagined
  sequences in real observations of an unsupervised RL agent and using a
  closed-form solution to imitation learning that allows the RL agent to mimic
  the grounded observations. Our method, RLZero, is the first to our knowledge to
  show zero-shot language to behavior generation abilities without any
  supervision on a variety of tasks on simulated domains. We further show that
  RLZero can also generate policies zero-shot from cross-embodied videos such as
  those scraped from YouTube.
paper_ai_keywords: Reinforcement Learning reward function reward hacking language instructions zero-shot grounding imagine project imitate observation sequence video-language models task descriptions unsupervised RL agent imitation learning closed-form solution RLZero zero-shot language-to-behavior policy cross-embodied videos