title: Leveraging Skills from Unlabeled Prior Data for Efficient Online
    Exploration
summary: Unsupervised pretraining has been transformative in many supervised domains.
  However, applying such ideas to reinforcement learning (RL) presents a unique
  challenge in that fine-tuning does not involve mimicking task-specific data,
  but rather exploring and locating the solution through iterative
  self-improvement. In this work, we study how unlabeled prior trajectory data
  can be leveraged to learn efficient exploration strategies. While prior data
  can be used to pretrain a set of low-level skills, or as additional off-policy
  data for online RL, it has been unclear how to combine these ideas effectively
  for online exploration. Our method SUPE (Skills from Unlabeled Prior data for
  Exploration) demonstrates that a careful combination of these ideas compounds
  their benefits. Our method first extracts low-level skills using a variational
  autoencoder (VAE), and then pseudo-relabels unlabeled trajectories using an
  optimistic reward model, transforming prior data into high-level, task-relevant
  examples. Finally, SUPE uses these transformed examples as additional
  off-policy data for online RL to learn a high-level policy that composes
  pretrained low-level skills to explore efficiently. We empirically show that
  SUPE reliably outperforms prior strategies, successfully solving a suite of
  long-horizon, sparse-reward tasks. Code: https://github.com/rail-berkeley/supe.
paper_authors._id: 6719e6dccc4568fb62d392ec
paper_authors.name: Max Wilcoxson
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-28T15:49:01.754Z
paper_authors._id: 6568d51a16053f9a304585f9
paper_authors.avatarUrl: /avatars/2dc0596e321c978b9513cf5f67141c2a.svg
paper_authors.fullname: Max Wilcoxson
paper_authors.type: user
paper_authors.user: mwilcoxson
paper_authors._id: 6719e6dccc4568fb62d392ed
paper_authors.name: Qiyang Li
paper_authors._id: 6719e6dccc4568fb62d392ee
paper_authors.name: Kevin Frans
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-28T15:48:46.738Z
paper_authors._id: 6321e71ca97afe3c4c66f7b5
paper_authors.avatarUrl: /avatars/02cf0d5779417b37e34cc6340bffacad.svg
paper_authors.fullname: Kevin Frans
paper_authors.type: user
paper_authors.user: kvfrans
paper_authors._id: 6719e6dccc4568fb62d392ef
paper_authors.name: Sergey Levine
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-10-28T15:48:55.303Z
paper_authors._id: 665ce54120a307a3754849dd
paper_authors.avatarUrl: /avatars/e698726e9be61dd50ce2efe372ed5dac.svg
paper_authors.fullname: Sergey Levine
paper_authors.type: user
paper_authors.user: svlevine
paper_title: Leveraging Skills from Unlabeled Prior Data for Efficient Online
    Exploration
paper_summary: Unsupervised pretraining has been transformative in many supervised domains.
  However, applying such ideas to reinforcement learning (RL) presents a unique
  challenge in that fine-tuning does not involve mimicking task-specific data,
  but rather exploring and locating the solution through iterative
  self-improvement. In this work, we study how unlabeled prior trajectory data
  can be leveraged to learn efficient exploration strategies. While prior data
  can be used to pretrain a set of low-level skills, or as additional off-policy
  data for online RL, it has been unclear how to combine these ideas effectively
  for online exploration. Our method SUPE (Skills from Unlabeled Prior data for
  Exploration) demonstrates that a careful combination of these ideas compounds
  their benefits. Our method first extracts low-level skills using a variational
  autoencoder (VAE), and then pseudo-relabels unlabeled trajectories using an
  optimistic reward model, transforming prior data into high-level, task-relevant
  examples. Finally, SUPE uses these transformed examples as additional
  off-policy data for online RL to learn a high-level policy that composes
  pretrained low-level skills to explore efficiently. We empirically show that
  SUPE reliably outperforms prior strategies, successfully solving a suite of
  long-horizon, sparse-reward tasks. Code: https://github.com/rail-berkeley/supe.
paper_ai_summary: SUPE uses unlabeled trajectory data and a variational autoencoder to learn low-level skills and high-level exploration strategies in reinforcement learning.
paper_ai_keywords: variational autoencoder optimistic reward model offline RL online RL long-horizon tasks sparse-reward tasks