title: D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learning
summary: Offline reinforcement learning algorithms hold the promise of enabling
  data-driven RL methods that do not require costly or dangerous real-world
  exploration and benefit from large pre-collected datasets. This in turn can
  facilitate real-world applications, as well as a more standardized approach to
  RL research. Furthermore, offline RL methods can provide effective
  initializations for online finetuning to overcome challenges with exploration.
  However, evaluating progress on offline RL algorithms requires effective and
  challenging benchmarks that capture properties of real-world tasks, provide a
  range of task difficulties, and cover a range of challenges both in terms of
  the parameters of the domain (e.g., length of the horizon, sparsity of rewards)
  and the parameters of the data (e.g., narrow demonstration data or broad
  exploratory data). While considerable progress in offline RL in recent years
  has been enabled by simpler benchmark tasks, the most widely used datasets are
  increasingly saturating in performance and may fail to reflect properties of
  realistic tasks. We propose a new benchmark for offline RL that focuses on
  realistic simulations of robotic manipulation and locomotion environments,
  based on models of real-world robotic systems, and comprising a variety of data
  sources, including scripted data, play-style data collected by human
  teleoperators, and other data sources. Our proposed benchmark covers
  state-based and image-based domains, and supports both offline RL and online
  fine-tuning evaluation, with some of the tasks specifically designed to require
  both pre-training and fine-tuning. We hope that our proposed benchmark will
  facilitate further progress on both offline RL and fine-tuning algorithms.
  Website with code, examples, tasks, and data is available at
  https://sites.google.com/view/d5rl/
paper_authors._id: 66c2a724a15b4eed7f307fd7
paper_authors.name: Rafael Rafailov
paper_authors._id: 66c2a724a15b4eed7f307fd8
paper_authors.name: Kyle Hatch
paper_authors._id: 66c2a724a15b4eed7f307fd9
paper_authors.name: Anikait Singh
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-08-19T22:13:18.609Z
paper_authors._id: 6511ee845b7e52b0251fdee9
paper_authors.avatarUrl: https://cdn-avatars.huggingface.co/v1/production/uploads/6511ee845b7e52b0251fdee9/hTIwiIYBGOVnIrxtpri83.png
paper_authors.fullname: Anikait Singh
paper_authors.type: user
paper_authors.user: Asap7772
paper_authors._id: 66c2a724a15b4eed7f307fda
paper_authors.name: Laura Smith
paper_authors._id: 66c2a724a15b4eed7f307fdb
paper_authors.name: Aviral Kumar
paper_authors._id: 66c2a724a15b4eed7f307fdc
paper_authors.name: Ilya Kostrikov
paper_authors._id: 66c2a724a15b4eed7f307fdd
paper_authors.name: Philippe Hansen-Estruch
paper_authors.status: claimed_verified
paper_authors.statusLastChangedAt: 2024-12-17T08:04:52.912Z
paper_authors._id: 644eb923a8f384abd038a4f7
paper_authors.avatarUrl: /avatars/3d1a29c6149a6ffc482842f1438ba55a.svg
paper_authors.fullname: Philippe I Hansen-Estruch
paper_authors.type: user
paper_authors.user: philippehansen
paper_authors._id: 66c2a724a15b4eed7f307fde
paper_authors.name: Victor Kolev
paper_authors._id: 66c2a724a15b4eed7f307fdf
paper_authors.name: Philip Ball
paper_authors._id: 66c2a724a15b4eed7f307fe0
paper_authors.name: Jiajun Wu
paper_authors._id: 66c2a724a15b4eed7f307fe1
paper_authors.name: Chelsea Finn
paper_authors._id: 66c2a724a15b4eed7f307fe2
paper_authors.name: Sergey Levine
paper_title: D5RL: Diverse Datasets for Data-Driven Deep Reinforcement Learning
paper_summary: Offline reinforcement learning algorithms hold the promise of enabling
  data-driven RL methods that do not require costly or dangerous real-world
  exploration and benefit from large pre-collected datasets. This in turn can
  facilitate real-world applications, as well as a more standardized approach to
  RL research. Furthermore, offline RL methods can provide effective
  initializations for online finetuning to overcome challenges with exploration.
  However, evaluating progress on offline RL algorithms requires effective and
  challenging benchmarks that capture properties of real-world tasks, provide a
  range of task difficulties, and cover a range of challenges both in terms of
  the parameters of the domain (e.g., length of the horizon, sparsity of rewards)
  and the parameters of the data (e.g., narrow demonstration data or broad
  exploratory data). While considerable progress in offline RL in recent years
  has been enabled by simpler benchmark tasks, the most widely used datasets are
  increasingly saturating in performance and may fail to reflect properties of
  realistic tasks. We propose a new benchmark for offline RL that focuses on
  realistic simulations of robotic manipulation and locomotion environments,
  based on models of real-world robotic systems, and comprising a variety of data
  sources, including scripted data, play-style data collected by human
  teleoperators, and other data sources. Our proposed benchmark covers
  state-based and image-based domains, and supports both offline RL and online
  fine-tuning evaluation, with some of the tasks specifically designed to require
  both pre-training and fine-tuning. We hope that our proposed benchmark will
  facilitate further progress on both offline RL and fine-tuning algorithms.
  Website with code, examples, tasks, and data is available at
  https://sites.google.com/view/d5rl/
paper_ai_keywords: offline reinforcement learning reinforcement learning (RL) data-driven methods pre-collected datasets real-world applications standard approach offline RL methods initializations online fine-tuning evaluation benchmarks real-world tasks horizon sparsity of rewards narrow demonstration data broad exploratory data benchmark tasks realistic simulations robotic manipulation locomotion environments real-world robotic systems scripted data play-style data human teleoperators state-based strategies image-based strategies offline RL online fine-tuning pre-training fine-tuning