title: Offline Actor-Critic Reinforcement Learning Scales to Large Models
summary: We show that offline actor-critic reinforcement learning can scale to large
  models - such as transformers - and follows similar scaling laws as supervised
  learning. We find that offline actor-critic algorithms can outperform strong,
  supervised, behavioral cloning baselines for multi-task training on a large
  dataset containing both sub-optimal and expert behavior on 132 continuous
  control tasks. We introduce a Perceiver-based actor-critic model and elucidate
  the key model features needed to make offline RL work with self- and
  cross-attention modules. Overall, we find that: i) simple offline actor critic
  algorithms are a natural choice for gradually moving away from the currently
  predominant paradigm of behavioral cloning, and ii) via offline RL it is
  possible to learn multi-task policies that master many domains simultaneously,
  including real robotics tasks, from sub-optimal demonstrations or
  self-generated data.
paper_authors._id: 65c5a405d1dcbd30d8a5d624
paper_authors.name: Jost Tobias Springenberg
paper_authors._id: 65c5a405d1dcbd30d8a5d625
paper_authors.name: Abbas Abdolmaleki
paper_authors._id: 65c5a405d1dcbd30d8a5d626
paper_authors.name: Jingwei Zhang
paper_authors._id: 65c5a405d1dcbd30d8a5d627
paper_authors.name: Oliver Groth
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:06:09.624Z
paper_authors._id: 6372a44e0f3db8b7e625f6dc
paper_authors.avatarUrl: /avatars/814237f0198f69407a2981ee01abb5bc.svg
paper_authors.fullname: Oliver Groth
paper_authors.type: user
paper_authors.user: ogroth
paper_authors._id: 65c5a405d1dcbd30d8a5d628
paper_authors.name: Michael Bloesch
paper_authors._id: 65c5a405d1dcbd30d8a5d629
paper_authors.name: Thomas Lampe
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:06:21.569Z
paper_authors._id: 631fa7e9124782a19efd20f2
paper_authors.avatarUrl: /avatars/456ef70cdb2a6a1f79a078746e96034a.svg
paper_authors.fullname: Thomas Lampe
paper_authors.type: user
paper_authors.user: tlampe
paper_authors._id: 65c5a405d1dcbd30d8a5d62a
paper_authors.name: Philemon Brakel
paper_authors._id: 65c5a405d1dcbd30d8a5d62b
paper_authors.name: Sarah Bechtle
paper_authors._id: 65c5a405d1dcbd30d8a5d62c
paper_authors.name: Steven Kapturowski
paper_authors._id: 65c5a405d1dcbd30d8a5d62d
paper_authors.name: Roland Hafner
paper_authors.status: admin_assigned
paper_authors.statusLastChangedAt: 2024-02-09T10:06:37.277Z
paper_authors._id: 63bd22e10608fbc76b844c73
paper_authors.avatarUrl: /avatars/4a37ef01b25d047ba85296d12578429e.svg
paper_authors.fullname: Roland Hafner
paper_authors.type: user
paper_authors.user: LMTO007
paper_authors._id: 65c5a405d1dcbd30d8a5d62e
paper_authors.name: Nicolas Heess
paper_authors._id: 65c5a405d1dcbd30d8a5d62f
paper_authors.name: Martin Riedmiller
paper_title: Offline Actor-Critic Reinforcement Learning Scales to Large Models
paper_summary: We show that offline actor-critic reinforcement learning can scale to large
  models - such as transformers - and follows similar scaling laws as supervised
  learning. We find that offline actor-critic algorithms can outperform strong,
  supervised, behavioral cloning baselines for multi-task training on a large
  dataset containing both sub-optimal and expert behavior on 132 continuous
  control tasks. We introduce a Perceiver-based actor-critic model and elucidate
  the key model features needed to make offline RL work with self- and
  cross-attention modules. Overall, we find that: i) simple offline actor critic
  algorithms are a natural choice for gradually moving away from the currently
  predominant paradigm of behavioral cloning, and ii) via offline RL it is
  possible to learn multi-task policies that master many domains simultaneously,
  including real robotics tasks, from sub-optimal demonstrations or
  self-generated data.
paper_ai_summary: Offline actor-critic reinforcement learning with transformers can outperform supervised baselines in multi-task training, using self- and cross-attention for better generalization and learning from diverse data sources.
paper_ai_keywords: actor-critic reinforcement learning transformers scaling laws behavioral cloning Perceiver-based models self-attention cross-attention multi-task policies robotics tasks