Offline Actor-Critic Reinforcement Learning Scales to Large Models
We show that offline actor-critic reinforcement learning can scale to large
models - such as transformers - and follows similar scaling laws as supervised
learning. We find that offline actor-critic algorithms can outperform strong,
supervised, behavioral cloning baselines for multi-task training on a large
dataset containing both sub-optimal and expert behavior on 132 continuous
control tasks. We introduce a Perceiver-based actor-critic model and elucidate
the key model features needed to make offline RL work with self- and
cross-attention modules. Overall, we find that: i) simple offline actor critic
algorithms are a natural choice for gradually moving away from the currently
predominant paradigm of behavioral cloning, and ii) via offline RL it is
possible to learn multi-task policies that master many domains simultaneously,
including real robotics tasks, from sub-optimal demonstrations or
self-generated data.
65c5a405d1dcbd30d8a5d624
Jost Tobias Springenberg
65c5a405d1dcbd30d8a5d625
Abbas Abdolmaleki
65c5a405d1dcbd30d8a5d626
Jingwei Zhang
65c5a405d1dcbd30d8a5d627
Oliver Groth
admin_assigned
2024-02-09T10:06:09.624Z
6372a44e0f3db8b7e625f6dc
/avatars/814237f0198f69407a2981ee01abb5bc.svg
Oliver Groth
user
ogroth
65c5a405d1dcbd30d8a5d628
Michael Bloesch
65c5a405d1dcbd30d8a5d629
Thomas Lampe
admin_assigned
2024-02-09T10:06:21.569Z
631fa7e9124782a19efd20f2
/avatars/456ef70cdb2a6a1f79a078746e96034a.svg
Thomas Lampe
user
tlampe
65c5a405d1dcbd30d8a5d62a
Philemon Brakel
65c5a405d1dcbd30d8a5d62b
Sarah Bechtle
65c5a405d1dcbd30d8a5d62c
Steven Kapturowski
65c5a405d1dcbd30d8a5d62d
Roland Hafner
admin_assigned
2024-02-09T10:06:37.277Z
63bd22e10608fbc76b844c73
/avatars/4a37ef01b25d047ba85296d12578429e.svg
Roland Hafner
user
LMTO007
65c5a405d1dcbd30d8a5d62e
Nicolas Heess
65c5a405d1dcbd30d8a5d62f
Martin Riedmiller
Offline Actor-Critic Reinforcement Learning Scales to Large Models
We show that offline actor-critic reinforcement learning can scale to large
models - such as transformers - and follows similar scaling laws as supervised
learning. We find that offline actor-critic algorithms can outperform strong,
supervised, behavioral cloning baselines for multi-task training on a large
dataset containing both sub-optimal and expert behavior on 132 continuous
control tasks. We introduce a Perceiver-based actor-critic model and elucidate
the key model features needed to make offline RL work with self- and
cross-attention modules. Overall, we find that: i) simple offline actor critic
algorithms are a natural choice for gradually moving away from the currently
predominant paradigm of behavioral cloning, and ii) via offline RL it is
possible to learn multi-task policies that master many domains simultaneously,
including real robotics tasks, from sub-optimal demonstrations or
self-generated data.
Offline actor-critic reinforcement learning with transformers can outperform supervised baselines in multi-task training, using self- and cross-attention for better generalization and learning from diverse data sources.
actor-critic reinforcement learning transformers scaling laws behavioral cloning Perceiver-based models self-attention cross-attention multi-task policies robotics tasks