  The potential of offline reinforcement learning (RL) is that high-capacity
models trained on large, heterogeneous datasets can lead to agents that
generalize broadly, analogously to similar advances in vision and NLP. However,
recent works argue that offline RL methods encounter unique challenges to
scaling up model capacity. Drawing on the learnings from these works, we
re-examine previous design choices and find that with appropriate choices:
ResNets, cross-entropy based distributional backups, and feature normalization,
offline Q-learning algorithms exhibit strong performance that scales with model
capacity. Using multi-task Atari as a testbed for scaling and generalization,
we train a single policy on 40 games with near-human performance using up-to 80
million parameter networks, finding that model performance scales favorably
with capacity. In contrast to prior work, we extrapolate beyond dataset
performance even when trained entirely on a large (400M transitions) but highly
suboptimal dataset (51% human-level performance). Compared to
return-conditioned supervised approaches, offline Q-learning scales similarly
with model capacity and has better performance, especially when the dataset is
suboptimal. Finally, we show that offline Q-learning with a diverse dataset is
sufficient to learn powerful representations that facilitate rapid transfer to
novel games and fast online learning on new variations of a training game,
improving over existing state-of-the-art representation learning approaches.

Machine Learning