  In reinforcement learning (RL), key components of many algorithms are the
exploration strategy and replay buffer. These strategies regulate what
environment data is collected and trained on and have been extensively studied
in the RL literature. In this paper, we investigate the impact of these
components in the context of generalisation in multi-task RL. We investigate
the hypothesis that collecting and training on more diverse data from the
training environments will improve zero-shot generalisation to new tasks. We
motivate mathematically and show empirically that generalisation to tasks that
are "reachable'' during training is improved by increasing the diversity of
transitions in the replay buffer. Furthermore, we show empirically that this
same strategy also shows improvement for generalisation to similar but
"unreachable'' tasks which could be due to improved generalisation of the
learned latent representations.

Machine Learning