Published as a conference paper at ICLR 2018 DISTRIBUTED PRIORITIZED EXPERIENCE REPLAY
We propose a distributed architecture for deep reinforcement learning at scale, that enables agents to learn effectively from orders of magnitude more data than previously possible. The algorithm decouples acting from learning: the actors interact with their own instances of the environment by selecting actions according to a shared neural network, and accumulate the resulting experience in a shared experience replay memory; the learner replays samples of experience and updates the neural network. The architecture relies on prioritized experience replay to focus only on the most significant data generated by the actors. Our architecture substantially improves the state of the art on the Arcade Learning Environment, achieving better final performance in a fraction of the wall-clock training time.
A parallel aproach to DQN training is proposed, based on the idea of having multiple actors collecting data in parallel, while a single learner trains the model from experiences sampled from a central replay memory. Experiments on Atari game playing and two MuJoCo continuous control tasks show significant improvements in terms of training time and final performance compared to previous baselines.
The core idea is pretty straightforward but the paper does a very good job at demonstrating that it works very well, when implemented efficiently over a large cluster (which is not trivial). I also appreciate the various experiments to analyze the impact of several settings (instead of just reporting a new SOTA). Overall I believe this is definitely a solid contribution that will benefit both practitioners and researchers... as long as they got the computational resources to do so!
There are essentially two more things I would have really liked to see in this paper (maybe for future work?):
- Using all Rainbow components
- Using multiple learners (with actors cycling between them for instance)
Sharing your custom Tensorflow implementation of prioritized experience replay would also be a great bonus!
Minor points:
- Figure 1 does not seem to be referenced in the text 
- « In principle, Q-learning variants are off-policy methods » => not with multi-step unless you do some kind of correction! I think it is important to mention it even if it works well in practice (just saying « furthermore we are using a multi-step return » is too vague)
- When comparing the Gt targets for DQN vs DPG it strikes me that DPG uses the delayed weights phi- to select the action, while DQN uses current weights theta. I am curious to know if there is a good motivation for this and what impact this can have on the training dynamics.
- In caption of Fig. 5 25K should be 250K
- In appendix A why duplicate memory data instead of just using a smaller memory size?
- In appendix D it looks like experiences removed from memory are chosen by sampling instead of just removing the older ones as in DQN. Why use a different scheme?
- Why store rewards and gamma’s at each time step in memory instead of just the total discounted reward?
- It would have been better to re-use the same colors as in Fig. 2 for plots in the appendix
- Would Fig. 10 be more interesting with the full plot and a log scale on the x axis?