BACKPLAY: 'MAN MUSS IMMER UMKEHREN'
Model-free reinforcement learning (RL) requires a large number of trials to learn a good policy, especially in environments with sparse rewards. We explore a method to improve the sample efficiency when we have access to demonstrations. Our approach, Backplay, uses a single demonstration to construct a curriculum for a given task. Rather than starting each training episode in the environment's fixed initial state, we start the agent near the end of the demonstration and move the starting point backwards during the course of training until we reach the initial state. Our contributions are that we analytically characterize the types of environments where Backplay can improve training speed, demonstrate the effectiveness of Backplay both in large grid worlds and a complex four player zero-sum game (Pommerman), and show that Backplay compares favorably to other competitive methods known to improve sample efficiency. This includes reward shaping, behavioral cloning, and reverse curriculum generation.
This paper presents a method for increasing the efficiency of sparse reward RL methods through a backward curriculum on expert demonstrations. The method in the paper is as follows: assuming access to expert demonstration and a resettable simulator, the start state of the agent in the beginning of training is sampled from end of demonstration (close to the rewarding state) where the task of achieving the goal is easy. Then gradually through a curriculum this is shifted backwards in the demonstration, making the task gradually harder. 
The proposed method is closely related to 1) “Learning Montezuma’s Revenge from a Single Demonstration” a blog post and open-source code release by Salimans and Chen (OpenAI Blog, 2018) where they show that constructing a curriculum that gradually moves the starting state back from the end of a single demonstration to the beginning helps solve Montezuma’s revenge game 2) “Reverse Curriculum Generation for Reinforcement Learning” by Florensa et al. (CoRL 2017) , where they start the training to reach a goal from start states nearby a given goal state and gradually the agent is trained to solve the task from increasingly distant start states. 
The approach is evaluated on a pair of tasks, a maze environment and a stochastic four-player game, Pommerman. In the maze environment, they compare to vanilla PPO and Uniformly sampled starting points across the expert trajectory. The Backplay method outperforms the vanilla baseline, however, from the training curves (~3500 epochs) in the appendix A4, it looks like the Uniform sampling baseline is doing as well or better than the proposed method. As pointed out by the authors themselves the reverse curriculum does not seem necessary in this environment. Also, it is unclear to me whether the curves shown is comparable as the starting point of the agent, at least in the beginning of training, is close to the goal with higher success rate for the Backplay method compared to baselines. A good convincing assessment would be to report success rate against the same starting point for all methods preferably not from the starting point of the demonstrations to assess generalisation of these methods for which authors briefly report unsuccessful results. 
The Pommerman environment is more complex and the results reported are more interesting. Figure 3 shows the results on four different maps for which expert demonstrations are generated from a Finite-Machine Tree-search method (a competitive method in this environment). I’m slightly confused by the plots and the significant drops in performance once the curriculum is finished and agent encounters the start position of the demonstration trajectory (epoch 250). Is this affected by the schedule of the curriculum? Also, the choice of terminating training at epoch 550 is not clear as the method does not seem to have converged yet (the variance is quite high) and would be interesting to observe the dynamics of learning as the training proceeds and whether it converges to a stable policy at all. I am also slightly unclear regarding the performance difference between Standard method in Figure 3(c) and 3(d). If the Standard method is still the same baseline, vanilla PPO, why such huge performance difference? In my understanding, only the Uniform and Backplay methods should be affected by the quality of demonstrations? I believe this figure needs more explanation and clarity. I am also not clear on why Standard method is terminated at epoch 450 while other methods are trained until epoch 550. Figure 4 reports results of generalisation to 10 unseen maps but again the choice of terminating training after 550 epochs is not clear to me as the method again does not seem to have converged. 
Overall, the choice of parameters is not well motivated, these include the window size for sampling the start point, the schedule for shifting the start point, batch size (102400 seems large to me and this choice is never explained), horizon (in appendix A3 reported to be 1707 for Maze while in the main text it is reported as 200 steps), termination of training (3500 for Maze, Figure 7, and 550 in Pommerman, Figure 3). 
I commend the authors for honestly reporting their method’s shortcomings such as failure in generalisation, however, I find that the work lacks significance and quality. There is not much novelty in the proposed method and there is a clear lack of comparisons to existing sample efficient LfD techniques such as Generative Adversarial Imitation Learning (GAIL). I believe this paper requires substantial improvements for publication and is not up to the ICLR standards in its current form.