Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation
In this work, we propose to apply trust region optimization to deep reinforcement learning using a recently proposed Kronecker-factored approximation to the curvature. We extend the framework of natural policy gradient and propose to optimize both the actor and the critic using Kronecker-factored approximate curvature (K-FAC) with trust region; hence we call our method Actor Critic using Kronecker-Factored Trust Region (ACKTR). To the best of our knowledge, this is the first scalable trust region natural gradient method for actor-critic methods. It is also the method that learns non-trivial tasks in continuous control as well as discrete control policies directly from raw pixel inputs. We tested our approach across discrete domains in Atari games as well as continuous domains in the MuJoCo environment. With the proposed methods, we are able to achieve higher rewards and a 2-to 3-fold improvement in sample efficiency on average, compared to previous state-of-the-art on-policy actor-critic methods. Code is available at https://github.com/openai/baselines.
The manuscript discusses an important topic, which is optimization in deep reinforcement learning. The authors extend the use of Kronecker-Factored approximation to develop a second order optimization method for deep reinforcement learning. The optimization method use kronecker-factored approximation to the Fisher matrix to estimate the curvature of the cost, resulting in a scalable approximation to natural gradients. 
The authors demonstrate the power of the method (termed ACKTR) in terms of the performance of agents in Atari and Mujoco RL environments, and compare the proposed algorithm to two previous methods (A2C and TRPO).
Overall the manuscript is well-written and to my knowledge the methodology is a novel application to Kronecker-factored approximation. 
I have the following comments to improve the manuscript:
1- I believe the complexity of the ACKTR, A2C and TRPO algorithms are not the same. The authors compare the performance of these algorithms per iteration in most figures, which shows that ACKTR is superior to the other methods. I suggest that the authors make the same comparison with x-axis representing optimization time instead of number of iterations, thus taking into account the complexity of the different algorithms. This is done in Figure 2 for one of the games. I am proposing here that this time comparison also be performed in the other experiments.
2- I wonder how would a method like ADAM perform relative to ACKTR.
3- The details of the algorithm are distributed at different parts of section 3. I think it is important to include a summary of the algorithm at the end of section 3 or at least in the suppmats to make it clearer.