We propose to improve trust region policy search with normalizing flows policy. We illustrate that when the trust region is constructed by KL divergence constraint, normalizing flows policy can generate samples far from the 'center' of the previous policy iterate, which potentially enables better exploration and helps avoid bad local optima. We show that normalizing flows policy significantly improves upon factorized Gaussian policy baseline, with both TRPO and ACKTR, especially on tasks with complex dynamics such as Humanoid.
The papers proposed to use normalizing flow policies instead of Gaussian policies to improve exploration and achieve better sample complexity in practice. While I believe this idea has not specifically been tried in previous literature and the vague intuition that NF leads to more exploration that helps learning a better policy, the novelty of combining these two seems limited, and the paper does not seem to provide enough justification to using NF policies instead of alternative policy distributions both in theory and in the experiments.
1. About Section 4.2. I believe that the normalizing flow in question would transform the volume of a Gaussian? So there would exist some parameter setting for a flow model that also shrinks volume, thereby resulting in lower variance policies? The arguments would thereby depend heavily on the specific architecture and initialization of the flow model, which is not discussed in detail. 
Also, why is finding a high variance policy better in terms of the trust region argument? Isn't the whole point of using trust region that the new policy should be closer to old policy to prevent performance degradation? I also think that a fair comparison would be compare KL between normalizing flow policies, instead of KL between NF and Gaussian.
2. The TRPO experiments seem wrong -- at least the results don't match what is reported in the ACKTR paper for Reacher and InverseDoublePendulum envs -- there the TRPO policy at least learns something. Also TRPO in general does not perform as bad as it may seem, see "Deep RL that matters" paper by Henderson et al. Maybe this is because of using OpenAI baselines code which seems to have worse TRPO performance.
There is also no experiments on ACKTR on the small Mujoco tasks (even in the Appendix), which seems to be a rather big oversight given the authors have already done even harder tasks for ACKTR + NF.
Moreover I think a fair comparison is to use almost the same architecture for implicit and gaussian, where the only difference is where you sample the noise. For Gaussian with flows, you can first use an MLP to produce deterministic outputs and then use flow to generate the mean actions. Otherwise it is impossible to say whether the architecture or the implicit distribution contributes more to the success.
One could also use truncated Gaussian distributions / Beta distributions / Gaussian + tanh, since Mujoco actions beyond (-1, 1) is treated as -1 or 1, so Gaussian should already be bad. It is unclear whether NF is able to outperform these settings. 
Minor points:
- Fix citations. Please use \citep throughout.
- Is Equation (6) correct? Seems like \Sigma_i should be the inverse of g_i(\epsilon)? Also this is the "change of variables formula" not "chain rule".
- Why is normalizing flow not part of the background?
- Add legends in Figure (1)
- Figure 2(c), I believe with max entropy you could already obtain diverse ant trajectories?
- I believe in the context of generative models, "implicit" typically means the case where likelihood is not tractable? Here the likelihood is perfectly tractable.