Boosting Trust Region Policy Optimization by Normalizing flows Policy
We propose to improve trust region policy search with normalizing flows policy. We illustrate that when the trust region is constructed by KL divergence constraint, normalizing flows policy can generate samples far from the *center* of the previous policy iterate, which potentially enables better exploration and helps avoid bad local optima. We show that normalizing flows policy significantly improves upon factorized Gaussian policy baseline, with both TRPO and ACKTR, especially on tasks with complex dynamics such as Humanoid.
 The authors in this work present an approach to policy optimization that relies on an alternative policy formulation based on normalizing flows. This is a relatively simple modification (this is no criticism) that essentially uses the same TRPO algorithm as previous approaches, but a different mechanism for generating the distribution over actions. The crux of the authors’ approach is detailed in equations (6) and (7), although it could have been useful to see more of the discussion of the architecture from appendix B in the actual text of the paper. The authors then go on to analyze the properties and expressiveness of the resulting properties and show that it is more capable of capturing complex interactions than a simple Gaussian. It was somewhat unclear, however, in section 4.2 what the exact form of the policies being compared are. Is this a simple example with only the parameters of the Gaussian, or was the Gaussian parameterized by a multi-layer model? Further, one thing I would also have liked to see the authors question more is, for the problems they attack, whether this expressiveness is more useful “during exploration” or for the ultimate performance of the final policy. The authors, finally, show that this approach is able to out-perform the alternative Gaussian policy. Ultimately this approach seems to be a simple modification (or replacement) of the standard policy formulation, and one that seems to lead to good performance gains.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Boosting Trust Region Policy Optimization by Normalizing flows Policy
We propose to improve trust region policy search with normalizing flows policy. We illustrate that when the trust region is constructed by KL divergence constraint, normalizing flows policy can generate samples far from the *center* of the previous policy iterate, which potentially enables better exploration and helps avoid bad local optima. We show that normalizing flows policy significantly improves upon factorized Gaussian policy baseline, with both TRPO and ACKTR, especially on tasks with complex dynamics such as Humanoid.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The authors in this work present an approach to policy optimization that relies on an alternative policy formulation based on normalizing flows. This is a relatively simple modification (this is no criticism) that essentially uses the same TRPO algorithm as previous approaches, but a different mechanism for generating the distribution over actions. The crux of the authors’ approach is detailed in equations (6) and (7), although it could have been useful to see more of the discussion of the architecture from appendix B in the actual text of the paper. The authors then go on to analyze the properties and expressiveness of the resulting properties and show that it is more capable of capturing complex interactions than a simple Gaussian. It was somewhat unclear, however, in section 4.2 what the exact form of the policies being compared are. Is this a simple example with only the parameters of the Gaussian, or was the Gaussian parameterized by a multi-layer model? Further, one thing I would also have liked to see the authors question more is, for the problems they attack, whether this expressiveness is more useful “during exploration” or for the ultimate performance of the final policy. The authors, finally, show that this approach is able to out-perform the alternative Gaussian policy. Ultimately this approach seems to be a simple modification (or replacement) of the standard policy formulation, and one that seems to lead to good performance gains.
assistant
 The authors in this work present an approach to policy optimization that relies on an alternative policy formulation based on normalizing flows. This is a relatively simple modification (this is no criticism) that essentially uses the same TRPO algorithm as previous approaches, but a different mechanism for generating the distribution over actions. The crux of the authors’ approach is detailed in equations (6) and (7), although it could have been useful to see more of the discussion of the architecture from appendix B in the actual text of the paper. The authors then go on to analyze the properties and expressiveness of the resulting properties and show that it is more capable of capturing complex interactions than a simple Gaussian. It was somewhat unclear, however, in section 4.2 what the exact form of the policies being compared are. Is this a simple example with only the parameters of the Gaussian, or was the Gaussian parameterized by a multi-layer model? Further, one thing I would also have liked to see the authors question more is, for the problems they attack, whether this expressiveness is more useful “during exploration” or for the ultimate performance of the final policy. The authors, finally, show that this approach is able to out-perform the alternative Gaussian policy. Ultimately this approach seems to be a simple modification (or replacement) of the standard policy formulation, and one that seems to lead to good performance gains.