MARGINAL POLICY GRADIENTS: A UNIFIED FAMILY OF ESTIMATORS FOR BOUNDED ACTION SPACES WITH APPLICATIONS
Many complex domains, such as robotics control and real-time strategy (RTS) games, require an agent to learn a continuous control. In the former, an agent learns a policy over R d and in the latter, over a discrete set of actions each of which is parametrized by a continuous parameter. Such problems are naturally solved using policy based reinforcement learning (RL) methods, but unfortunately these often suffer from high variance leading to instability and slow convergence. Unnecessary variance is introduced whenever policies over bounded action spaces are modeled using distributions with unbounded support by applying a transformation T to the sampled action before execution in the environment. Recently, the variance reduced clipped action policy gradient (CAPG) was introduced for actions in bounded intervals, but to date no variance reduced methods exist when the action is a direction, something often seen in RTS games. To this end we introduce the angular policy gradient (APG), a stochastic policy gradient method for directional control. With the marginal policy gradients family of estimators we present a unified analysis of the variance reduction properties of APG and CAPG; our results provide a stronger guarantee than existing analyses for CAPG. Experimental results on a popular RTS game and a navigation task show that the APG estimator offers a substantial improvement over the standard policy gradient.
Summary
This paper derives a new policy gradient method for when continuous actions are transformed by a
normalization step, a process called angular policy gradients (APG). A generalization based on
a certain class of transformations is presented. The method is an instance of a 
Rao-Blackwellization process and hence reduces variance.
Detailed comments
I enjoyed the concept and, while relatively niche, appreciated the work done here and do believe it has clear applications. I am not convinced that the measure theoretic perspective is always
necessary to convey the insights, although I appreciate the desire for technical correctness. Still,
appealing to measure theory does reduces readership, and I encourage the authors to keep this in
mind as they revise the text.
Generally speaking it seems like a lot of technicalities for a relatively simple result:
marginalizing a distribution onto a lower-dimensional surface.
The paper positions itself generally as dealing with arbitrary transformations T, but really is 
about angular transformations (e.g. Definition 3.1). The generalization is relatively 
straightforward and was not too surprising given the APG theory. The paper would gain in clarity
if its scope was narrowed. 
It's hard for me to judge of the experimental results of section 5.3, given that there are no other 
benchmarks or provided reference paper. As a whole, I see APG as providing a minor benefit over PG.
Def 4.4: "a notion of Fisher information" -- maybe "variant" is better than "notion", which implies there are different kinds of Fisher information 
Def 3.1 mu is overloaded: parameter or measure?
4.4, law of total variation -- define 
Overall
This was a fun, albeit incremental paper. The method is unlikely to set new SOTA, but I appreciated
the appeal to measure theory to formalize some of the concepts.
Questions
What does E_{pi|s} refer to in Eqn 4.1?
Can you clarify what it means for the map T to be a sufficient statistic for theta? (Theorem 4.6)
Experiment 5.1: Why would we expect APG with a 2d Gaussian to perform better than a 1d Gaussian
on the angle?
Suggestions
Paragraph 2 of section 3 seems like the key to the whole paper -- I would make it more prominent.
I would include a short 'measure theory' appendix or equivalent reference for the lay reader.
I wonder if the paper's main aim is not actually to bring measure theory to the study of policy
gradients, which would be a laudable goal in and of itself. ICLR may not in this case be the right
venue (nor are the current results substantial enough to justify this) but I do encourage authors to
consider this avenue, e.g. in a journal paper.
= Revised after rebuttal =
I thank the authors for their response. I think this work deserves to be published, in particular because it presents a reasonably straightforward result that others will benefit from. However, I do encourage further work to
1) Provide stronger empirical results (these are not too convincing).
2) Beware of overstating: the argument that the framework is broadly applicable is not that useful, given that it's a lot of work to derive closed-form marginalized estimators.