Sample Efficient Policy Gradient Methods with Recursive Variance Reduction
Improving the sample efficiency in reinforcement learning has been a long-standing research problem. In this work, we aim to reduce the sample complexity of existing policy gradient methods. We propose a novel policy gradient algorithm called SRVR-PG, which only requires footnote{ notation hides constant factors.} episodes to find an -approximate stationary point of the nonconcave performance function (i.e., such that ). This sample complexity improves the existing result for stochastic variance reduced policy gradient algorithms by a factor of . In addition, we also propose a variant of SRVR-PG with parameter exploration, which explores the initial policy parameter from a prior probability distribution. We conduct numerical experiments on classic control problems in reinforcement learning to validate the performance of our proposed algorithms.
The authors propose a new stochastic reduced variance policy gradient estimator, which combines a baseline GPOMDP estimator with a control variate integrating past gradients by importance re-weighting. The authors establish the sample complexity of gradient descent using the proposed estimator, and further demonstrate its effectiveness through some simple empirical results. I believe this paper is a good contribution for ICLR. The result is relevant and interesting, and extends recent ideas around reduced-variance policy gradient estimators. The paper is overall easy to read, and presents its ideas clearly. Some detailed comments: - The wording of theorem 4.5 and corollary 4.7 could be somewhat clarified. In particular, I did not see Phi defined in the main text, and given its definition in the appendix, I believe theorem 4.5 could be stated simply in terms of J, avoiding any additional notation. Similarly, corollary 4.7 could be stated somewhat more clearly, and in particular, the choice of S should be made explicit. In the appendix, I could not find a definition for phi. - The empirical results presented are interesting, although I wish they were more comprehensive. In particular, it would be valuable to more exhaustively evaluate the impact of the hyper-parameters N, B and m. The authors should also clarify how the current values were chosen. Given that the theoretical results also apply to projected gradient descent, it would be interesting to see empirical results in that case.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Sample Efficient Policy Gradient Methods with Recursive Variance Reduction
Improving the sample efficiency in reinforcement learning has been a long-standing research problem. In this work, we aim to reduce the sample complexity of existing policy gradient methods. We propose a novel policy gradient algorithm called SRVR-PG, which only requires footnote{ notation hides constant factors.} episodes to find an -approximate stationary point of the nonconcave performance function (i.e., such that ). This sample complexity improves the existing result for stochastic variance reduced policy gradient algorithms by a factor of . In addition, we also propose a variant of SRVR-PG with parameter exploration, which explores the initial policy parameter from a prior probability distribution. We conduct numerical experiments on classic control problems in reinforcement learning to validate the performance of our proposed algorithms.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



The authors propose a new stochastic reduced variance policy gradient estimator, which combines a baseline GPOMDP estimator with a control variate integrating past gradients by importance re-weighting. The authors establish the sample complexity of gradient descent using the proposed estimator, and further demonstrate its effectiveness through some simple empirical results. I believe this paper is a good contribution for ICLR. The result is relevant and interesting, and extends recent ideas around reduced-variance policy gradient estimators. The paper is overall easy to read, and presents its ideas clearly. Some detailed comments: - The wording of theorem 4.5 and corollary 4.7 could be somewhat clarified. In particular, I did not see Phi defined in the main text, and given its definition in the appendix, I believe theorem 4.5 could be stated simply in terms of J, avoiding any additional notation. Similarly, corollary 4.7 could be stated somewhat more clearly, and in particular, the choice of S should be made explicit. In the appendix, I could not find a definition for phi. - The empirical results presented are interesting, although I wish they were more comprehensive. In particular, it would be valuable to more exhaustively evaluate the impact of the hyper-parameters N, B and m. The authors should also clarify how the current values were chosen. Given that the theoretical results also apply to projected gradient descent, it would be interesting to see empirical results in that case.
assistant
The authors propose a new stochastic reduced variance policy gradient estimator, which combines a baseline GPOMDP estimator with a control variate integrating past gradients by importance re-weighting. The authors establish the sample complexity of gradient descent using the proposed estimator, and further demonstrate its effectiveness through some simple empirical results. I believe this paper is a good contribution for ICLR. The result is relevant and interesting, and extends recent ideas around reduced-variance policy gradient estimators. The paper is overall easy to read, and presents its ideas clearly. Some detailed comments: - The wording of theorem 4.5 and corollary 4.7 could be somewhat clarified. In particular, I did not see Phi defined in the main text, and given its definition in the appendix, I believe theorem 4.5 could be stated simply in terms of J, avoiding any additional notation. Similarly, corollary 4.7 could be stated somewhat more clearly, and in particular, the choice of S should be made explicit. In the appendix, I could not find a definition for phi. - The empirical results presented are interesting, although I wish they were more comprehensive. In particular, it would be valuable to more exhaustively evaluate the impact of the hyper-parameters N, B and m. The authors should also clarify how the current values were chosen. Given that the theoretical results also apply to projected gradient descent, it would be interesting to see empirical results in that case.