Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines
Policy gradient methods have enjoyed great success in deep reinforcement learning but suffer from high variance of gradient estimates. The high variance problem is particularly exasperated in problems with long horizons or high-dimensional action spaces. To mitigate this issue, we derive a bias-free action-dependent baseline for variance reduction which fully exploits the structural form of the stochastic policy itself and does not make any additional assumptions about the MDP. We demonstrate and quantify the benefit of the action-dependent baseline through both theoretical analysis as well as numerical results, including an analysis of the suboptimality of the optimal state-dependent baseline. The result is a computationally efficient policy gradient algorithm, which scales to high-dimensional control problems, as demonstrated by a synthetic 2000-dimensional target matching task. Our experimental results indicate that action-dependent baselines allow for faster learning on standard reinforcement learning benchmarks and high-dimensional hand manipulation and synthetic tasks. Finally, we show that the general idea of including additional information in baselines for improved variance reduction can be extended to partially observed and multi-agent tasks.
 In this paper, the authors investigate variance reduction techniques for agents with multi-dimensional policy outputs, in particular when they are conditionally independent (*factored*). With the increasing focus on applying RL methods to continuous control problems and RTS type games, this is an important problem and this technique seems like an important addition to the RL toolbox. The paper is well written, the method is easy to implement, and the algorithm seems to have clear positive impact on the presented experiments. - The derivations in pages 4-6 are somewhat disconnected from the rest of the paper: the optimal baseline derivation is very standard (even if adapted to the slightly different situation situated here), and for reasons highlighted by the authors in this paper, they are not often used; the *marginalized* baseline is more common, and indeed, the authors adopt this one as well. In light of this (and of the paper being quite a bit over the page limit)- is this material (4.2->4.4) mostly not better suited for the appendix? Same for section 4.6 (which I believe is not used in the experiments). - The experimental section is very strong; regarding the partial observability experiments, assuming actions are here factored as well, I could see four baselines (two choices for whether the baseline has access to the goal location or not, and two choices for whether the baseline has access to the vector ). It*s not clear which two baselines are depicted in 5b - is it possible to disentangle the effect of providing and the location of the hole to the baseline? (side note: it is an interesting idea to include information not available to the agent as input to the baseline though it does feel a bit *iffy* ; the agent requires information to train, but is not provided the information to act. Out of curiosity, is it intended as an experiment to verify the need for better baselines? Or as a *fair* training procedure?) - Minor: in equation 2- is the correct exponent not t*? Also since is define with a scaling (to make it an actual distribution), I believe the definition of should also be multiplied by (as well as equation 2).
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines
Policy gradient methods have enjoyed great success in deep reinforcement learning but suffer from high variance of gradient estimates. The high variance problem is particularly exasperated in problems with long horizons or high-dimensional action spaces. To mitigate this issue, we derive a bias-free action-dependent baseline for variance reduction which fully exploits the structural form of the stochastic policy itself and does not make any additional assumptions about the MDP. We demonstrate and quantify the benefit of the action-dependent baseline through both theoretical analysis as well as numerical results, including an analysis of the suboptimality of the optimal state-dependent baseline. The result is a computationally efficient policy gradient algorithm, which scales to high-dimensional control problems, as demonstrated by a synthetic 2000-dimensional target matching task. Our experimental results indicate that action-dependent baselines allow for faster learning on standard reinforcement learning benchmarks and high-dimensional hand manipulation and synthetic tasks. Finally, we show that the general idea of including additional information in baselines for improved variance reduction can be extended to partially observed and multi-agent tasks.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 In this paper, the authors investigate variance reduction techniques for agents with multi-dimensional policy outputs, in particular when they are conditionally independent (*factored*). With the increasing focus on applying RL methods to continuous control problems and RTS type games, this is an important problem and this technique seems like an important addition to the RL toolbox. The paper is well written, the method is easy to implement, and the algorithm seems to have clear positive impact on the presented experiments. - The derivations in pages 4-6 are somewhat disconnected from the rest of the paper: the optimal baseline derivation is very standard (even if adapted to the slightly different situation situated here), and for reasons highlighted by the authors in this paper, they are not often used; the *marginalized* baseline is more common, and indeed, the authors adopt this one as well. In light of this (and of the paper being quite a bit over the page limit)- is this material (4.2->4.4) mostly not better suited for the appendix? Same for section 4.6 (which I believe is not used in the experiments). - The experimental section is very strong; regarding the partial observability experiments, assuming actions are here factored as well, I could see four baselines (two choices for whether the baseline has access to the goal location or not, and two choices for whether the baseline has access to the vector ). It*s not clear which two baselines are depicted in 5b - is it possible to disentangle the effect of providing and the location of the hole to the baseline? (side note: it is an interesting idea to include information not available to the agent as input to the baseline though it does feel a bit *iffy* ; the agent requires information to train, but is not provided the information to act. Out of curiosity, is it intended as an experiment to verify the need for better baselines? Or as a *fair* training procedure?) - Minor: in equation 2- is the correct exponent not t*? Also since is define with a scaling (to make it an actual distribution), I believe the definition of should also be multiplied by (as well as equation 2).
assistant
 In this paper, the authors investigate variance reduction techniques for agents with multi-dimensional policy outputs, in particular when they are conditionally independent (*factored*). With the increasing focus on applying RL methods to continuous control problems and RTS type games, this is an important problem and this technique seems like an important addition to the RL toolbox. The paper is well written, the method is easy to implement, and the algorithm seems to have clear positive impact on the presented experiments. - The derivations in pages 4-6 are somewhat disconnected from the rest of the paper: the optimal baseline derivation is very standard (even if adapted to the slightly different situation situated here), and for reasons highlighted by the authors in this paper, they are not often used; the *marginalized* baseline is more common, and indeed, the authors adopt this one as well. In light of this (and of the paper being quite a bit over the page limit)- is this material (4.2->4.4) mostly not better suited for the appendix? Same for section 4.6 (which I believe is not used in the experiments). - The experimental section is very strong; regarding the partial observability experiments, assuming actions are here factored as well, I could see four baselines (two choices for whether the baseline has access to the goal location or not, and two choices for whether the baseline has access to the vector ). It*s not clear which two baselines are depicted in 5b - is it possible to disentangle the effect of providing and the location of the hole to the baseline? (side note: it is an interesting idea to include information not available to the agent as input to the baseline though it does feel a bit *iffy* ; the agent requires information to train, but is not provided the information to act. Out of curiosity, is it intended as an experiment to verify the need for better baselines? Or as a *fair* training procedure?) - Minor: in equation 2- is the correct exponent not t*? Also since is define with a scaling (to make it an actual distribution), I believe the definition of should also be multiplied by (as well as equation 2).