Backpropagation through the Void: Optimizing control variates for black-box gradient estimation
Gradient-based optimization is the foundation of deep learning and reinforcement learning. Even when the mechanism being optimized is unknown or not differentiable, optimization using high-variance or biased gradient estimates is still often the best strategy. We introduce a general framework for learning low-variance, unbiased gradient estimators for black-box functions of random variables, based on gradients of a learned function. These estimators can be jointly trained with model parameters or policies, and are applicable in both discrete and continuous settings. We give unbiased, adaptive analogs of state-of-the-art reinforcement learning methods such as advantage actor-critic. We also demonstrate this framework for training discrete latent-variable models.
 The paper considers the problem of choosing the parameters of distribution to maximize an expectation over that distribution. This setting has attracted a huge interest in ML communities (that is related to learning policy in RL as well as variational inference with hidden variables). The paper provides a framework for such optimization, by interestingly combining three standard ways. Given Tucker et al, its contribution is somehow incremental, but I think it is an interesting idea to use neural networks for control variate to handle the case where f is unknown. The main issue of this paper seems to be in limited experimental results; they only showed three quite simple experiments (I guess they need to focus one setting; RL or VAE). Moreover, it would be good to actually show if the variation of g_hat is much smaller than other standard methods. There is missing reference at page 8.
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
Backpropagation through the Void: Optimizing control variates for black-box gradient estimation
Gradient-based optimization is the foundation of deep learning and reinforcement learning. Even when the mechanism being optimized is unknown or not differentiable, optimization using high-variance or biased gradient estimates is still often the best strategy. We introduce a general framework for learning low-variance, unbiased gradient estimators for black-box functions of random variables, based on gradients of a learned function. These estimators can be jointly trained with model parameters or policies, and are applicable in both discrete and continuous settings. We give unbiased, adaptive analogs of state-of-the-art reinforcement learning methods such as advantage actor-critic. We also demonstrate this framework for training discrete latent-variable models.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 The paper considers the problem of choosing the parameters of distribution to maximize an expectation over that distribution. This setting has attracted a huge interest in ML communities (that is related to learning policy in RL as well as variational inference with hidden variables). The paper provides a framework for such optimization, by interestingly combining three standard ways. Given Tucker et al, its contribution is somehow incremental, but I think it is an interesting idea to use neural networks for control variate to handle the case where f is unknown. The main issue of this paper seems to be in limited experimental results; they only showed three quite simple experiments (I guess they need to focus one setting; RL or VAE). Moreover, it would be good to actually show if the variation of g_hat is much smaller than other standard methods. There is missing reference at page 8.
assistant
 The paper considers the problem of choosing the parameters of distribution to maximize an expectation over that distribution. This setting has attracted a huge interest in ML communities (that is related to learning policy in RL as well as variational inference with hidden variables). The paper provides a framework for such optimization, by interestingly combining three standard ways. Given Tucker et al, its contribution is somehow incremental, but I think it is an interesting idea to use neural networks for control variate to handle the case where f is unknown. The main issue of this paper seems to be in limited experimental results; they only showed three quite simple experiments (I guess they need to focus one setting; RL or VAE). Moreover, it would be good to actually show if the variation of g_hat is much smaller than other standard methods. There is missing reference at page 8.