ARM: Augment-REINFORCE-Merge Gradient for Stochastic Binary Networks
To backpropagate the gradients through stochastic binary layers, we propose the augment-REINFORCE-merge (ARM) estimator that is unbiased, exhibits low variance, and has low computational complexity. Exploiting variable augmentation, REINFORCE, and reparameterization, the ARM estimator achieves adaptive variance reduction for Monte Carlo integration by merging two expectations via common random numbers. The variance-reduction mechanism of the ARM estimator can also be attributed to either antithetic sampling in an augmented space, or the use of an optimal anti-symmetric "self-control" baseline function together with the REINFORCE estimator in that augmented space. Experimental results show the ARM estimator provides state-of-the-art performance in auto-encoding variational inference and maximum likelihood estimation, for discrete latent variable models with one or multiple stochastic binary layers. Python code for reproducible research is publicly available.
 In this paper the authors propose a new variance-reduction technique to use when computing an expected loss gradient where the expectation is with respect to independent binary random variables, e.g. for training VAEs with a discrete latent space. The paper is interesting, highly relevant, simple to implement, suggests many possible extensions, and shows good results on the experiments performed. However the exposition leaves a lot to be desired. Major comments: The authors devote several pages of fairly dense mathematics to deriving the ARM estimate in section 2 (up to section 2.5). However I found it relatively easy to derive (15) directly, using elementary results such as the law of total expectation and a single 1-dimensional integral, in about 10 lines of equations. As the authors note, deriving (4) from (15) requires an extra line or two. In my opinion it would greatly improve the clarity of the paper to use a more direct and straightforward derivation (perhaps with the interesting historical account of how the authors first derived this result given in an appendix). I could understand the more lengthy derivation being helpful if it gave insight into the source of variance reduction, but I don*t see this personally, and the current discussion of variance reduction does not refer to the derivation of (15) at all. The analysis of variance in section 2.6 leaves a lot to be desired. The central claim of the paper is that this method reduces variance, so it is an important section! Firstly, the variance of ARM vs AR is interesting, but the variance of ARM vs REINFORCE seems also highly relevant. Secondly, it seems like it would be very informative to look at the ratio of stdev to the mean for the ARM gradient estimate, since the true gradient is multiplied by sigmoid(phi) sigmoid(-phi) and so is very small if the probability of z = 1 is close to 0 or 1, exactly in the same regime where ARM has an advantage in variance reduction over AR. For example, it may be that learning in this regime is very difficult due to the weak gradient even if the estimate is extremely low variance. Thirdly and somewhat relatedly, in this same regime (z = 1 close to 0 or 1) the ARM gradient estimate is very often 0, meaning no learning takes place, so it seems a bit strange to argue that the new method is fantastic in the regime where it*s almost always not learning! Of course, not learning is better than adding lots of spurious variance as reinforce would, but perhaps this could be made clearer. Finally, the theoretical analysis involving correlation gives very little insight and is extremely hand-wavy. A short worked example in the 1D or 2D case explicitly computing the variance of REINFORCE, AR and ARM seems like it would be highly informative. Minor comments: In the introduction, "*approximately* maximizing the marginal likelihood" might be more accurate, since as given in (28) the exact marginal likelihood is not optimized in practice, and the exact marginal likelihood is not of the form (1) but is rather the logarithm of something of the form (1). I wasn*t clear why "equal in distribution" was used a few things for things that are simply equal, such as just above (5). In section 2.3, I don*t see any real reason the estimates in (9) and (11) "could be highly positively correlated", other than an argument along the lines of the simple one given in section 2.6 that they*re often equal and so zero. As an aside, in section 3.1, it is great not to assume conditional independence of the binary latent variables across layers, but assuming conditional independence within each layer is still very restrictive. It is reasonable for the generative distribution to have this property, since the resulting net can still be essentially "universal" by stacking enough layers, but assuming this factorization in the variational distribution is highly restrictive with hard-to-reason-about consequences for the learned generative model. I realize this is a commonly used assumption and the authors are interested in the variance reduction properties of their approach rather than the training itself, but I just mention that it would be great to see extensions of the current work that can cope tractably with correlated latent variables within each layer. In section 3.2, according to my understanding of standard terminology, "maximum likelihood inference" is a misnomer and would normally be "maximum likelihood estimation", since maximum likelihood is a method for estimating parameters whereas inference is about inferring latent variable values given parameters. In section 4, it would be great to see some plots of explicit variance estimates of the different methods, given the overall goal of the paper (unless I just missed this?), even though figure 1 gives some insight into the variance characteristics. In section 4.2, the expression log 1/K .sum_k Bernoulli... differs in the placement of log from Jang et al (2017). Which is the standard convention for this task?
You are a member of the scientific community tasked with peer review. Review the following paper title and abstract.
### Title + Abstract
ARM: Augment-REINFORCE-Merge Gradient for Stochastic Binary Networks
To backpropagate the gradients through stochastic binary layers, we propose the augment-REINFORCE-merge (ARM) estimator that is unbiased, exhibits low variance, and has low computational complexity. Exploiting variable augmentation, REINFORCE, and reparameterization, the ARM estimator achieves adaptive variance reduction for Monte Carlo integration by merging two expectations via common random numbers. The variance-reduction mechanism of the ARM estimator can also be attributed to either antithetic sampling in an augmented space, or the use of an optimal anti-symmetric "self-control" baseline function together with the REINFORCE estimator in that augmented space. Experimental results show the ARM estimator provides state-of-the-art performance in auto-encoding variational inference and maximum likelihood estimation, for discrete latent variable models with one or multiple stochastic binary layers. Python code for reproducible research is publicly available.
user
<think>Okay, to do a decent peer review, I should start with summarizing what the work is about. This includes novelty, contributions, clarity, and impact. I might include suggestions to issues that I find critical to the usefulness of the work. Starting with novelty, there are different types of novelty. The first aspect of novelty is scope and scale. Is the work covering a broader perspective than previous works? The second aspect is methodology. Is the proposed methodology novel compared to previous methods? Do they provide any significant advantage over previous methodologies? The third aspect is results. Are the results different compared previous work? Does this lead to any novel theories or an extension of current theories? Next is clarity, I will be reading the work in depth and analyze if any sentences can be potentially ambiguous or misleading. If any sentences are ambiguous, I will state them and give a suggestion on how to rephrase such sentence or how to make more clear to the audience. Lastly, impact is important. I will attempt to predict how such work will impact the scientific community. This might include the applications of such work and how the work might transform the way we study a certain topic. Finally, I will might only mention issues that are critical. This is to keep the review as concise and helpful as possible.</think>



 In this paper the authors propose a new variance-reduction technique to use when computing an expected loss gradient where the expectation is with respect to independent binary random variables, e.g. for training VAEs with a discrete latent space. The paper is interesting, highly relevant, simple to implement, suggests many possible extensions, and shows good results on the experiments performed. However the exposition leaves a lot to be desired. Major comments: The authors devote several pages of fairly dense mathematics to deriving the ARM estimate in section 2 (up to section 2.5). However I found it relatively easy to derive (15) directly, using elementary results such as the law of total expectation and a single 1-dimensional integral, in about 10 lines of equations. As the authors note, deriving (4) from (15) requires an extra line or two. In my opinion it would greatly improve the clarity of the paper to use a more direct and straightforward derivation (perhaps with the interesting historical account of how the authors first derived this result given in an appendix). I could understand the more lengthy derivation being helpful if it gave insight into the source of variance reduction, but I don*t see this personally, and the current discussion of variance reduction does not refer to the derivation of (15) at all. The analysis of variance in section 2.6 leaves a lot to be desired. The central claim of the paper is that this method reduces variance, so it is an important section! Firstly, the variance of ARM vs AR is interesting, but the variance of ARM vs REINFORCE seems also highly relevant. Secondly, it seems like it would be very informative to look at the ratio of stdev to the mean for the ARM gradient estimate, since the true gradient is multiplied by sigmoid(phi) sigmoid(-phi) and so is very small if the probability of z = 1 is close to 0 or 1, exactly in the same regime where ARM has an advantage in variance reduction over AR. For example, it may be that learning in this regime is very difficult due to the weak gradient even if the estimate is extremely low variance. Thirdly and somewhat relatedly, in this same regime (z = 1 close to 0 or 1) the ARM gradient estimate is very often 0, meaning no learning takes place, so it seems a bit strange to argue that the new method is fantastic in the regime where it*s almost always not learning! Of course, not learning is better than adding lots of spurious variance as reinforce would, but perhaps this could be made clearer. Finally, the theoretical analysis involving correlation gives very little insight and is extremely hand-wavy. A short worked example in the 1D or 2D case explicitly computing the variance of REINFORCE, AR and ARM seems like it would be highly informative. Minor comments: In the introduction, "*approximately* maximizing the marginal likelihood" might be more accurate, since as given in (28) the exact marginal likelihood is not optimized in practice, and the exact marginal likelihood is not of the form (1) but is rather the logarithm of something of the form (1). I wasn*t clear why "equal in distribution" was used a few things for things that are simply equal, such as just above (5). In section 2.3, I don*t see any real reason the estimates in (9) and (11) "could be highly positively correlated", other than an argument along the lines of the simple one given in section 2.6 that they*re often equal and so zero. As an aside, in section 3.1, it is great not to assume conditional independence of the binary latent variables across layers, but assuming conditional independence within each layer is still very restrictive. It is reasonable for the generative distribution to have this property, since the resulting net can still be essentially "universal" by stacking enough layers, but assuming this factorization in the variational distribution is highly restrictive with hard-to-reason-about consequences for the learned generative model. I realize this is a commonly used assumption and the authors are interested in the variance reduction properties of their approach rather than the training itself, but I just mention that it would be great to see extensions of the current work that can cope tractably with correlated latent variables within each layer. In section 3.2, according to my understanding of standard terminology, "maximum likelihood inference" is a misnomer and would normally be "maximum likelihood estimation", since maximum likelihood is a method for estimating parameters whereas inference is about inferring latent variable values given parameters. In section 4, it would be great to see some plots of explicit variance estimates of the different methods, given the overall goal of the paper (unless I just missed this?), even though figure 1 gives some insight into the variance characteristics. In section 4.2, the expression log 1/K .sum_k Bernoulli... differs in the placement of log from Jang et al (2017). Which is the standard convention for this task?
assistant
 In this paper the authors propose a new variance-reduction technique to use when computing an expected loss gradient where the expectation is with respect to independent binary random variables, e.g. for training VAEs with a discrete latent space. The paper is interesting, highly relevant, simple to implement, suggests many possible extensions, and shows good results on the experiments performed. However the exposition leaves a lot to be desired. Major comments: The authors devote several pages of fairly dense mathematics to deriving the ARM estimate in section 2 (up to section 2.5). However I found it relatively easy to derive (15) directly, using elementary results such as the law of total expectation and a single 1-dimensional integral, in about 10 lines of equations. As the authors note, deriving (4) from (15) requires an extra line or two. In my opinion it would greatly improve the clarity of the paper to use a more direct and straightforward derivation (perhaps with the interesting historical account of how the authors first derived this result given in an appendix). I could understand the more lengthy derivation being helpful if it gave insight into the source of variance reduction, but I don*t see this personally, and the current discussion of variance reduction does not refer to the derivation of (15) at all. The analysis of variance in section 2.6 leaves a lot to be desired. The central claim of the paper is that this method reduces variance, so it is an important section! Firstly, the variance of ARM vs AR is interesting, but the variance of ARM vs REINFORCE seems also highly relevant. Secondly, it seems like it would be very informative to look at the ratio of stdev to the mean for the ARM gradient estimate, since the true gradient is multiplied by sigmoid(phi) sigmoid(-phi) and so is very small if the probability of z = 1 is close to 0 or 1, exactly in the same regime where ARM has an advantage in variance reduction over AR. For example, it may be that learning in this regime is very difficult due to the weak gradient even if the estimate is extremely low variance. Thirdly and somewhat relatedly, in this same regime (z = 1 close to 0 or 1) the ARM gradient estimate is very often 0, meaning no learning takes place, so it seems a bit strange to argue that the new method is fantastic in the regime where it*s almost always not learning! Of course, not learning is better than adding lots of spurious variance as reinforce would, but perhaps this could be made clearer. Finally, the theoretical analysis involving correlation gives very little insight and is extremely hand-wavy. A short worked example in the 1D or 2D case explicitly computing the variance of REINFORCE, AR and ARM seems like it would be highly informative. Minor comments: In the introduction, "*approximately* maximizing the marginal likelihood" might be more accurate, since as given in (28) the exact marginal likelihood is not optimized in practice, and the exact marginal likelihood is not of the form (1) but is rather the logarithm of something of the form (1). I wasn*t clear why "equal in distribution" was used a few things for things that are simply equal, such as just above (5). In section 2.3, I don*t see any real reason the estimates in (9) and (11) "could be highly positively correlated", other than an argument along the lines of the simple one given in section 2.6 that they*re often equal and so zero. As an aside, in section 3.1, it is great not to assume conditional independence of the binary latent variables across layers, but assuming conditional independence within each layer is still very restrictive. It is reasonable for the generative distribution to have this property, since the resulting net can still be essentially "universal" by stacking enough layers, but assuming this factorization in the variational distribution is highly restrictive with hard-to-reason-about consequences for the learned generative model. I realize this is a commonly used assumption and the authors are interested in the variance reduction properties of their approach rather than the training itself, but I just mention that it would be great to see extensions of the current work that can cope tractably with correlated latent variables within each layer. In section 3.2, according to my understanding of standard terminology, "maximum likelihood inference" is a misnomer and would normally be "maximum likelihood estimation", since maximum likelihood is a method for estimating parameters whereas inference is about inferring latent variable values given parameters. In section 4, it would be great to see some plots of explicit variance estimates of the different methods, given the overall goal of the paper (unless I just missed this? ), even though figure 1 gives some insight into the variance characteristics. In section 4.2, the expression log 1/K .sum_k Bernoulli... differs in the placement of log from Jang et al (2017). Which is the standard convention for this task?