UNDERSTANDING STRAIGHT-THROUGH ESTIMATOR IN TRAINING ACTIVATION QUANTIZED NEURAL NETS
Training activation quantized neural networks involves minimizing a piecewise constant function whose gradient vanishes almost everywhere, which is undesirable for the standard back-propagation or chain rule. An empirical way around this issue is to use a straight-through estimator (STE) (Bengio et al., 2013)  in the backward pass only, so that the "gradient" through the modified chain rule becomes non-trivial. Since this unusual "gradient" is certainly not the gradient of loss function, the following question arises: why searching in its negative direction minimizes the training loss? In this paper, we provide the theoretical justification of the concept of STE by answering this question. We consider the problem of learning a two-linear-layer network with binarized ReLU activation and Gaussian input data. We shall refer to the unusual "gradient" given by the STE-modifed chain rule as coarse gradient. The choice of STE is not unique. We prove that if the STE is properly chosen, the expected coarse gradient correlates positively with the population gradient (not available for the training), and its negation is a descent direction for minimizing the population loss. We further show the associated coarse gradient descent algorithm converges to a critical point of the population loss minimization problem. Moreover, we show that a poor choice of STE leads to instability of the training algorithm near certain local minima, which is verified with CIFAR-10 experiments.
Summary:
The paper presents an analysis of training single-layer hard-threshold (binary activation) networks for regression with a mean-squared loss function using two different straight-through estimators: the original identity-function STE and a ReLU-based STE. The paper demonstrates that training with the latter against the population loss is guaranteed to converge to a critical point, whereas using the former can cause instability in the training.
    Pros:
        - Interesting analysis that provides a novel method for determining which gradient estimators are effective for training single-layer binarized networks and which are not.
        - The paper is fairly clear, despite being quite technical; however, I did find myself jumping around a lot to refer back to previous results or definitions so the ordering and layout could definitely be improved.
    Cons:
        - Related work is missing and some claims in the paper are wrong as a result.
        - A single-layer binarized network is essentially just a perceptron, which we know how to learn already, so it’s not clear how this analysis will benefit analysis of multi-layer binarized networks (however, since it seems like a novel analysis approach, it’s possible that it can be extended). This connection is not made in the paper.
        - The paper does not analyze the most common and successful straight-through estimator: the saturated straight-through estimator, which uses the derivative of the hard_tanh activation (e.g., see [2]) and is a shifted and scaled version of the clipped ReLU STE.
Overall, I like the paper but it has too many issues currently for me to give it a high score. However, if my questions and comments are addressed sufficiently, I would be happy to improve my score.
Detailed questions and comments:
1.	The claim that “we make the first theoretical justification for the concept of STE” is wrong and should be significantly toned down and clarified. Bengio et al. (2013), which this paper cites, provides some theoretical justification already, as do papers on target propagation, such as [1] and [2]. These, as well as additional papers cited in [2] are quite relevant and should also be cited and discussed.
2.	The claim that “it is not the gradient of any function” is also wrong. Each STE is the gradient of a particular function, but is not the gradient of the function used in the forward pass. Please clarify.
3.	The single-layer binarized network architecture studied in this paper can equivalently be framed as a linear function of a collection of single-layer perceptrons with shared weights. Obviously, much work has been done on analyzing the perceptron architecture. Why is none of it discussed in this paper? How does that work relate to the work done in this paper? How does the convolutional layer used here change the results of that related work? 
4.	(a) Is there an intuition for why the derivative of the ReLU performs better (i.e., converges) better than the identity? Why does clipping the bottom make it work better? I do not see this explained in the text anywhere and it would be helpful to include this. 
(b) Further, depending on the reasoning given, it seems that clipping the top may also be useful (as in the clipped ReLU, which is a shifted and scaled version of the saturated STE discussed in Hubara et al. and [2]). Does your analysis extend to this STE? This would be very useful, as the SSTE/clipped ReLU is the most commonly used STE and the most empirically successful (as validated by your own experiments, as well as in previous work on training binary networks). Also, the SSTE/clipped ReLU is an even better approximation of the step function.
(c) Cai et al. (2017) is not the first use of the clipped ReLU activation function, since it is equivalent to the SSTE when using sign(x) \in {-1, +1} instead of your activation function (\sigma(x) \in {0, 1}) (i.e., you can shift and scale everything to get equivalent results).
5.	In section 3.1, you mention that when using the derivative of the ReLU for the STE then \mu`(x) = \sigma(x). Is this just a coincidence or does this fact help with convergence?
6.	Why did you choose to train your networks initialized with the weights from their full-precision counterparts? When you train using different initializations, does this significantly affect your results?
7.	The improved empirical performance of the clipped ReLU / SSTE is unsurprising but why does the vanilla ReLU STE perform so poorly on CIFAR-10 with ResNet-20 with 2 bit quantization?
8.	In the end, it’s not clear that training single-layer hard-threshold networks is particularly important. Instead, the goal of quantization, etc. is to train multi-layer hard-threshold networks. Can this analysis be extended to such networks? Does it say anything about training such networks currently?
9.	The acknowledgments section is just the text from the style file.
10.	The capitalization is wrong in a number of places in your references.
[1] Difference Target Propagation. Lee, Zhang, Fischer, and Bengio. ECML/PKDD (2015).
[2] Deep Learning as a Mixed Convex-Combinatorial Optimization Problem. Friesen and Domingos. ICLR (2018).
------------------------
After reading the author response, they have sufficiently addressed my main concerns. I think that this is a good paper that will be of interest to those concerned with understanding the training of activation-quantized / hard-threshold neural networks. I have thus increased my score.