THE HIGH-DIMENSIONAL GEOMETRY OF BINARY NEURAL NETWORKS
Recent research has shown that one can train a neural network with binary weights and activations at train time by augmenting the weights with a high-precision continuous latent variable that accumulates small changes from stochastic gradient descent. However, there is a dearth of work to explain why one can effectively capture the features in data with binary weights and activations. Our main result is that the neural networks with binary weights and activations trained using the method of Courbariaux, Hubara et al. (2016) work because of the high-dimensional geometry of binary vectors. In particular, the ideal continuous vectors that extract out features in the intermediate representations of these BNNs are well-approximated by binary vectors in the sense that dot products are approximately preserved. Furthermore, the results and analysis used on BNNs are shown to generalize to neural networks with ternary weights and activations. Compared to previous research that demonstrated good classification performance with BNNs, our work explains why these BNNs work in terms of HD geometry. Our theory serves as a foundation for understanding not only BNNs but a variety of methods that seek to compress traditional neural networks. Furthermore, a better understanding of multilayer binary neural networks serves as a starting point for generalizing BNNs to other neural network architectures such as recurrent neural networks.
This paper investigates numerically and theoretically the reasons behind the empirical success of binarized neural networks. Specifically, they observe that:
(1) The angle between continuous vectors sampled from a spherical symmetric distribution and their binarized version is relatively small in high dimensions (proven to be about 37 degrees when the dimension goes to infinity), and this demonstrated empirically to be true for the binarized weight matrices of a convenet.
(2) Except the first layer, the dot product of weights*activations in each layer is highly correlated with the dot product of (binarized weights)*activations in each layer. There is also a strong correlation between (binarized weights)*activations and (binarized weights)*(binarized activations). This is claimed to entail that the continuous weights of the binarized neural net approximate the continuous weights of a non-binarized neural net trained in the same manner.
(3) To correct the issue with the first layer in (2) it is suggested to use a random rotation, or simply use continues weights in that layer.
The first observation is interesting, is explained clearly and convincingly, and is novel to the best of my knowledge.
The second observation is much less clear to me. Specifically,
a.	The author claim that “A sufficient condition for \delta u to be the same in both cases is L’(x = f(u)) ~ L’(x = g(u))”. However, I’m not sure if I see why this is true: in a binarized neural net, u also changes, since the previous layers are also binarized. 
b.	Related to the previous issue, it is not clear to me if in figure 3 and 5, did the authors binarize the activations of that specific layer or all the layers? If it is the first case, I would be interested to know the latter: It is possible that if all layers are binarized, then the differences between the binarized and non-binarized version become more amplified.
c.	For BNNs, where both the weights and activations are binarized, shouldn’t we compare weights*activations to (binarized weights)*(binarized activations)?
d.	To make sure, in figure 4, the permutation of the activations was randomized (independently) for each data sample? If not, then C is not proportional the identity matrix, as claimed in section 5.3.
e.	It is not completely clear to me that batch-normalization takes care of the scale constant (if so, then why did XNOR-NET needed an additional scale constant?), perhaps this should be further clarified. 
The third observation seems less useful to me. Though a random rotation may improve angle preservation in certain cases (as demonstrated in Figure 4), it may hurt classification performance (e.g., distinguishing between 6 and 9 in MNIST). Furthermore, since it uses non-binary operations, it is not clear if this rotation may have some benefits (in terms of resource efficiency) over simply keeping the input layer non-binarized.
To summarize, the first part is interesting and nice, the second part was not clear to me, and the last part does not seem very useful. 
%%% After Author's response %%%
a. My mistake. Perhaps it should be clarified in the text that u are the weights. I thought that g(u) is a forward propagation function, and therefore u is the neural input (i.e., pre-activation).
Following the author's response and revisions, I have raised my grade.