This paper shows how to train binary networks to within a few percent points (∼ 3 − 5%) of the full precision counterpart with a negligible increase in the computational cost. In particular, we first show how to build a strong baseline, which already achieves state-of-the-art accuracy, by combining recently proposed advances, and carefully tuning the optimization procedure. Secondly, we show that by attempting to minimize the discrepancy between the output of the binary and the corresponding real-valued convolution additional significant accuracy gains can be obtained. We materialize this idea in two complementary ways: (1) with a loss function, during training, by matching the spatial attention maps computed at the output of the binary and real-valued convolutions, and (2) in data-driven manner, by using the real-valued activations being available during inference prior to the binarization process for re-scaling the activations right after the binary convolution. Finally, we show that, when putting all of our improvements together, the resulting model reduces the gap to its real-valued counterpart to less than 3% and 5% top-1 error on CIFAR-100 and ImageNet, respectively, when using a ResNet-18 architecture.
This paper greatly reduces the gap between binarized and real valued imagenet, using a variety of techniques. The most significant contributions of this paper are engineering based, and the careful combination and integration of approaches from previous papers. I believe that this is of significant practical importance to the field. I particularly appreciate the effort put into developing a very strong baseline that combined ideas from many previous papers!
My biggest concern is that ResNet is itself a very wasteful architecture in terms of compute and parameter count. If the goal is to develop a compute- and memory-efficient architecture, it would be good to also consider real-valued-network baselines that were proposed with computational and/or memory efficiency as a design goal.
Additionally, the specific choices for the new student-teacher loss, and new scaling network architecture, seem fairly ad-hoc.
Detailed comments:
"this implies a reduction of 32× in memory usage" , assuming the parameter count is held constant.
Fig 1 right: This is motivated in terms of preserving scaling factors that are lost by the binarization, but the functional form for this makes it look a lot like a learned gating operation. If the sigmoid is dropped from the architecture, does performance worsen? It would be nice to see some discussion of the degree to which this is helpful because it reverses information loss due to binarization, vs. introduces a new architectural feature which is itself helpful.
Add a sentence describing what "double skip connections" are. I wasn't familiar with this phrase.
eq. 2:
This functional form is pretty weird.
Why is Q a square norm rather than a norm? Square error on an already-squared property is an unusual choice.
Why is the denominator itself a norm? Taking the norm of a square norm is similarly an unusual choice. (eg, why not just take an average or sum over Q) 
Say what Q_s and Q_t are (student and teacher network from context)
"thus, at test time, these scaling factors will not be fixed but rather inferred from data" nit: Would not generally call this an inference process. "Inference" typically refers to values that are computed indirectly (eg by Bayesian reasoning), while in this case the values are computed directly. Would rather say that scaling factors are a function of data, or are determined by data, or similar.
"By doing so, more than 1/3 of the remaining gap with the real-valued network is bridged." text is shifting back and forth between using % and fractional gap to describe benefits. Would just use one measure consistently.
Computational cost analysis:
This is very useful.
Note though that ResNet is a very wasteful architecture in terms of compute! It would be good to include a comparison to imagenet architectures that have computational and memory efficiency as a design goal. (eg, MobileNet comes to mind)
Very nice on the ablation studies.