) CIFAR Senior Fellow. (*) Indicates equal contribution
We introduce a method to train Binarized Neural Networks (BNNs) -neural networks with binary weights and activations at run-time. At train-time the binary weights and activations are used for computing the parameter gradients. During the forward pass, BNNs drastically reduce memory size and accesses, and replace most arithmetic operations with bit-wise operations, which is expected to substantially improve power-efficiency. To validate the effectiveness of BNNs, we conducted two sets of experiments on the Torch7 and Theano frameworks. On both, BNNs achieved nearly state-of-the-art results over the MNIST, CIFAR-10 and SVHN datasets. We also report our preliminary results on the challenging ImageNet dataset. Last but not least, we wrote a binary matrix multiplication GPU kernel with which it is possible to run our MNIST BNN 7 times faster than with an unoptimized GPU kernel, without suffering any loss in classification accuracy. The code for training and running our BNNs is available on-line.
The paper presents a method for training neural networks with binary weights and activations. It is essentially based on back-propagation where weights and activations are binarized during the forward pass, but the backward pass uses real-valued gradients and weight updates. The networks use batch normalisation and are optimised with Adam or AdaMax, which are also speeded-up by using binary shifts instead of multiplications. The framework is evaluated on MNIST, SVHN, CIFAR-10, and ImageNet where it performs worse than real-valued nets, but still demonstrates reasonable results. The authors also present the benchmarks of the GPU (CUDA) implementation of the binary matrix multiplication, which achieves a substantial speed-up over real-valued matrix multiplication.
Positives:  1) The topic of speeding-up training and evaluation of deep neural nets is very relevant, and the approach of moving the computationally intensive operations from real values to binary looks promising.  2) The method is evaluated on several datasets (including large-scale ImageNet).  Negatives:  1) The contribution is rather incremental, the main difference with BinaryConnect is that the activations are also binarized, which requires a rather minor change in the algorithm.  2) Experimental evaluation is carried out using two different frameworks (Torch and Theano), with different networks, optimisers, binarization schemes, etc. As a result, the experimental section does not look cohesive and is hard to analyse or refer to in future work. Instead of providing the recipes for training BNNs, it confuses the reader.  3) The real-valued baselines for the same networks and optimisers are missing (apart from the ImageNet experiments). This is probably the most important question the experimental evaluation should answer -- how big is the performance gap between real-valued and binary nets, all other things being equal. Such baselines are only presented for ImageNet, where the gap is rather large, e.g. 71%->47.1% for GoogLeNet. It would be useful to see similar comparisons for other datasets.  Questions:  1) For back-propagation through binarized activations, it is proposed to clip the gradients (eq. 5), but this approach is not used for back-propagation w.r.t. the binarized weights. Instead, it is effectively assumed that the gradient w.r.t. real-valued weights is the same as the gradient w.r.t. the binary weights g_{W_k^b}, but then there is clipping after the weight update. Why is back-propagation through binarized weights and activations done differently?  Summary:  Binary networks are a promising direction, and in principle the paper is interesting enough to be presented at NIPS, but the experimental evaluation does require improvement/re-writing. I would recommend the paper for borderline acceptance (conditioned on the authors improving the experimental section).