BinaryConnect: Training Deep Neural Networks with binary weights during propagations
Deep Neural Networks (DNN) have achieved state-of-the-art results in a wide range of tasks, with the best results obtained with large training sets and large models. In the past, GPUs enabled these breakthroughs because of their greater computational speed. In the future, faster computation at both training and test time is likely to be crucial for further progress and for consumer applications on low-power devices. As a result, there is much interest in research and development of dedicated hardware for Deep Learning (DL). Binary weights, i.e., weights which are constrained to only two possible values (e.g. -1 or 1), would bring great benefits to specialized DL hardware by replacing many multiply-accumulate operations by simple accumulations, as multipliers are the most space and powerhungry components of the digital implementation of neural networks. We introduce BinaryConnect, a method which consists in training a DNN with binary weights during the forward and backward propagations, while retaining precision of the stored weights in which gradients are accumulated. Like other dropout schemes, we show that BinaryConnect acts as regularizer and we obtain near state-of-the-art results with BinaryConnect on the permutation-invariant MNIST, CIFAR-10 and SVHN.
This paper suggests to modification to the backpropagation algorithm which enables training of multilayer neural networks with binary weights. This is done by binarizing the weights only during the forward and backward propagation phases (while updating their real value stored). This should accelerate training speed, and much more significantly, testing speed.The fact that one can use binary weights during training, and still get good test performance, is novel and rather surprising (I had to check the code to believe this). The (minor) comments I have are about the relation to previous works: 1. It is not clear, why on line 373, it is said that the comparisons were done on different datasets. Indeed ref [29] is done for text classification, but doesn't ref [30] use the same regular (permutation invariant) mnist as in this paper? If that is the case, I see several possible explanations to the differences in results. First, EBP didn't use the tricks used in this paper (L2-SVM output, batch normalization, and adjustable/decaying learning rate), except dropout. Increasing the margin (using L2-svm) can be especially useful for decreasing the discretization sensitivity. Second, larger networks and many more epochs were used here. Third, in EBP also the neurons are binary (which is more hardware friendly then the hard sigmoid used here).2. Lines 369-371 are slightly inaccurate. In EBP [29,30]: it is also possible to use only a single binary network, at the price of reduced performance (as was done in [30]). This is somewhat similar to the decrease of performance in BinaryConnect (det.) over BinaryConnect (stoch.). Also, there a few typos and grammatical error ("Let us compared..."). In summary, very interesting paper!%%%Edit, after Author's Feedback: I raised the score from 8 to 9 due to the new experimental results, which seem quite promising. I mostly like very much the fact that you can train with binary weights, in contrast to previous works. Note that is FPGA section should be improved. It should be made more concrete (showing at least a diagram how the weights are placed and how the data are routed through the network. Specifically, how to route the convolutional layers) or removed.