Backpropagation for Energy-Efficient Neuromorphic Computing
Solving real world problems with embedded neural networks requires both training algorithms that achieve high performance and compatible hardware that runs in real time while remaining energy efficient. For the former, deep learning using backpropagation has recently achieved a string of successes across many domains and datasets. For the latter, neuromorphic chips that run spiking neural networks have recently achieved unprecedented energy efficiency. To bring these two advances together, we must first resolve the incompatibility between backpropagation, which uses continuous-output neurons and synaptic weights, and neuromorphic designs, which employ spiking neurons and discrete synapses. Our approach is to treat spikes and discrete synapses as continuous probabilities, which allows training the network using standard backpropagation. The trained network naturally maps to neuromorphic hardware by sampling the probabilities to create one or more networks, which are merged using ensemble averaging. To demonstrate, we trained a sparsely connected network that runs on the TrueNorth chip using the MNIST dataset. With a high performance network (ensemble of 64), we achieve 99.42% accuracy at 108 µJ per image, and with a high efficiency network (ensemble of 1) we achieve 92.7% accuracy at 0.268 µJ per image.
The authors present an interesting paper about a backpropagation training method using spike probabilities and takes into account the hardware constraints of a particular platform which has spiking neurons and discrete synapses. This is a topic of current interest in mapping deep networks to multi-neuron hardware platforms.Quality: The proposed training method is useful especially in consideration of new multi-neuron hardware platforms with constraints. Clarity: The paper is easy to read. The claims made at the end of Section 1 can be reworded because as it stands, if not read carefully, suggests that the paper proposes for the first time a training methodology that employs spiking neurons, synapses with reduced precision. I would not necessarily put the demonstration of running this network on a specific hardware platform, in this case TN, as a novel feature to be included in this case. Running the network on TN is a validation of the training method.In Section 2, what does it mean for 0.15 bits per synapse? The network topology is not clearly described in one place. How much longer is the training time because of the probabilistic synaptic connection update? Why is the result from a single ensemble of the 30 core network so low especially when compared to the 5 core network? Why are the results of the different ensembles of the 5 core network about the same? What are the spike rates for the inputs during testing? Is the input a spike train? Ref 12 on line 399 includes constraints (e.g. bias=0) during training so it is not just a train-then-constrain alone approach (typo on this line, "approach approach").Originality: Can the authors provide a discussion or comparison with other spiking backprop-type rules like SpikeProp? Significance: The development of new training methods that consider constraints of hardware platforms is of great interest, The constraints which are considered during training in this paper are based on the TN architecture, these constraints might not all apply to other platforms. If the results are based on a 64 ensemble, that means more hardware resources have to be dedicated to obtain the accuracy needed of the network.