SLAYER: Spike Layer Error Reassignment in Time
Configuring deep Spiking Neural Networks (SNNs) is an exciting research avenue for low power spike event based computation. However, the spike generation function is non-differentiable and therefore not directly compatible with the standard error backpropagation algorithm. In this paper, we introduce a new general backpropagation mechanism for learning synaptic weights and axonal delays which overcomes the problem of non-differentiability of the spike function and uses a temporal credit assignment policy for backpropagating error to preceding layers. We describe and release a GPU accelerated software implementation of our method which allows training both fully connected and convolutional neural network (CNN) architectures. Using our software, we compare our method against existing SNN based learning approaches and standard ANN to SNN conversion techniques and show that our method achieves state of the art performance for an SNN on the MNIST, NMNIST, DVS Gesture, and TIDIGITS datasets.
Event based methods such as SpikeProp [10] and EvSpikeProp [11] have the derivative term defined only around the firing time, whereas [12][13][14] ignore the temporal effect of spike signal. In Section 3.1 we describe the strengths and weaknesses of these approaches in more detail.
The main contribution of this paper is a general method of error backpropagation for SNNs (Section 3) which we call Spike LAYer Error Reassignment (SLAYER). SLAYER distributes the credit of error back through the SNN layers, much like the traditional backprop algorithm distributes error back through an ANN's layers. However, unlike backprop, SLAYER also distributes the credit of error back in time because a spiking neuron's current state depends on its previous states (and therefore, on the previous states of its input neurons). SLAYER can simultaneously learn both synaptic weights and axonal delays, which only a few previous works have attempted [15,16].
We have developed and released 3 a CUDA accelerated framework to train SNNs using SLAYER. We demonstrate SLAYER achieving state of the art accuracy for an SNN on neuromorphic datasets (Section 4) for visual digit recognition, action recognition, and spoken digit recognition.
The rest of the paper is organized as follows. We start by introducing notation for a general model of a spiking neuron and extending it to a multi-layer SNN in Section 2. Then, in Section 3 we discuss previously published methods for learning SNN parameters before deriving the SLAYER backpropagation formulae. In Section 4, we demonstrate the effectiveness of SLAYER on different benchmark datasets before concluding in Section 5.
This work contributes a new error backpropagation method for spiking neural networks which is capable of learning both weight and axonal delay parameters in a spiking neural network, and establishes a new SOTA on multiple tasks.  This work helps address the long-held challenge in the neuromorphic community of designing a learning algorithm capable of learning (approximately) as well as standard deep neural networks for fixed inputs, but is also capable of learning from time-varying signals which more closely match the native representation of spiking neural networks.  This submission was prepared carefully and clearly, with a thorough set of comparison benchmarks to other standard spiking benchmarks.  The benchmarks further span both image and audio challenges.  Recent work is compared against and referenced, and CNNs and fully-connected architectures similar to those used in the state-of-the-art are used.  The derivation of the modified backpropagation rule is well-described.  Further, a GPU-optimized toolbox will also be released, all bolstering the work's quality, clarity, and significance.  Discretizing temporal history to enable backpropagation and approximating a Dirac function with a more easily-differentiable function is not an entirely surprising approach, but it is used effectively here.  It would be useful to see the rule tested on more difficult temporal data than TIDIGITS, in which the SOM on top of MFCCs provides significant precomputation; similarly, a more thorough examination of learning axonal delays would improve the motivation for adding the additional complexity.  For example, do learned delays in NMNIST correspond to saccade timing, or just play a role in expanding the parameter space for learning?  Finally, some of the choices, e.g. spike rates for true vs. negative classes seem arbitrary.  Why were 180 spikes/sec used in DVS gesture classification for true and 30 spikes/sec used for false classes? The work could be improved by comparing against ANNs on spiking data, and referencing works such as [1] that featurize on top of neuromorphic data to suggest whether the limited performance is now more from the SNN models or the input streams themselves.  However these are small points.  Overall, this work has the potential to strongly impact researchers working in neuromorphic models.  [1] Anumula, Jithendar, et al. “Feature Representations for Neuromorphic Audio Spike Streams.” Frontiers in Neuroscience 12 (2018): 23. https://doi.org/10.3389/fnins.2018.00023.