Hybrid Macro/Micro Level Backpropagation for Training Deep Spiking Neural Networks
Spiking neural networks (SNNs) are positioned to enable spatio-temporal information processing and ultra-low power event-driven neuromorphic hardware. However, SNNs are yet to reach the same performances of conventional deep artificial neural networks (ANNs), a long-standing challenge due to complex dynamics and non-differentiable spike events encountered in training. The existing SNN error backpropagation (BP) methods are limited in terms of scalability, lack of proper handling of spiking discontinuities, and/or mismatch between the ratecoded loss function and computed gradient. We present a hybrid macro/micro level backpropagation (HM2-BP) algorithm for training multi-layer SNNs. The temporal effects are precisely captured by the proposed spike-train level post-synaptic potential (S-PSP) at the microscopic level. The rate-coded errors are defined at the macroscopic level, computed and back-propagated across both macroscopic and microscopic levels. Different from existing BP methods, HM2-BP directly computes the gradient of the rate-coded loss function w.r.t tunable parameters. We evaluate the proposed HM2-BP algorithm by training deep fully connected and convolutional SNNs based on the static MNIST [14] and dynamic neuromorphic N-MNIST [26]. HM2-BP achieves an accuracy level of 99.49% and 98.88% for MNIST and N-MNIST, respectively, outperforming the best reported performances obtained from the existing SNN BP algorithms. Furthermore, the HM2-BP produces the highest accuracies based on SNNs for the EMNIST [3] dataset, and leads to high recognition accuracy for the 16-speaker spoken English letters of TI46 Corpus [16], a challenging spatio-temporal speech recognition benchmark for which no prior success based on SNNs was reported. It also achieves competitive performances surpassing those of conventional deep learning models when dealing with asynchronous spiking streams.
[Response to rebuttal: I am very impressed and greatly appreciate the amount of experiments that the authors managed to successfully perform during the rebuttal phase. The additional results strengthen the empirical part of the paper significantly and further improve the experimental rigor of the paper. I am not entirely sure if I would rate the submission in the top 50%, but I would be somewhat upset if it got rejected. I therefore raise my overall score to an 8. I greatly appreciate the additional datasets and analysis in Fig. 1 of the rebuttal. I also want to encourage the authors to take some of the minor issues that were raised, for instance the quality and usefulness of the figures, into account, but I understand that one page of rebuttal is not sufficient to address all issues and I feel like the authors correctly prioritized addressing the issues raised.]  The paper introduces a novel method for formalizing spiking neurons that allows for spiking neural networks which can be trained via stochastic gradient descent. In contrast to previous approaches to address this topic, the proposed model allows to incorporate temporal spiking dynamics while still maintaining differentiability w.r.t. the weights. This is achieved by formulating a rate-model (the macro-level) where the rate depends in a differentiable fashion on (approximate) leaky-integrate-and-fire dynamics (the micro level). In particular, the dependency of the rate on non-differentiable spikes is replaced by expressing the contribution of pre-synaptic spike trains on post-synaptic firing counts (and firing times) as a continuous function, termed “spike-train level post-synaptic potential” (S-PSP). This in turn allows for deriving closed-form gradient update equations for the weights. The method is evaluated on MNIST (fully connected network and convolutional neural network) and neuromorphic MNIST (fully connected network). On these experiments, the method outperforms competitor methods by a small margin. The widespread use of spiking neural networks in machine learning is arguably hindered by a lack of good training algorithms that scale well to large networks of neurons, particularly for the case of supervised learning. Due to the non-differentiable nature of spiking neurons, gradient descent cannot be applied in a straightforward way and several ways to alleviate this shortcoming have been proposed in the literature. The method presented in the paper addresses a timely and significant problem in an original and novel way that works well in the experiments shown. The writing is clear, with a well-written introduction and enough background for a machine learning practitioner to understand the method without having to resort to textbooks on spiking neuron models. I also greatly appreciate that the paper focuses clearly on machine learning applications and does not give in to the temptation of overloading the paper with additionally claiming biological plausibility (in a hand-wavy fashion). The authors also provide code to reproduce their results and also allowing for easy extension to experiment with additional ideas or set up additional experiments, which is definitely a plus. Besides minor issues, my main criticism is that the authors claim superior performance of the method even though they outperform competitors often by less than 0.2% accuracy on MNIST (comparing a single run with random initialization). While the claim of providing state-of-the-art results is technically not wrong, I would rather claim that the method performs on-par with competitor methods and further experiments (on more challenging datasets) are required for conclusive statements. However, I consider the latter overall a tolerable issue and suggest to accept the paper. My comments below are suggestions on how to further improve the quality and significance of the paper.  (1 Decoupling of micro level) Could the authors clarify their particular choice of Eq. (25)? Are there any obvious alternatives (o_j and o_k have a linear effect, but does it necessarily have to be multiplicative or is this a choice motivated by mathematical convenience)? Can the authors provide some quantitative analysis on how sensitive the approximation is w.r.t. the assumption that \hat{\alpha} does not change substantially (e.g. a histogram of how much \hat{\alpha} actually changes in typical experiments, or some comparison of numerical gradients vs. gradients based on the approximation)? (2 Error bars) Even though it has become widely accepted in the neural network community to report results on single runs, without sensitivity-analysis w.r.t. hyper-parameter settings, I want to actively encourage the community to re-introduce more experimental rigor by setting high standards. Whenever possible “with reasonable effort” it would be great if results were reported with “error bars” (e.g. mean/median accuracy over a small number of runs with an appropriate error-bar measure). While I do not strictly require the authors to re-run multiple instances of their experiments for this publication, I would consider a higher score if error bars were provided. (3 Additional datasets) MNIST (and neuromorphic MNIST) seems to have saturated as a data-set to compare different methods for spiking neural network training. It would be an interesting move to start evaluating on more challenging datasets. FashionMNIST should be really straightforward, but it would also be interesting to see more challenging problems on data-sets with temporal / event-driven signals in the long run. (4 Figures) The clarity of figures, particularly of figure captions could still be improved. Given the limited space of the manuscript you might want to push some parts of the derivation or some details of the experiments to the appendix. -) Fig. 1: there are lots of different kinds of arrows in the figure, a bit more space and text in the figure as well as the caption would help. Also, what does the “(+/-) Multiple Spikes” refer to? -) Fig. 2: What does the shaded yellow area correspond to? Why is there no re-set of the middle curve after the second post-synaptic spike? -) Fig. 4: The figure does not add much more clarity compared to the equations, but I think with a bit of polishing (and more space) it could add value to the paper.  (5 Control exp.) For the spiking conv net on MNIST, the results are better than for the MLP (Table 3 vs. 1) which is not too unexpected. However, for Table 3 data augmentation was used as well, but not for Table 1. It would be interesting to see how the result in Table 3 changes without data augmentation – i.e. how much of a performance improvement is gained by using a conv net over an MLP? (6 Convergence speed) One advantage of the proposed method is that it seems to converge faster to high accuracies compared to competitor methods. Since the final accuracies are almost on-par, it would be interesting to have more quantitative analysis on convergence behavior (i.e. convergence curves for the different methods). I understand, of course, that this might not be easily possible since the competitor papers might not report such data and would require re-implementations. If such curves cannot be easily obtained, I do not expect the authors to re-implement other methods during rebuttal, but if possible it would be nice to include convergence curves (i.e. accuracy vs. training epochs) for the proposed method, such that future methods can compare against these results. (7 Supplementary code) The accompanying release of code in the style of an easily extendable and customizable framework is a big plus and could potentially be very beneficial to the community. For the final release of the code please consider adding a bit more documentation and some documented examples. I also highly appreciate that the authors tested their code on Windows and Linux.