Published as a conference paper at ICLR 2018 TEMPORALLY EFFICIENT DEEP LEARNING WITH SPIKES
The vast majority of natural sensory data is temporally redundant. For instance, video frames or audio samples which are sampled at nearby points in time tend to have similar values. Typically, deep learning algorithms take no advantage of this redundancy to reduce computations. This can be an obscene waste of energy. We present a variant on backpropagation for neural networks in which computation scales with the rate of change of the data -not the rate at which we process the data. We do this by implementing a form of Predictive Coding wherein neurons communicate a combination of their state, and their temporal change in state, and quantize this signal using Sigma-Delta modulation. Intriguingly, this simple communication rule give rise to units that resemble biologically-inspired leaky integrate-and-fire neurons, and to a spike-timing-dependent weight-update similar to Spike-Timing Dependent Plasticity (STDP), a synaptic learning rule observed in the brain. We demonstrate that on MNIST, on a temporal variant of MNIST, and on Youtube-BB, a dataset with videos in the wild, our algorithm performs about as well as a standard deep network trained with backpropagation, despite only communicating discrete values between layers.
This paper presents a novel method for spike based learning that aims at reducing the needed computation during learning and testing when classifying temporal redundant data. This approach extends the method presented on Arxiv on Sigma delta quantized networks (Peter O’Connor and Max Welling. Sigma delta quantized networks. arXiv preprint arXiv:1611.02024, 2016b.). Overall, the paper is interesting and promising; only a few works tackle the problem of learning with spikes showing the potential advantages of such form of computing. The paper, however, is not flawless. The authors demonstrate the method on just two datasets, and effectively they show results of training only for Feed-Forward Neural Nets (the authors claim that “the entire spiking network end-to-end works” referring to their pre-trained VGG19, but this paper presents only training for the three top layers). Furthermore, even if suitable datasets are not available, the authors could have chosen to train different architectures. The first dataset is the well-known benchmark MNIST also presented in a customized Temporal-MNIST. Although it is a common base-line, some choices are not clear: why using a FFNN instead that a CNN which performs better on this dataset; how data is presented in terms of temporal series – this applies to the Temporal MNIST too; why performances for Temporal MNIST – which should be a more suitable dataset — are worse than for the standard MNIST; what is the meaning of the right column of Figure 5 since it’s just a linear combination of the GOps results. For the second dataset, some points are not clear too: why the labels and the pictures seem not to match (in appendix E); why there are more training iterations with spikes w.r.t. the not-spiking case. Overall, the paper is mathematically sound, except for the “future updates” meaning which probably deserves a clearer explanation. Moreover, I don’t see why the learning rule equations (14-15) are described in the appendix, while they are referred constantly in the main text. The final impression is that the problem of the dynamical range of the hidden layer activations is not fully resolved by the empirical solution described in Appendix D: perhaps this problem affects CCNs more than FFN. 
Finally, there are some minor issues here and there (the authors show quite some lack of attention for just 7 pages):
-	Two times “get” in “we get get a decoding scheme” in the introduction;
-	Two times “update” in “our true update update as” in Sec. 2.6;
-	Pag3 correct the capital S in 2.3.1
-	Pag4 Figure 1 increase font size (also for Figure2); close bracket after Equation 3; N (number of spikes) is not defined
-	Pag5 “one-hot” or “onehot”; 
-	in the inline equation the sum goes from n=1 to S, while in eq.(8) it goes from n=1 to N;
-	Eq(10)(11)(12) and some lines have a typo (a \cdot) just before some of the ws;
-	Pag6 k_{beta} is not defined in the main text;
-	Pag7 there are two “so that” in 3.1; capital letter “It used 32x10^12..”; beside, here, why do not report the difference in computation w.r.t. not-spiking nets?
-	Pag7 in 3.2 “discussed in 1” is section 1?
-	Pag14 Appendix E, why the labels don’t match the pictures;
-	Pag14 Appendix F, explain better the architecture used for this experiment.