Similar to models of brain-like computation, artificial deep neural networks rely on distributed coding, parallel processing and plastic synaptic weights. Training deep neural networks with the error-backpropagation algorithm, however, is considered bio-implausible. An appealing alternative to training deep neural networks is to use one or a few hidden layers with fixed random weights or trained with an unsupervised, local learning rule and train a single readout layer with a supervised, local learning rule. We find that a network of leaky-integrate-andfire neurons with fixed random, localized receptive fields in the hidden layer and spike timing dependent plasticity to train the readout layer achieves 98.1% test accuracy on MNIST, which is close to the optimal result achievable with errorbackpropagation in non-convolutional networks of rate neurons with one hidden layer. To support the design choices of the spiking network, we systematically compare the classification performance of rate networks with a single hidden layer, where the weights of this layer are either random and fixed, trained with unsupervised Principal Component Analysis or Sparse Coding, or trained with the backpropagation algorithm. This comparison revealed, first, that unsupervised learning does not lead to better performance than fixed random projections for large hidden layers on digit classification (MNIST) and object recognition (CI-FAR10); second, networks with random projections and localized receptive fields perform significantly better than networks with all-to-all connectivity and almost reach the performance of networks trained with the backpropagation algorithm. The performance of these simple random projection networks is comparable to most current models of bio-plausible deep learning and thus provides an interesting benchmark for future approaches.
This article compares different methods to train a two-layer spiking neural network (SNN) in a bio-plausible way on the MNIST dataset, showing that fixed localized random connections that form the hidden layer, in combination with a supervised local learning rule on the output layer can achieve close to state-of-the-art accuracy compared to other SNN architectures. The authors investigate three methods to train the first layer in an unsupervised way: principal component analysis (PCA) on the rates, sparse coding of activations, and fixed random local receptive fields.  Each of the methods is evaluated on the one hand in a time-stepped simulator, using LIF neurons and on the other hand using a rate-approximated model which allows for faster simulations. Results are compared between each other and as reference with  standard backpropagation and feedback alignment.  The main finding is that localized random projections outperform other unsupervised ways of computing first layer features, and with many hidden neurons approaches backpropagation results. These results are summarized in Table 8, which compares results of the paper and other state-of-the-art and bio-plausible SNNs. PCA and sparse coding work worse on MNIST than local random projections, regardless if the network is rate-based, spike-based or a regular ANN trained with the delta rule. Feedback Alignment, although only meant for comparison, performs best of the algorithms investigated in this paper.
In general the question how to train multi-layer spiking neural networks in a bio-plausible way is very relevant for computational neuroscience, and has attracted some attention from the machine learning community in recent years (e.g. Bengio et al. 2015, Scellier & Bengio 2016, Sacramento et al. 2018). It is therefore a suitable topic for ICLR. Of course the good performance of single-layer random projections is not surprising, because it is essentially the idea of the Extreme Learning Machine, and this concept has been well studied also for neuromorphic approaches (e.g. Yao & Basu, 2017), and versions with local receptive fields exist as well (Huang et al. 2015 "Local Receptive Fields Based Extreme Learning Machine"). While the comparison of different unsupervised methods on MNIST is somehow interesting, it fails to show any deeper insights because MNIST is a particularly simple task, and already the CIFAR 10 results are far away from the state-of-the-art (which is >96% using CNNs). Another interesting comparison that is missing is with clustering weights, which has shown good performance for CNNs e.g. in (Coates & Ng, 2012) or (Dundar et al. 2015), and is also unsupervised.
The motivation is not 100% clear because the first experiment uses spikes, and shows a non-negligible difference to rate models (the authors claim it's almost the same, but for MNIST differences of 0.5% are significant). All later results are purely about rate models. The authors apparently did not explore e.g. conversion techniques as in (Diehl et al. 2015) to make the spiking results match the rate versions better e.g. by weight normalization.
I would rate the significance to the SNN community as average, and to the entire ICLR community as low. The significance would be higher if it was shown that this method scales to deeper networks or at least can be utilized in deeper architectures. Scrutinizing the possibilitites with random projections on the other hand could lead to more interesting results. But the best results here are obtained with 5000 neurons with 10x10 receptive fields on images of size 28x28, thus the representation is more than overcomplete, and of higher complexity than a convolution layer with 3x3 kernels and many input maps.
Because the results provide only limited insights beyond MNIST I can therefore not support acceptance at ICLR.
Pros:
+ interesting comparison of unsupervised feature learning techniques
+ interesting topic of bio-plausible deep learning
Cons:
- only MNIST, no indications if method will scale
- results are not better than state-of-the-art
Minor comments:
The paper is generally well-written and structured, although some of the design choices could have been explained in more detail. Generally, it is not discussed if random connections have any advantage over other spiking models in terms of accuracy, efficiency or speed, besides the obvious fact that one does not have to train this layer. 
The title is a bit confusing. While it's not wrong, I had to read it multiple times to understand what was meant.
The first sentence in the caption for Fig. 2 is also confusing, mixing the descriptions of panel A and B. Also, in A membrane potentials are shown, but the post-membrane potential seems to integrate a constant current instead of individual spikes. Is this already the rate approximation of Eq. 2? Or is it because of the statement in the caption that they both receive very high external inputs. In general, the figures in panel A and B do not make the dynamics of the network or the supervised STDP much clearer. 
Principal Component Analysis and Sparse Coding are done algorithmically instead of using a sort of nonlinear Hebbian Learning as in Lillicrap 2016. It would have been interesting to see if this changes the comparatively bad results for PCA and SC.
In Fig. 3, the curve in the random projections case is not saturated, maybe it would have been interesting to go above n_h = 5000. As there are 784 input neurons, a convolutional neural network with 7 filter banks already would have around 5000 neurons, but in this case each filter would be convolved over the whole image, while with random projections the filter only exists locally. 
In Eq. 1, the notation is a bit ambigous: The first delta-function seems to be the Dirac-delta for continuous t, while the second delta is a Kronecker-delta with discrete t.
In A.1 and A.4.2 it is stated that the output of a layer is u_{t+1} = W u_t + b but I think in both cases it should be W a_t + b where a_t = phi(u_t). Otherwise, you just have a linear model and no activations. 
In Table 3, a typo: "eq. equation"