DEEP CONVOLUTIONAL NETWORKS AS SHALLOW GAUSSIAN PROCESSES
We show that the output of a (residual) convolutional neural network (CNN) with an appropriate prior over the weights and biases is a Gaussian process (GP) in the limit of infinitely many convolutional filters, extending similar results for dense networks. For a CNN, the equivalent kernel can be computed exactly and, unlike "deep kernels", has very few parameters: only the hyperparameters of the original CNN. Further, we show that this kernel has two properties that allow it to be computed efficiently; the cost of evaluating the kernel for a pair of images is similar to a single forward pass through the original CNN with only one filter per layer. The kernel equivalent to a 32-layer ResNet obtains 0.84% classification error on MNIST, a new record for GPs with a comparable number of parameters.
This paper shows that deep convolutional networks (CNNs, without pooling) with a suitable prior over weights can be seen as shallow Gaussian processes (GPs) with a specific covariance function. It shows that this covariance function can be computed efficiently (when compared to previous attempts at resembling convolutional networks with GPs), with a cost that only depends linearly on the number of layers and the input dimensionality, i.e.~O(N^2 L D). 
To show the equivalence between deep CNNs and shallow GPs, the paper uses similar ideas to those proposed by Matthews et al (2018a) and Lee et al (2017), i.e. using the multivariate central limit theorem in very large networks, where in the case of this paper the limit is taken as the number of channels at each layer goes to infinity. Therefore, from a theoretical perspective, these ideas have been proposed before. However, the paper presents a novel efficient way to compute the convolutional kernel, which I believe has merits on its own. 
However, the model setting for classification (where deep CNNs have been successful) and consequent evaluation on the MNIST dataset is less than convincing. One of the main motivations for Bayesian CNNs and GPs (and the paper argue for this in the intro) is to be able to provide good uncertainty estimates. However, the classification problem is framed in a regression setting, where neither probabilistic estimates are evaluated or even provided. Indeed, only the error rate is given on Table 1. To me, this is certainly not enough for a Bayesian/GP method and it is a critical deficiency of the paper in its current form. While I understand having a non-Gaussian likelihood will complicate things and conflate the kernel contribution with the approximations, I believe it is necessary to provide and evaluate such probabilistic estimates and compare them to other GP approaches (even using other less than satisfying methods such as calibration/scaling). Along a similar vein, it is unclear what objective function was used for hyper-parameter learning but, given that the authors actually “sample hyper-parameters”, I am guessing a proper probabilistic objective such as the marginal likelihood is out of the question.
Other (perhaps minor) deficiencies is that the method is not scalable to large datasets (I am even surprised the authors managed to run this on full MNIST) and that no theoretical analysis is done (e.g. as in Mattews et al, 2018a). 
Minor comments:
* In the intro, “Other methods such as Gaussian Processes”: GPs are not a method and I believe the authors really mean here Gaussian process regression. 
* The prior variance over filters in Eq (3) divides over the number of channels.  Why does a Gaussian prior with infinite precision make sense here?
* The authors should report the state of the art of using GPs for MNIST classification using non-convolutional kernels).