Published as a conference paper at ICLR 2018 DEEP NEURAL NETWORKS AS GAUSSIAN PROCESSES
It has long been known that a single-layer fully-connected neural network with an i.i.d. prior over its parameters is equivalent to a Gaussian process (GP), in the limit of infinite network width. This correspondence enables exact Bayesian inference for infinite width neural networks on regression tasks by means of evaluating the corresponding GP. Recently, kernel functions which mimic multi-layer random neural networks have been developed, but only outside of a Bayesian framework. As such, previous work has not identified that these kernels can be used as covariance functions for GPs and allow fully Bayesian prediction with a deep neural network. In this work, we derive the exact equivalence between infinitely wide deep networks and GPs. We further develop a computationally efficient pipeline to compute the covariance function for these GPs. We then use the resulting GPs to perform Bayesian inference for wide deep neural networks on MNIST and CIFAR-10. We observe that trained neural network accuracy approaches that of the corresponding GP with increasing layer width, and that the GP uncertainty is strongly correlated with trained network prediction error. We further find that test performance increases as finite-width trained networks are made wider and more similar to a GP, and thus that GP predictions typically outperform those of finite-width networks. Finally we connect the performance of these GPs to the recent theory of signal propagation in random neural networks.
Neal (1994) showed that a one hidden layer Bayesian neural network, under certain conditions, converges to a Gaussian process as the number of hidden units approaches infinity. Neal (1994) and Williams (1997) derive the resulting kernel functions for such Gaussian processes when the neural networks have certain transfer functions.
Similarly, the authors show an analogous result for deep neural networks with multiple hidden layers and an infinite number of hidden units per layer, and show the form of the resulting kernel functions. For certain transfer functions, the authors perform a numerical integration to compute the resulting kernels. They perform experiments on MNIST and CIFAR-10, doing classification by scaled regression. 
Overall, the work is an interesting read, and a nice follow-up to Neal’s earlier observations about 1 hidden layer neural networks. It combines several insights into a nice narrative about infinite Bayesian deep networks. However, the practical utility, significance, and novelty of this work -- in its current form -- are questionable, and the related work sections, analysis, and experiments should be significantly extended. 
In detail:
(1) This paper misses some obvious connections and references, such as 
* Krauth et. al (2017): “Exploring the capabilities and limitations of Gaussian process models” for recursive kernels with GPs.
* Hazzan & Jakkola (2015): “Steps Toward Deep Kernel Methods from Infinite Neural Networks” for GPs corresponding to NNs with more than one hidden layer.
* The growing body of work on deep kernel learning, which “combines the inductive biases and representation learning abilities of deep neural networks with the non-parametric flexibility of Gaussian processes”. E.g.: (i) “Deep Kernel Learning” (AISTATS 2016); (ii) “Stochastic Variational Deep Kernel Learning” (NIPS 2016); (iii) “Learning Scalable Deep Kernels with Recurrent Structure” (JMLR 2017). 
These works should be discussed in the text.
(2) Moreover, as the authors rightly point out, covariance functions of the form used in (4) have already been proposed. It seems the novelty here is mainly the empirical exploration (will return to this later), and numerical integration for various activation functions. That is perfectly fine -- and this work is still valuable. However, the statement “recently, kernel functions for multi-layer random neural networks have been developed, but only outside of a Bayesian framework” is incorrect. For example, Hazzan & Jakkola (2015) in “Steps Toward Deep Kernel Methods from Infinite Neural Networks” consider GP constructions with more than one hidden layer. Thus the novelty of this aspect of the paper is overstated. 
See also comment [*] later on the presentation. In any case, the derivation for computing the covariance function (4) of a multi-layer network is a very simple reapplication of the procedure in Neal (1994). What is less trivial is estimating (4) for various activations, and that seems to the major methodological contribution. 
Also note that multidimensional CLT here is glossed over. It’s actually really unclear whether the final limit will converge to a multidimensional Gaussian with that kernel without stronger conditions.  This derivation should be treated more thoroughly and carefully.
(3) Most importantly, in this derivation, we see that the kernels lose the interesting representations that come from depth in deep neural networks. Indeed, Neal himself says that in the multi-output settings, all the outputs become uncorrelated. Multi-layer representations are mostly interesting because each layer shares hidden basis functions. Here, the sharing is essentially meaningless, because the variance of the weights in this derivation shrinks to zero. 
In Neal’s case, the method was explored for single output regression, where the fact that we lose this sharing of basis functions may not be so restrictive. However, these assumptions are very constraining for multi-output classification and also interesting multi-output regressions.
[*]: Generally, in reading the abstract and introduction, we get the impression that this work somehow allows us to use really deep and infinitely wide neural networks as Gaussian processes, and even without the pain of training these networks. “Deep neural networks without training deep networks”. This is not an accurate portrayal. The very title “Deep neural networks as Gaussian processes” is misleading, since it’s not really the deep neural networks that we know and love. In fact, you lose valuable structure when you take these limits, and what you get is very different than a standard deep neural network. In this sense, the presentation should be re-worked.
(4) Moreover, neural networks are mostly interesting because they learn the representation. To do something similar with GPs, we would need to learn the kernel. But here, essentially no kernel learning is happening. The kernel is fixed. 
(5) Given the above considerations, there is great importance in understanding the practical utility of the proposed approach through a detailed empirical evaluation. In other words, how structured is this prior and does it really give us some of the interesting properties of deep neural networks, or is it mostly a cute mathematical trick?  
Unfortunately, the empirical evaluation is very preliminary, and provides no reassurance that this approach will have any practical relevance:
(i) Directly performing regression on classification problems is very heuristic and unnecessary.
(ii) Given the loss of dependence between neurons in this approach, it makes sense to first explore this method on single output regression, where we will likely get the best idea of its useful properties and advantages. 
(iii) The results on CIFAR10 are very poor. We don’t need to see SOTA performance to get some useful insights in comparing for example parametric vs non-parametric, but 40% more error than SOTA makes it very hard to say whether any of the observed patterns hold weight for more competitive architectural choices. 
A few more minor comments:
(i) How are you training a GP exactly on 50k training points? Even storing a 50k x 50k matrix requires about 20GB of RAM. Even with the best hardware, computing the marginal likelihood dozens of times to learn hyperparameters would be near impossible. What are the runtimes?
(ii) "One benefit in using the GP is due to its Bayesian nature, so that predictions have uncertainty estimates (Equation (9)).”  The main benefit of the GP is not the uncertainty in the predictions, but the marginal likelihood which is useful for kernel learning.