Infinite-width neural networks have been extensively used to study the theoretical properties underlying the extraordinary empirical success of standard, finite-width neural networks. Nevertheless, until now, infinite-width networks have been limited to at most two hidden layers. To address this shortcoming, we study the initialisation requirements of these networks and show that the main challenge for constructing them is defining the appropriate sampling distributions for the weights. Based on these observations, we propose a principled approach to weight initialisation that correctly accounts for the functional nature of the hidden layer activations and facilitates the construction of arbitrarily many infinite-width layers, thus enabling the construction of arbitrarily deep infinite-width networks. The main idea of our approach is to iteratively reparametrise the hidden-layer activations into appropriately defined reproducing kernel Hilbert spaces and use the canonical way of constructing probability distributions over these spaces for specifying the required weight distributions in a principled way. Furthermore, we examine the practical implications of this construction for standard, finite-width networks. In particular, we derive a novel weight initialisation scheme for standard, finite-width networks that takes into account the structure of the data and information about the task at hand. We demonstrate the effectiveness of this weight initialisation approach on the MNIST, CIFAR-10 and Year Prediction MSD datasets.
Pros:
This paper uses kernel mappings between any two layers for weight initialisation. Using the representer theorem, a proper distribution for weights is constructed in H_{k_i} instead of being learned by \phi_i, and then is formulated as a GP. 
Cons:
However, there are some key issues.
1. The so-called “infinite width” is just yielded by kernels in RKHS for weight initialization. For practical implementation, the authors use this scheme with random Fourier features to construct finite-width network. A key issue is that how to guarantee that the approximated weights are still in the same space? For example, weights can be in RKHS, but their approximation might be not in RKHS. See in [S1] for details.
 
[S1] Generalization Properties of Learning with Random Features, NIPS 2017.
 
2. Experimental part is not very convincing. First, the authors just compare different initialization schemes. The used architectures are simple and not representative. Second, the overall performance is not satisfactory, and the compared classification datasets are quite small. Overall, the experimental results are inadequate and unconvincing.
Summary:
The paper attempts to proposal a weight initialization scheme to enable infinite deep infinite-width networks. However, there are some key issues not address such as whether the approximated weights are still in the same space and the limited experimental results.
Response to rebuttal:
The authors have addressed my question about the weights being still in the same RKHS. I still think the motivation and experiments are not very satisfactory. 
Therefore the paper is very borderline. However, I would like to bump my rating a bit higher.