summaries: The authors show that in the infinite width limit, deep GPs becomes a single-layer GP, generalizing the result that deep neural networks at initialization become a GP in the infinite width limit. The authors show that increasing depth of the GP makes the distribution of the final layer of deep GPs deviate from being Gaussian at finite width. The authors claim that this simplification implies that large widths are detrimental to neural network performance beyond some point. The paper argues that deep Gaussian Processes with large widths and more than 2 layers have undesirable properties. In particular, the paper shows that firstly, deep GP prior in the GP limit is essentially shallow, and secondly, deep GP posterior then loses data adaptivity in the hidden layers. The paper further examines the role of depth and argues that the deep GP prior, if not equal to a GP, has heavy-tailed behaviors.  ================================== Post-rebuttal:  Thanks for the reply and clarification!  While I agree that very large width hurts in the context of deep GP as shown in the paper, I also agree with other reviewers that the message that very large width hurts generally is overly strong. The literature has a mixed bag of results concerning widths -- theoretically or empirically, so a general statement should be validated very carefully.  I'd like to keep my score.      This paper provides a narrative of composite Gaussian process models. In specific the paper provides insights into the interplay between depth and width in the context of capacity and representative power. The paper further draws conclusions on the implications that this has for composite finite basis function models such as neural networks.  The paper shows that in the limit of infinite width a Deep Gaussian Process (DGP) converges to an ordinary Gaussian Process (GP). Indeed, in the limit of infinite width the covariance function of each hidden GP layer becomes deterministic (as a consequence of the strong law of large numbers). This allows proving that the marginal prior covariance  of the DGP becomes gaussian (Theorem1). The  consequence of this theorem is that as the width increases DGP models lose their ability to adapt the hidden covariance functions to the input data (section 4). This theoretical result is supported by numerical experiments (section 6.1). The paper also addresses the convergence of a DGP to a GP with increasing depth and width (section 5). Since it is known that deep neural networks (DNN) approach a GP as they are made wider and wider,  Theorem 1 is used to  claim that a very large width can be detrimental to the expressivity of a DNN (and not only a DGP).   
abstractText: Large width limits have been a recent focus of deep learning research: modulo computational practicalities, do wider networks outperform narrower ones? Answering this question has been challenging, as conventional networks gain representational power with width, potentially masking any negative effects. Our analysis in this paper decouples capacity and width via the generalization of neural networks to Deep Gaussian Processes (Deep GP), a class of nonparametric hierarchical models that subsume neural nets. In doing so, we aim to understand how width affects (standard) neural networks once they have sufficient capacity for a given modeling task. Our theoretical and empirical results on Deep GP suggest that large width can be detrimental to hierarchical models. Surprisingly, we prove that even nonparametric Deep GP converge to Gaussian processes, effectively becoming shallower without any increase in representational power. The posterior, which corresponds to a mixture of data-adaptable basis functions, becomes less datadependent with width. Our tail analysis demonstrates that width and depth have opposite effects: depth accentuates a model’s non-Gaussianity, while width makes models increasingly Gaussian. We find there is a “sweet spot” that maximizes test performance before the limiting GP behavior prevents adaptability, occurring at width = 1 or width = 2 for nonparametric Deep GP. These results make strong predictions about the same phenomenon in conventional neural networks trained with L2 regularization (analogous to a Gaussian prior on parameters): we show that such neural networks may need up to 500− 1000 hidden units for sufficient capacity—depending on the dataset—but further width degrades performance.
authors.name: Geoff Pleiss
authors.name: John P. Cunningham
references.authors: D. Agrawal T. Papamarkou J. Hinkle
references.title: Wide neural networks with bottlenecks are deep Gaussian processes
references.venue: Journal of Machine Learning Research, 21(175):1–66,
references.authors: L. Aitchison
references.title: Why bigger is not always better: on finite and infinite neural networks
references.venue: ICML,
references.authors: L. Aitchison A. Yang S.W. Ober
references.title: Deep kernel processes
references.venue: ICML,
references.authors: Z. Allen-Zhu Y. Li
references.title: What can ResNet learn efficiently, going beyond kernels
references.venue: In NeurIPS,
references.authors: Z. Allen-Zhu Y. Li
references.title: Backward feature correction: How deep learning performs deep learning
references.venue: arXiv preprint arXiv:2001.04413,
references.authors: Z. Allen-Zhu Y. Li Y. Liang
references.title: Learning and generalization in overparameterized neural networks, going beyond two layers
references.venue: NeurIPS,
references.authors: S. Arora S. Du W. Hu Z. Li R. Wang
references.title: Fine-grained analysis of optimization and generalization for overparameterized two-layer neural networks
references.venue: ICML,
references.authors: S. Arora S.S. Du W. Hu Z. Li R. Salakhutdinov R. Wang
references.title: On exact computation with an infinitely wide neural net
references.venue: NeurIPS,
references.authors: S. Arora S.S. Du Z. Li R. Salakhutdinov R. Wang D. Yu
references.title: Harnessing the power of infinitely wide deep nets on small-data tasks
references.venue: ICLR,
references.authors: Y. Bai J.D. Lee
references.title: Beyond linearization: On quadratic and higher-order approximation of wide neural networks
references.venue: ICLR,
references.authors: M. Belkin D. Hsu S. Ma S. Mandal
references.title: Reconciling modern machine-learning practice and the classical bias–variance trade-off
references.venue: Proceedings of the National Academy of Sciences, 116(32):15849–15854,
references.authors: Y. Bengio O. Delalleau N. Le Roux
references.title: The curse of dimensionality for local kernel machines
references.authors: P. Billingsley
references.title: Convergence of probability measures
references.venue: John Wiley & Sons,
references.authors: E. Bingham J.P. Chen M. Jankowiak F. Obermeyer N. Pradhan T. Karaletsos R. Singh P. Szerlip P. Horsfall N.D. Goodman
references.title: Pyro: Deep universal probabilistic programming
references.venue: The Journal of Machine Learning Research, 20(1):973–978,
references.authors: C.M. Bishop
references.title: Pattern recognition and machine learning
references.venue: Springer,
references.authors: K. Blomqvist S. Kaski M. Heinonen
references.title: Deep convolutional Gaussian processes
references.venue: Joint European Conference on Machine Learning and Knowledge Discovery in Databases,
references.authors: C. Blundell J. Cornebise K. Kavukcuoglu D. Wierstra
references.title: Weight uncertainty in neural network
references.venue: ICML,
references.authors: T. Bui D. Hernández-Lobato J. Hernandez-Lobato Y. Li R. Turner
references.title: Deep Gaussian processes for regression using approximate expectation propagation
references.venue: ICML,
references.authors: Y. Cao Q. Gu
references.title: Generalization bounds of stochastic gradient descent for wide and deep neural networks
references.venue: NeurIPS,
references.authors: M. Chen Y. Bai J.D. Lee T. Zhao H. Wang C. Xiong R. Socher
references.title: Towards understanding hierarchical learning: Benefits of neural representations
references.venue: NeurIPS,
references.authors: L. Chizat F. Bach
references.title: On the global convergence of gradient descent for over-parameterized models using optimal transport
references.venue: NeurIPS,
references.authors: Y. Cho L. Saul
references.title: Kernel methods for deep learning
references.venue: NeurIPS,
references.authors: K. Cutajar E.V. Bonilla P. Michiardi M. Filippone
references.title: Random feature expansions for deep Gaussian processes
references.venue: ICML,
references.authors: Z. Dai A.C. Damianou J. González N.D. Lawrence
references.title: Variational auto-encoded deep Gaussian processes
references.venue: ICLR,
references.authors: A. Damianou
references.title: Deep Gaussian processes and variational propagation of uncertainty
references.venue: PhD thesis, University of Sheffield,
references.authors: A. Damianou N. Lawrence
references.title: Deep Gaussian processes
references.venue: AISTATS,
references.authors: S. Du J. Lee H. Li L. Wang X. Zhai
references.title: Gradient descent finds global minima of deep neural networks
references.venue: ICML,
references.authors: M.M. Dunlop M.A. Girolami A.M. Stuart A.L. Teckentrup
references.title: How deep are deep Gaussian processes
references.venue: Journal of Machine Learning Research,
references.authors: V. Dutordoir M. Wilk A. Artemev J. Hensman
references.title: Bayesian image classification with deep convolutional Gaussian processes
references.venue: AISTATS,
references.authors: V. Dutordoir J. Hensman M. van der Wilk C.H. Ek Z. Ghahramani N. Durrande
references.title: Deep neural networks as point estimates for deep Gaussian processes
references.venue: NeurIPS,
references.authors: V. Dutordoir H. Salimbeni E. Hambro J. McLeod F. Leibfried A. Artemev M. van der Wilk J. Hensman M.P. Deisenroth S. John
references.title: GPflux: A library for deep Gaussian processes
references.venue: arXiv preprint arXiv:2104.05674,
references.authors: D. Duvenaud O. Rippel R. Adams Z. Ghahramani
references.title: Avoiding pathologies in very deep networks
references.venue: AISTATS,
references.authors: A.Y. Foong D.R. Burt Y. Li R.E. Turner
references.title: On the expressiveness of approximate inference in bayesian neural networks
references.venue: NeurIPS,
references.authors: S. Fort G.K. Dziugaite M. Paul S. Kharaghani D.M. Roy S. Ganguli
references.title: Deep learning versus kernel learning: an empirical study of loss landscape geometry and the time evolution of the neural tangent kernel
references.venue: NeurIPS,
references.authors: Y. Gal Z. Ghahramani
references.title: Dropout as a Bayesian approximation: Representing model uncertainty in deep learning
references.venue: ICML,
references.authors: J.R. Gardner G. Pleiss K.Q. Weinberger D. Bindel A.G. Wilson
references.title: GPyTorch: Blackbox matrixmatrix Gaussian process inference with GPU acceleration
references.venue: NeurIPS,
references.authors: A. Garriga-Alonso C.E. Rasmussen L. Aitchison
references.title: Deep convolutional networks as shallow Gaussian processes
references.venue: ICLR,
references.authors: M. Geiger A. Jacot S. Spigler F. Gabriel L. Sagun S. d’Ascoli G. Biroli C. Hongler M. Wyart
references.title: Scaling description of generalization with number of parameters in deep learning
references.venue: Journal of Statistical Mechanics: Theory and Experiment,
references.authors: M.G. Genton
references.title: Classes of kernels for machine learning: a statistics perspective
references.venue: Journal of Machine Learning Research, 2(Dec):299–312,
references.authors: B. Ghorbani S. Mei T. Misiakiewicz A. Montanari
references.title: Limitations of lazy training of two-layers neural networks
references.venue: NeurIPS,
references.authors: B. Ghorbani S. Mei T. Misiakiewicz A. Montanari
references.title: When do neural networks outperform kernel methods
references.venue: In NeurIPS,
references.authors: E. Golikov
references.title: Towards a general theory of infinite-width limits of neural classifiers
references.venue: ICML,
references.authors: I. Goodfellow Y. Bengio A. Courville Y. Bengio
references.title: Deep learning
references.venue: MIT press Cambridge,
references.authors: J. Halverson A. Maiti K. Stoner
references.title: Neural networks and quantum field theory
references.venue: Machine Learning: Science and Technology, 2(3),
references.authors: M. Havasi J.M. Hernández-Lobato J.J. Murillo-Fuentes
references.title: Inference in deep Gaussian processes using stochastic gradient Hamiltonian Monte Carlo
references.venue: NeurIPS,
references.authors: K. He X. Zhang S. Ren J. Sun
references.title: Deep residual learning for image recognition
references.venue: CVPR,
references.authors: M.D. Hoffman A. Gelman
references.title: The no-u-turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo
references.venue: Journal of Machine Learning Research, 15(1):1593–1623,
references.authors: J. Hron Y. Bahri R. Novak J. Pennington J. Sohl-Dickstein
references.title: Exact posterior distributions of wide bayesian neural networks
references.venue: ICML Workshop on Uncertainty and Robustness in Deep Learning,
references.authors: J. Hron Y. Bahri J. Sohl-Dickstein R. Novak
references.title: Infinite attention: NNGP and NTK for deep attention networks
references.venue: ICML,
references.authors: P. Izmailov S. Vikram M.D. Hoffman A.G. Wilson
references.title: What are bayesian neural network posteriors really like
references.venue: In ICML,
references.authors: A. Jacot F. Gabriel C. Hongler
references.title: Neural tangent kernel: Convergence and generalization in neural networks
references.venue: NeurIPS,
references.authors: D.P. Kingma J. Ba
references.title: Adam: A method for stochastic optimization
references.venue: ICLR,
references.authors: A. Krizhevsky
references.title: Learning multiple layers of features from tiny images
references.authors: Y. LeCun L. Bottou Y. Bengio P. Haffner
references.title: Gradient-based learning applied to document recognition
references.venue: Proceedings of the IEEE, 86(11):2278–2324,
references.authors: J. Lee Y. Bahri R. Novak S.S. Schoenholz J. Pennington J. Sohl-Dickstein
references.title: Deep neural networks as Gaussian processes
references.venue: ICLR,
references.authors: J. Lee L. Xiao S.S. Schoenholz Y. Bahri R. Novak J. Sohl-Dickstein J. Pennington
references.title: Wide neural networks of any depth evolve as linear models under gradient descent
references.venue: NeurIPS,
references.authors: J. Lee S. Schoenholz J. Pennington B. Adlam L. Xiao R. Novak J. Sohl-Dickstein
references.title: Finite versus infinite neural networks: an empirical study
references.venue: NeurIPS,
references.authors: H. Li Z. Xu G. Taylor C. Studer T. Goldstein
references.title: Visualizing the loss landscape of neural nets
references.venue: NeurIPS,
references.authors: Y. Li T. Ma H.R. Zhang
references.title: Learning over-parametrized two-layer neural networks beyond NTK
references.venue: COLT,
references.authors: C. Louizos M. Welling
references.title: Structured and efficient variational deep learning with matrix Gaussian posteriors
references.venue: ICML,
references.authors: C.-K. Lu S.C.-H. Yang X. Hao P. Shafto
references.title: Interpretable deep Gaussian processes with moments
references.venue: AISTATS,
references.authors: Z. Lu H. Pu F. Wang Z. Hu L. Wang
references.title: The expressive power of neural networks: A view from the width
references.venue: NeurIPS,
references.authors: A.G. d. G. Matthews M. Rowland J. Hron R.E. Turner Z. Ghahramani
references.title: Gaussian process behaviour in wide deep neural networks
references.venue: In ICLR,
references.authors: S. Mei A. Montanari P.-M. Nguyen
references.title: A mean field view of the landscape of two-layer neural networks
references.venue: Proceedings of the National Academy of Sciences, 115(33):E7665–E7671,
references.authors: C.A. Micchelli Y. Xu H. Zhang
references.title: Universal kernels
references.venue: Journal of Machine Learning Research, 7(12),
references.authors: G. Montúfar R. Pascanu K. Cho Y. Bengio
references.title: On the number of linear regions of deep neural networks
references.venue: NeurIPS,
references.authors: P. Nakkiran G. Kaplun Y. Bansal T. Yang B. Barak I. Sutskever
references.title: Deep double descent: Where bigger models and more data hurt
references.venue: ICLR,
references.authors: R.M. Neal
references.title: Bayesian learning for neural networks
references.venue: PhD thesis, University of Toronto,
references.authors: Q. Nguyen M. Hein
references.title: The loss surface of deep and wide neural networks
references.venue: ICML,
references.authors: R. Novak L. Xiao J. Lee Y. Bahri G. Yang J. Hron D.A. Abolafia J. Pennington J. Sohl-Dickstein
references.title: Bayesian deep convolutional networks with many channels are Gaussian processes
references.venue: ICLR,
references.authors: S.W. Ober L. Aitchison
references.title: Global inducing point variational posteriors for Bayesian neural networks and deep Gaussian processes
references.venue: ICML,
references.authors: A. Paszke S. Gross F. Massa A. Lerer J. Bradbury G. Chanan T. Killeen Z. Lin N. Gimelshein L. Antiga
references.title: PyTorch: An imperative style, high-performance deep learning library
references.venue: NeurIPS,
references.authors: B. Poole S. Lahiri M. Raghu J. Sohl-Dickstein S. Ganguli
references.title: Exponential expressivity in deep neural networks through transient chaos
references.venue: NeurIPS,
references.authors: M. Raghu B. Poole J. Kleinberg S. Ganguli J. Sohl-Dickstein
references.title: On the expressive power of deep neural networks
references.venue: ICML,
references.authors: A. Rahimi B. Recht
references.title: Random features for large-scale kernel machines
references.venue: In NeurIPS,
references.authors: C.E. Rasmussen C. Williams
references.title: Gaussian processes for machine learning, volume 1
references.venue: MIT Press,
references.authors: A. Romero N. Ballas S.E. Kahou A. Chassang C. Gatta Y. Bengio
references.title: Fitnets: Hints for thin deep nets
references.venue: ICLR,
references.authors: H. Salimbeni M. Deisenroth
references.title: Doubly stochastic variational inference for deep Gaussian processes
references.venue: NeurIPS,
references.authors: I.J. Schoenberg
references.title: Metric spaces and completely monotone functions
references.venue: Annals of Mathematics, pages 811–841,
references.authors: V. Shankar A. Fang W. Guo S. Fridovich-Keil J. Ragan-Kelley L. Schmidt B. Recht
references.title: Neural kernels without tangents
references.venue: ICML,
references.authors: M. Soltanolkotabi A. Javanmard J.D. Lee
references.title: Theoretical insights into the optimization landscape of over-parameterized shallow neural networks
references.venue: IEEE Transactions on Information Theory, 65(2):742–769,
references.authors: M. Telgarsky
references.title: Benefits of depth in neural networks
references.venue: COLT,
references.authors: M. Vladimirova J. Verbeek P. Mesejo J. Arbel
references.title: Understanding priors in Bayesian neural networks at the unit level
references.venue: ICML,
references.authors: Y. Wang M. Brubaker B. Chaib-Draa R. Urtasun
references.title: Sequential inference for deep Gaussian process
references.venue: AISTATS,
references.authors: C. Wei J. Lee Q. Liu T. Ma
references.title: Regularization matters: Generalization and optimization of neural nets vs their induced kernel
references.venue: NeurIPS,
references.authors: A. Yaglom
references.title: Correlation theory of stationary and related random functions
references.venue: Springer Series in Statistics, New York,
references.authors: G. Yang
references.title: Tensor programs I: Wide feedforward or recurrent neural networks of any architecture are gaussian processes
references.venue: NeurIPS,
references.authors: G. Yang
references.title: Tensor programs II: Neural tangent kernel for any architecture
references.venue: arXiv preprint arXiv:2006.14548,
references.authors: G. Yang E.J. Hu
references.title: Tensor programs IV: Feature learning in infinite-width neural networks
references.venue: ICML,
references.authors: G. Yehudai O. Shamir
references.title: On the power and limitations of random features for understanding neural networks
references.venue: NeurIPS,
references.authors: S. Zagoruyko N. Komodakis
references.title: Wide residual networks
references.venue: BMVC,
references.authors: J.A. Zavatone-Veth C. Pehlevan
references.title: Exact priors of finite neural networks
references.venue: NeurIPS,
sections.heading: 1 Introduction
sections.text: Research has shown that deeper neural networks tend to be more expressive and efficient than wider networks under a variety of metrics [e.g. 21, 63, 67, 70, 74, 75, 78, 83]. Nevertheless, there is resurgent interest in wide models due in part to empirical successes [e.g. 92] and theoretical analyses of limiting behavior. When randomly initialized to create a distribution over functions, neural networks converge to Gaussian processes (GP) as width increases. This result, first proved for 2-layer networks [69], has been extended to deeper networks [56, 64], convolutional networks [38, 71], and other architectures [50, 88]. A similar limit exists for gradient-trained networks, which behave increasingly like kernel machines under the neural tangent kernel [e.g. 6, 8, 28, 39, 52, 57, 89].
  While these limits simplify analyses, there is something unsettling about reducing neural networks to kernel methods. Neal [69, p. 161] describes the GP limit as “disappointing,” noting that “infinite networks do not have hidden units that represent ‘hidden features’. . . often seen [as the] interesting aspect of neural network learning.” Recent work indeed shows that learned hierarchical features can be exponentially more efficient than the fixed shallow representations of kernels [e.g. 4, 5, 8, 11, 13, 21, 41, 42, 60, 91]. At the same time, wider networks can more accurately model complex functions
  35th Conference on Neural Information Processing Systems (NeurIPS 2021).
  [44]. Thus, wide limits appear to confound opposing phenomenon: increased capacity makes them more expressive, yet the loss of hierarchical features seems to make them less expressive. This may explain the mixed empirical performance of limiting models: outperforming finite width models in some scenarios [e.g. 9, 38, 39, 58], yet falling short on more complex tasks [e.g. 8, 11, 35, 57, 81].
  This paper aims to decouple these effects of large width. Our goal is to understand the inductive biases of wide networks, after a network has “sufficient” capacity for a given modeling task. We ask: If we control for the effects of increased capacity, what—if any—value remains in wide networks?
  To achieve this control, we note that a typical neural network layer corresponds to a finite basis, where elementwise nonlinearities transform each hidden feature into a single basis function. In order to decouple width from capacity, one could generalize these layers so that each nonlinearity produces any number of basis functions; if each hidden feature gives rise to an infinite and universal basis, then hidden layers would have infinite representational capacity regardless of width. This generalization is in fact a well-studied class of hierarchical models—Deep Gaussian Processes (Deep GP) [19, 24, 26, 27, 30, 32, 46, 79]—where standard neural net layers are replaced with vector-valued Gaussian processes. Indeed, typical neural networks are a degenerate Deep GP subclass [1, 2, 33, 72].
  We therefore have a generalization of neural networks where capacity is controlled, from which we can glean insights about conventional networks that have sufficient representational power for a given modeling task. Surprisingly, despite using Gaussian processes as the primary hierarchical component, we prove that Deep GP converge to (single-layer) GP in their infinite width limit (Thm. 1). Troubling implications immediately ensue: large width is strictly detrimental to Deep GP, as the limiting model collapses to a shallower version of itself. We support this theorem with an analysis of neural network and Deep GP posteriors, which become less adaptable as width increases. Specifically, we show that the posterior mean corresponds to a mixture of functions drawn from data-dependent (and thus adaptive) reproducing kernel Hilbert spaces, formalizing the above claim from Neal [69]. As width increases, this mixture collapses to the data-independent kernel of the limiting GP, implying that wider models have less feature learning. Finally, we present a novel tail analysis which indicates that width and depth have opposite effects: depth accentuates non-Gaussianity, sharpening peaks and fattening tails, whereas width increases Gaussianity (Thms. 2 and 3).
  Our theoretical results hold for Deep GP and conventional (parametric) neural networks alike. Experiments confirm that—after a model achieves sufficient capacity1—width can become harmful to model fit and performance. For nonparametric Deep GP, a width of 1 or 2 often achieves the best performance. Neural networks—because of their parametric nature—naturally require more hidden units before achieving optimal accuracy. Nevertheless, for Bayesian neural networks and conventional (optimized) neural networks trained with L2 regularization, performance degrades after a certain width. On small datasets (N ≤ 1000) with low dimensionality, we find that models with ≤ 16 hidden units achieve best test set performance. On larger datasets like CIFAR10, this “sweet spot” occurs later (at ≈ 500 hidden units for sufficiently deep models), yet performance degrades beyond this width. We note that these trends do not necessarily hold for models that do not have a probabilistic interpretation—i.e. optimized neural networks trained without (or nearly without) L2 regularization. Nevertheless, our findings suggest that narrower models have better inductive biases, and wide models perform well in spite of —not because of—large width.
sections.heading: 2 Setup
sections.heading: 2.1 Related Work
sections.text: Effects of width. Works have shown that, given finite parameters, deeper models are more expressive than wider models [63, 67, 74, 75, 83]. Similarly to our work, Aitchison [2] recognises the link between finite neural networks and Deep GP, and argues that finite neural networks have flexibility in the top-layer representation that is absent in the infinite-width limit. Halverson et al. [45] draw a connection to quantum field theory to argue that neural networks become “simpler” near their infinite-width limit. In the non-probabilistic setting, it is worth noting that wide models have been shown to have favorable optimization landscapes [7, 28, 59, 70, 82] and are resistant to overfitting via double descent [12, 20, 68]. Our work controls for these factors by examining nonparametric hierarchical models with exact Bayesian inference, and thus does not disagree with these other works.
  1We offer a formal notion of “sufficient capacity” in Appx. B.5.
  Infinite width limits have received renewed interest in Bayesian [38, 50, 57, 64, 69, 71, 88] and non-Bayesian [6, 8, 22, 28, 43, 52, 57, 65, 89, 90] settings. Most of these works show that neural networks converge to kernel methods, though recent work suggests that this limiting behavior can be avoided with different parameterizations [e.g. 22, 43, 65, 90]. Similarly to Lee et al. [57], our Deep GP limit analysis sequentially increases the width of each layer, though we hypothesize a similar proof exists where the width of all layers increases simultaneously (akin to [64]).
  Deep GP are introduced by Damianou and Lawrence [27]. A large portion of Deep GP research has thus far focused on scalable approximate inference methods [19, 24, 25, 32, 46, 72, 79, 85]. Though prior work has studied tail properties of neural networks [84, 93] and Deep GP with RBF kernels [62], our work is—to the best of our knowlege—the first general result for Deep GP tails. Duvenaud et al. [33] and Dunlop et al. [29] investigate pathological behaviors that arise with depth, while Agrawal et al. [1] note that “bottlenecked” Deep GP have better performance and correlations among predictive tasks. Our work complements these analysis by characterizing the effects of width.
  Connections between Deep GP and neural networks. Many researchers have noted connections between neural networks and Deep GP [e.g. 24, 31, 36, 61]. Duvenaud et al. [33] suggest that infinitely-wide neural networks with intermediate bottleneck layers are nonparametric Deep GP. Agrawal et al. [1] formalize this connection, but note that not all Deep GP can be constructed from bottlenecked neural networks (see Appx. E). In contrast to these prior works, we avoid reducing Deep GP to neural networks, and instead reduce neural networks to degenerate Deep GP.
sections.heading: 2.2 A Covariance Perspective on Gaussian Process Limiting Behavior
sections.text: To decouple the effects of increasing width and capacity, we first prove a new result about GP limits for a more general class of models, including Deep GP as well as typical neural networks. This result forms a necessary foundation for the subsequent theorems that are a main contribution of this work. To begin, note that the proof technique introduced by Neal [69] and extended by others [38, 50, 56, 64, 71, 88] relies on the multivariate central limit theorem, which requires a model with additive structure. Deep GP do not generally decompose in an additive manner, so we establish a more general proof technique. For simplicity, we first present it in the context of neural networks, and then extend it to a more general class of models.
  Consider the 2-layer neural network f2(f1(x)), with f1 : RD → RH1 and f2 : RH1 → R:
  f1(·) = W>1 (·) + βb1, f2(·) = 1√H1w > 2 σ(·) + βb2. (1)
  σ(·) is an elementwise nonlinearity, β is a positive constant, and W1, b1, w2, and b2 are i.i.d. Normal. With randomly initialized parameters, f2(f1(·)) : RD → R is a prior distribution over functions, and this distribution converges to a GP in the infinite width limit [69].
  Lemma 1. The neural network defined in Eq. (1) is a Gaussian process if and only if—for any finite set of inputs X = [x1, . . . ,xN ]—the conditional prior covariance Ef2|X,W1,b1 [f2f>2 ] is almost surely equal to the marginal prior covariance Ef2|X[f2f>2 ], where f2 | X , [f2(f1(x1)), . . . , f2(f1(xN ))].
  Proof. By definition, f2(f1(·)) is a GP if and only if f2 | X is multivariate Gaussian for any X. From Eq. (1), we have p(f2 | X,W1,b1) = N (0,KW1,b1(X,X)), where [KW1,b1(X,X)]ij = β2+ 1H1σ(W > 1 xi+βb1)
  >σ(W>1 xj +βb1) is the appropriate kernel Gram matrix. Using Jensen’s inequality, we have a lower bound on the characteristic function of f2 | X:
  E f2|X
  [ exp ( it>f2 )] = E
  W1,b1
  [ E
  f2|X,W1,b1
  [ exp ( it>f2 )]] (law of total expectation)
  = EW1,b1 [ exp ( − 12t >KW1,b1(X,X)t )]
  (char. func. of a Gaussian)
  ≥ exp ( − 12t
  > E W1,b1 [KW1,b1(X,X)] t
  ) . (convexity of exp)
  This lower bound happens to be the characteristic function ofN (0,EW1,b1 [KW1,b1(X,X)]). Since exp is strictly convex, the characteristic function of f2 | X equals the Gaussian lower bound ∀t if and only if p(KW1,b1(X,X) |W1,b1) = Ef |X,W1,b1 [f2f>2 ] is a constant with probability 1.
  Seeing that 1H1σ(W > 1 xi + βb1) >σ(W>1 xj + βb1) becomes a.s. constant as H1 →∞, Lemma 1 re-establishes the result of Neal [69] (see Appx. E.1). Critically, unlike Neal’s proof, Lemma 1 neither relies on the central limit theorem nor requires f2(f1(·)) to be a neural network; it holds if p(f2 | f1(x1), . . . , f1(xN )) is Gaussian. Therefore, we can generalize it to a larger class of models:
  Lemma 2. Let f2(f1(·)) : RD → R be a hierarchical model where f2(·) : RH1 → R is a GP and f1(·) : RD → RH1 is a random vector-valued function (including a multilayer hierarchical model). Then f2(f1(·)) is a GP if and only if Ef2|X,f1(·)[f2f>2 ] = Ef2|X[f2f>2 ] a.s. for all X = [x1, . . . ,xN ].
  The covariance perspective from Lemmas 1 and 2 is revealing about GP limits. As Ef2|X,f1(·)[f2f>2 ] converges to Ef2|X[f2f>2 ] the model output becomes less and less dependent on f1(·). In other words, f2(f1(·)) loses its hierarchical nature. We reiterate that Lemma 2 has no requirements about f2(f1(·)) transitioning from a finite to infinite basis, nor does it require f2(f1(·)) to have additive structure. We demonstrate its generality in the next section with surprising—and troubling—implications.
sections.heading: 3 Deep Gaussian Processes Collapse to Shallow Gaussian Processes
sections.text: Deep GP [19, 26, 27, 79] are hierarchical models where layers f1(·) . . . fL(·) are (vector-valued) GP:
  DGP(x) = fL ◦ . . . ◦ f1 (x) , fi(·) = [f (1)i (·), . . . , f (Hi) i (·)], f (j) i (·) i.i.d∼ GP [0, ki(·, ·)] . (2)
  Hi is the width of the ith GP layer, and the output dimensions of each fi(·) are independent. By using GP as the primary hierarchical building blocks, Deep GP are generally nonparametric and, assuming the GP layers use universal kernels [66], have infinite representational capacity (see Appx. B.1).
  Deep GP versus GP. Deep GP seek to offer more expressivity: conventional single-layer GP—though also nonparametric—are inherently limited by the choice of the prior covariance function [19, 79]. For example, a GP with a RBF covariance is not suitable for data with discontinuities or sharp changes. However, stacking two RBF GP together—f2(f1(·))—can overcome this limitation, since f1(x) can encode a warping of x that “smoothes” the input data for f2(·) (as we will show in Fig. 1). Empirically, Deep GP have been shown to offer much more accurate predictive posteriors than standard GP [e.g. 17, 24, 26, 27, 30, 46, 79].
  Deep GP versus neural networks. (Bayesian) feed-forward neural networks are a strict subclass of Deep GP, albeit a degenerate one [2, 61, 72]. The first neural network layer is a GP with a linear kernel, while subsequent layers are GP with the kernel k(z, z′) = β2 + 1Hi−1 ∑Hi−1 i=1 σ(zi) σ(zi). A neural network, unlike other Deep GP, does not have infinite capacity. Put loosely, a single neural network hidden unit corresponds to a single basis, while in general a single Deep GP unit corresponds to a potentially-infinite basis. See [1, 2, 24, 31, 33, 72] and Appx. B.2 for more discussion on this connection. The critical takeaway is that all of our Deep GP results apply to neural networks as well.
sections.heading: 3.1 Wide Deep GP are Gaussian Processes
sections.text: Having established a model where width does not effect capacity, we now establish what remaining effects width has. Empirical evidence suggests that the choice of width impacts Deep GP predictions [19, 46]. In practice it is common to make Deep GP as wide as comparably-sized neural networks; Salimbeni and Deisenroth [79] for example train Deep GP with ≥ 30 units per layer. Surprisingly, here we prove that—in the limit of infinite width—Deep GP collapse to single-layer Gaussian processes. Our proof relies on the conditional covariance analysis of the previous section. If the GP layers have non-pathological covariance functions2—the Deep GP conditional covariance becomes almost surely constant with width (see Lemma 3, Appx. E). Combining this with Lemma 2:
  Theorem 1. Let fL ◦ . . .◦ f1 (x) be a zero-mean Deep GP (Eq. 2), where each layer satisfies Assumptions 1 and 2 (non-pathological prior covariances that scale with dimensionality—see Appx. E.3). Then limHL−1→∞ · · · limH1→∞ fL ◦ . . . ◦ f1 (x) converges in distribution to a (single-layer) GP.
  2Any textbook kernel (isotropic kernels, dot product kernels, etc.) or any covariance function with a Fourier-Steiljes representation is “non-pathological;” see Appx. E.3 for formal assumptions.
  (See Appx. E for proof.) The implications of Thm. 1 are paradoxical and unsettling. Deep GP are motivated as a more powerful model than standard GP. However, as we make the model wider, we arrive back where we started—a Gaussian process (although one with a different prior covariance).
  A neural network gains representational power in its GP limit, transitioning from a finite-basis model to a nonparametric model. The Deep GP limit on the other hand has no additional representational power, since Deep GP are already universal approximators at any width. (Indeed this fact motivates their use as a control.) The only difference between finite and infinite width Deep GP is the prior distribution itself: transitioning from non-Gaussian to Gaussian with increasing width. In the next section, we investigate how this transition affects model performance.
sections.heading: 4 Large Width Limits the Adaptability of Hierarchical Posteriors
sections.text: Even with Thm. 1 and its troubling suggestions, it is not immediately clear exactly what is lost in the infinite-width limit. Here, we quantify specific differences in the predictive capabilities of narrow versus wide models. In particular, we analyze Deep GP/neural network posterior distributions, rather than focusing on a single model trained through optimization. We show that these posteriors correspond to a mixture of data-dependent adaptable bases; however, as width increases this mixture collapses to the (data-independent) basis of the limiting GP. This result formalizes the often vague notion of feature learning, and demonstrates that it is indeed lost in kernel limits.
  Hierarchical posteriors correspond to a data-adaptable bases. Consider the (finite-width) 2- layer Deep GP f2(f1(·)), where k1(·, ·) and k2(·, ·) are the covariance functions of f1(·) and f2(·). Given training data X,y, define F1 , [f1(x1), . . . , f1(xN )] and f2 , [f2(f1(x1)), . . . , f2(f1(xN ))]. Let x∗ be a test input, and let f∗1 and f ∗ 2 equal f1(x
  ∗) and f2(f1(x∗)) (see Fig. 5 in Appx. B.3 for a graphical model). Crucially, f2 and f∗2 only depend on F1 and f ∗ 1 through the covariances K2(F1,F1), k2(F1, f∗1 ), and k2(f ∗ 1 , f ∗ 1 ) (which we abbreviate as K2, k ∗ 2, and k ∗∗ 2 ):
  p(f2 | K2) ∼ N (0,K2) , p(f∗2 | k∗∗2 ,k∗2,K2, f2) ∼ N ( k∗>2 K −1 2 f2, k ∗∗ 2 − k∗>2 K−12 k∗2 ) ,
  By D-separation [e.g. 16, Ch. 8], we can factorize the posterior distribution as:
  p(f∗2 , f2,K2,k ∗ 2, k ∗∗ 2 | y) = p(f∗2 | f2,K2,k∗2, k∗∗2 ) p(f2 | K2,y) p(K2,k∗2, k∗∗2 | y). (3)
  See derivation in Appx. B.3. Applying the factorization in Eq. (3), the posterior mean is:
  E f∗2 |y [f∗2 ] = E K2,k∗2 |y
  [ E
  f2|K2,y
  [ k∗>2 K −1 2 f2 ]] = E
  K2,k∗2 |y
  [ k∗>2
  α
  K−12 E f2|K2,y
  [f2] ]
  (4)
  = E f1(x∗),f1(x1),...,f1(xN )|y
  [∑N i=1 αi k2(f1(xi), f1(x ∗)) ] , (5)
  where the second line follows from K2 and k∗2 being deterministic given f1(x ∗), f1(x1), . . ., f1(xN ). The term inside the Eq. (5) expectation is a function from the reproducing kernel Hilbert space (RKHS) defined by k2(f1(·), f1(·)). We can thus interpret this expectation as an infinite mixture of functions from different Hilbert spaces. Because the mixture distribution p(f1(x∗), f1(x1), . . . , f1(xN ) | y) depends on y, Eq. (5) is an adaptive data-dependent mixture of RKHS.
  Adaptability is lost in the Gaussian process limit. What happens to Eq. (5) as f2(f1(·)) becomes a Gaussian process in the limit of infinite-width? Recall from Lemma 2 that the conditional prior covariance becomes deterministic as f2(f1(·)) converges to a GP. In other words, the prior and posterior distributions over K2 and k∗2 become atomic: p(K2,k ∗ 2) = p(K2,k ∗ 2 | y) = δ [Klim,k∗lim ], where Klim and k∗lim are shorthand for E[f2f>2 ] and E[f2f∗2 ] respectively. Eq. (4) thus collapses to:
  lim H1→∞ E f∗2 |y [f∗2 ] = E δ[Klim,k∗lim]
  [ k∗>2 α ] = ∑N i=1 αi klim(xi,x ∗), (6)
  which is no longer a mixture of functions from different RKHS. It is instead a function from a single RKHS (that of the limiting GP prior).3 In other words, while Deep GP (and neural networks) perform kernel learning (or feature learning) to adapt to training data, this ability is lost with large width.
  Example. Consider a Deep GP with RBF covariances k1(x,x′) = exp ( −‖x− x′‖2/(2D) ) and k2(f1(x), f1(x ′)) = exp ( −‖f1(x)− f1(x′)‖2/(2H1) ) . As we show in Appx. G, this Deep GP converges to a GP with klim(x,x′) = exp(exp(−‖x − x′‖2/(2D)) − 1). Note that this limiting covariance is stationary and is ill-equipped to model the data step in Fig. 1. However, because f1(·) is nonlinear, k2(f1(x), f1(x′)) is nonstationary. Fig. 1 (top left) shows that the width-1 Deep GP posterior accurately models this data. The posterior covariance Ef1(x),f1(x′)|y[k2(f1(x), f1(x′))] (bottom left) features long-range correlations near x = ±1 and short-range correlations near x = 0. As width increases, we lose this nonstationarity and the posterior becomes a worse fit.
sections.heading: 5 The Difference Between Width and Depth: A Tail Analysis
sections.text: Our work so far has troubling implications for large width. On the other hand, empirical evidence has shown that depth improves Deep GP performance—as it does for neural nets [e.g. 46, 72, 79] (though pathologies can emerge [29, 33]). Through a novel tail analysis, we show that width makes Deep GP priors more Gaussian, while depth makes them less Gaussian. In other words, width and depth have opposite effects on Deep GP tails, results that again also apply to typical neural networks.
  Deep GP/neural networks are sharply peaked and heavy tailed. The proof technique used in Lemma 1 can be used to similarly bound the moment generating function of Deep GP marginals:
  E f2
  [ et >f2 ] = E
  F1
  [ E
  f2|F1
  [ et >f2 ]] ≥ exp ( 1
  2 t> E F1 [K2(F1,F1)] t
  ) = E
  g∼N (0,Klim)
  [ et >g ] , (7)
  where Klim=Ef2 [f2f>2 ]=EF1 [K2(F1,F1)]. Generalizing these bounds to deeper models, we have: Theorem 2. Let fL◦. . .◦f1(·) be a zero-mean Deep GP. Given a finite set of inputs X = [x1, . . . ,xN ], define f` = [(f` ◦ . . . ◦ f1(x1)), . . . , (f` ◦ . . . ◦ f1(xN ))] for ` ∈ [1, L], and define Klim = EfL [fLf>L ]. Then, p(fL = 0) ≥ N (g = 0;0,Klim). Theorem 3. Let t ∈ RN . Using the same setup, notation, and assumptions as Thm. 2, the odd moments of t>fL are zero and the even moments larger than 2 are super-Gaussian, i.e. EfL [(t>fL)r] ≥ Eg∼N (0,Klim)[(t>g)r] for all even r ≥ 4. Moreover, if kL(·, ·) is bounded almost everywhere, the moment generating function EfL [exp(t>fL)] exists and is similarly super-Gaussian.
  (See Appx. F for proofs.) Thm. 2 states that Deep GP marginals are more sharply peaked than a moment-matched Gaussian, while Thm. 3 states that they are also more heavy tailed.
  Increasing depth leads to sharper peaks and heavier tails. To understand how depth affects this tail behavior, we examine the Jensen gap in Eq. (7). Consider a 3-layer Deep GP f3(f2(f1(·))). If we
  3To rigorously argue that the infinite-width posterior collapses in this way, we can invoke Proposition 1 from Hron et al. [49]. See Appx. B.4 for details.
  extend Eq. (7) to 3-layer models, we see that the Jensen gap cascades:
  E F1
  [ E
  F2|F1
  [ exp ( 1
  2 t>K3t )]] MGF of 3-layer Deep GP marginal ≥ E F1 [ exp ( 1 2 t> E F2|F1 [K3] t )] MGF of 2-layer Deep GP marginal ≥ exp ( 1 2 t>E F1 [ E F2|F1 [K3] ] t ) MGF ofN (0,EF1 [EF2|F1 [K3]]) ,
  where K3 is short for K3(F2(F1(X)),F2(F1(X))). The middle term is the moment generating function of a 2-layer Deep GP marginal (where the second layer has covariance Ef2(·)[k3(f2(·), f2(·))]). The right-most term is the moment generating function of a (single-layer) Gaussian. Generalizing this cascade, we see that deeper models are more heavy-tailed. A similar analysis on the characteristic function shows that the peak at the prior mean also becomes sharper with depth (see Appx. F).
  Adding additional layers to a Deep GP will change the model’s prior covariance, and thus the effects of depth cannot solely be explained by a tail analysis [29, 33]. Nevertheless, if we control for this change in covariance, we indeed see that depth leads to heavier tails. In Fig. 2 we compare 2-layer and 3-layer Deep GP. The 3-layer models use GP layers with additively-decomposing RBF covariances, while the 2-layer models use layers constructed to match the 3-layer models’ prior covariance (see Appx. H for construction details). The N = 2 marginal densities for the 3-layer models (bottom row) are more stretched than the 2-layer densities (top row). We further confirm these effects in Appx. D.
  Increasing width leads to flatter peaks and Gaussian tails. Conversely, consider what happens when we make the model wider. We define the sequence of increasingly wide 2-layer Deep GP:{
  DGP(m)(·) , 1√ m m∑ i=1 f (i) 2 (f (i) 1 (·))
  } ,
  f (i) 1 (·) i.i.d∼ GP [0, k1(·, ·)] ,
  f (i) 2 (·) i.i.d∼ GP [0, k2(·, ·)] . (8)
  DGP(m)(·) is a width-m Deep GP, where the second layer decomposes additively over the m dimensions. By linearity of expectation, each model in the sequence shares the same prior covariance: E[DGP(1)(x) DGP(1)(x′)] = E[DGP(2)(x) DGP(2)(x′)] = . . . , klim(x,x′). Though each model has the same marginal covariance, the conditional covariance Ef2|F1 [f2f>2 ] = 1 m ∑m i=1 K2(f (i) 1 , f (i) 1 ) becomes increasingly concentrated around Klim(X,X) as m increases. This consequentially shrinks the Jensen gap in Eq. (7), and so the Deep GP marginals become increasingly Gaussian. We again visualize this effect in Fig. 2, which depicts marginal densities from 2-layer and 3-layer Deep GP of various width (see Appx. H for details). Compared with the limiting GP (right), the width-1 densities (left) appear sharper near [0, 0] and more stretched at the tails. As width increases, the peaks and tails look increasingly Gaussian (see also Fig. 7 in Appx. D). In this sense, width has the opposite effect as depth—deeper marginals are less Gaussian, while wider marginals are more Gaussian.
sections.heading: 6 Experiments
sections.heading: 6.1 Regression with Deep GP and Bayesian Neural Networks
sections.text: To isolate the effects of width and depth, each experiment compares Deep GP/Bayesian neural networks that share the same first and second prior moments, and the Deep GP models use GP layers with universal kernels. To remove any potential side effects from approximate inference methods, we sample Deep GP/neural network posteriors using NUTS [48] and do not use any stochastic inducing point [46, 79] or finite basis [24] approximations. This inference is costly and scales cubically with N ; therefore, we subsample all training datasets to N ≤ 1000. See Appx. H for experimental details.
  Effect of width. We compare 2-layer Deep GP of various width on 6 regression datasets from the UCI dataset repository [10] (see Appx. D for 3-layer results). The first GP layers use a RBF kernel for the prior covariance, while the second layers use a sum of one-dimensional RBF covariance functions. We additionally compare against the limiting (single-layer) GP with the same prior covariance (Lim. GP). For each dataset, we choose hyperparameters that maximize the Lim. GP log marginal likelihood. In Fig. 3 (top row) we see a near-monotonic performance degradation as width increases. The width = 2 optimum may represent the “sweet spot” for Deep GP width, but it may instead be a side-effect of inference difficulties for width = 1 models (see Appx. D for a control experiment). Regardless, as our theory predicts, width is detrimental to Deep GP predictive performance.
  We repeat the experiment for 2-layer neural networks (and 3-layer models in Appx. D), where here the Lim. GP corresponds to the arc-cosine kernel [23, 56]. Fig. 3 indicates an optimal width with regards to test set log likelihood, usually between 8− 16 hidden units. We expect this optimum exists (and differs from the Deep GP optimum) because narrow models have too few basis functions for these datasets. Nevertheless, after sufficient capacity, width is harmful to Bayesian neural networks.
  Adaptable versus non-adaptable RKHS. One way to measure the “fit” of a kernel k(·, ·) on a regression training dataset X,y is the Gaussian log marginal likelihood logN (y;0,K(X,X)+σ2I), where σ2 is an observational noise parameter [e.g. 77]. To demonstrate how Deep GP/neural network
  posteriors correspond to adaptable RKHS mixtures, the bottom row of Fig. 3 plots the “kernel fit” of k2(f1(·), f1(·)) for posterior samples of f1(·) (see Eq. 5). A higher fit corresponds to a model that is better adapted to the dataset X,y. We see that narrower Deep GP almost universally achieve better kernel fit than wider Deep GP, which converge to the same fit as the limiting GP. (Standard deviations, depicted by shaded regions, are generally imperceptible.) Bayesian neural networks achieve best “kernel fit” at 8− 16 hidden units, and then converge to the limiting Deep GP with further width.
  Effect of depth, controlling for covariance. Table 1 displays Deep GP test set log likelihood as a function of depth. Again, we isolate the tail effects of depth by ensuring that all models share the same first and second moments. We construct a GP and a 2-layer Deep GP that match the moments of a 3-layer width = 2 Deep GP with RBF covariances, and we use hyperparameters that maximize the limiting GP marginal likelihood for each dataset. Note that computing the limiting covariance of ≥ 3 layer models involves intractable integrals that we approximate with quadrature (see Appx. G). Our findings confirm that—in this controlled setting—depth unlike width improves test set performance.
sections.heading: 6.2 Standard (Optimized, Non-Bayesian) Neural Networks
sections.text: We now turn to standard (optimized, non-Bayesian) neural networks. While our theoretical results primarily apply to full posteriors over models, our goal is to see if our theory can also be predictive in “real world” neural networks without a Bayesian treatment. There is reason to believe that our theory should be applicable in these settings, since standard neural network training with L2 regularization is equivalent to maximum a posteriori inference with Gaussian priors. To that end, we ensure some correspondence between these experiments and our Bayesian experiments. In particular, we measure the effects of width on networks with fixed values of L2 regularization,4 which corresponds to a fixed prior on neural network parameters. Additionally, models are trained without data augmentation, as data augmentation does not have a probabilistic interpretation [51].
  Fig. 4 (left) depicts test set accuracy for increasingly wide models trained on MNIST [55]. Each network is a MLP with 3 layers (i.e. 2 hidden layers). Following the GP-limiting neural network construction in Eq. (1), we scale the outputs of layer ` by 1/ √ H`−1. We measure the effect of width over networks with various L2 regularization constants (10−5, 10−6, 10−7, and 10−8) which respectively correspond to priors of N (0, 2), N (0, 20), N (0, 200), and N (0, 2000) when N = 50,000. We train these sequences on various-sized subsets of the training data (N = 500, N = 5,000, and N = 50,000). From this figure we can observe several phenomena. For larger values of L2 regularization, we see a distinct maximum in accuracy, typically around width ≈ 1,000. For smaller values of L2 regularization, wider models tend to perform better (and indeed, for this dataset/model combination it appears that less regularization tends to be beneficial to overall performance). We would note that these low regularization constants correspond to arguably unreasonable parametric
  4In other words, we do not consider the regularization constant to be a hyperparameter that we optimize over for the purposes of these experiments.
  priors like N (0, 2000), and so a Bayesian interpretation of these models may not be applicable. In such settings, it is more likely that the interpolation analysis of Belkin et al. [12] is a better model of performance, since this analysis explicitly focuses on the low-regularization setting.
  Fig. 4 (right) depicts 8- and 14-layer ResNets [47] trained on the CIFAR-10 dataset [54]. We use the hyperparameters from the original ResNet paper, which have been shown to be efficacious on both narrow and wide variants of ResNet models [92]. (This training procedure uses a L2 coefficient of 10−4, which corresponds to a prior of N (0, 0.2) for each parameter when N = 50,000.) For both depths, we observe that performance is optimal when width is between 500 and 1,000. While it is possible that different hyperparameters may yield different outcomes, these results indeed suggest that large width can adversely affect standard neural networks once sufficient capacity is reached.
sections.heading: 7 Discussion
sections.text: This paper shows that, across typical neural networks (with L2 regularization), Deep GP, and Bayesian neural networks, large width can be detrimental to model performance.
  Even with these results, we can ask when width might be desirable? First, we note that our results analyze exact posteriors or MAP solutions, and does not focus on practical considerations with regards to obtaining these solutions. We do not consider the effect that width might have on approximate inference methods, which are commonly used with Bayesian neural networks and Deep GP in practice [e.g. 18, 34, 79]. For conventional neural networks, poor conditioning and non-convexity make it challenging to obtain a MAP solution. The optimization dynamics—which depend on numerous factors like learning rates, initializations, and choice of optimizer—may be improved by width, as wider models tend to have more favorable optimization landscapes [7, 28, 59, 70, 82]. Consequentially, wider models may obtain better performance due to these practical considerations.
  Secondly—as noted in Sec. 6.2—while we notice detrimental effects of width on neural networks with a Bayesian interpretation (i.e. inferring a parameter posterior or optimizing parameters with L2 regularization), we do not see these effects when such an interpretation does not exist (i.e. optimizing parameters with almost no L2 regularization). Our theoretical findings assume that layers are conditionally Gaussian, and different priors may have different effects. We note that much of the preliminary works on NTK assume no explicit regularization during training [28, 52, 57] (with the notable exception of Wei et al. [86]), and so our findings may be at odds with the empirical findings around these models [e.g. 9, 38, 39]. Moreover, recent work has proposed (non-Bayesian) infinite-width constructions that avoid any limiting kernel behavior [e.g. 22, 43, 65, 90], and so our findings would not apply to these models. We emphasize that our results do not conflict with these prior works, but rather reflect a different perspective. The models we study correspond to a Gaussian prior on parameters, and so relaxing this correspondence may lessen the consequences of width that we observe. Nevertheless, our results suggest that the inductive bias of width may be harmful, even if these undesirable effects can be avoided via careful construction.
  Finally, it is worth considering when one might still choose a conventional shallow GP over a deep model. An often-touted benefit of Gaussian processes is the ability to encode prior domain knowledge via the choice of covariance function. In Appx. C, we prove that certain prior covariances cannot be expressed by adaptable hierarchical models. For example, a Deep GP that is composed of stationary GP layers cannot model anti-correlations a priori (Thm. 4, Appx. C), whereas (single-layer) stationary GP can have positive and negative prior covariances. Nevertheless, Deep GP are capable of modeling many common covariance functions, including the RBF, Matérn, and rational quadratic kernels. In Appx. C we demonstrate a 2-layer Deep GP construction of any width that is capable of producing prior covariances that match most isotropic kernels (Thm. 5, Appx. C). In other words, a Deep GP can match the first and second moments of most GP, while also offering an adaptable posterior.
  Acknowledgments and Disclosure of Funding
  We would like to thank Elliott Gordon-Rodriguez for his help with the proofs. This work was supported by the Simons Foundation, McKnight Foundation, the Grossman Center, and the Gatsby Charitable Trust.
title: The Limitations of Large Width in Neural Networks: A Deep Gaussian Process Perspective