The committee machine: Computational to statistical gaps in learning a two-layers neural network
Heuristic tools from statistical physics have been used in the past to locate the phase transitions and compute the optimal learning and generalization errors in the teacher-student scenario in multi-layer neural networks. In this contribution, we provide a rigorous justification of these approaches for a two-layers neural network model called the committee machine. We also introduce a version of the approximate message passing (AMP) algorithm for the committee machine that allows to perform optimal learning in polynomial time for a large set of parameters. We find that there are regimes in which a low generalization error is information-theoretically achievable while the AMP algorithm fails to deliver it; strongly suggesting that no efficient algorithm exists for those cases, and unveiling a large computational gap.
While the traditional approach to learning and generalization follows the Vapnik-Chervonenkis [1] and Rademacher [2] worst-case type bounds, there has been a considerable body of theoretical work on calculating the generalization ability of neural networks for data arising from a probabilistic model within the framework of statistical mechanics [3,4,5,6,7]. In the wake of the need to understand the effectiveness of neural networks and also the limitations of the classical approaches [8], it is of interest to revisit the results that have emerged thanks to the physics perspective. This direction is currently experiencing a strong revival, see e.g. [9,10,11,12].
Of particular interest is the so-called teacher-student approach, where labels are generated by feeding i.i.d. random samples to a neural network architecture (the teacher) and are then presented to another neural network (the student) that is trained using these data. Early studies computed the information theoretic limitations of the supervised learning abilities of the teacher weights by a student who is given m independent n-dimensional examples with α ≡ m/n = Θ(1) and n → ∞ [3,4,7]. These works relied on non-rigorous heuristic approaches, such as the replica and cavity methods [13,14]. Additionally no provably efficient algorithm was provided to achieve the predicted learning abilities, and it was thus difficult to test those predictions, or to assess the computational difficulty.
Recent developments in statistical estimation and information theory -in particular of approximate message passing algorithms (AMP) [15,16,17,18], and a rigorous proof of the replica formula for the optimal generalization error [11]-allowed to settle these two missing points for single-layer neural networks (i.e. without any hidden variables). In the present paper, we leverage on these works, and provide rigorous asymptotic predictions and corresponding message passing algorithm for a class of two-layers networks.
The committee machine is a simple and natural model for a 2-layer neural network. (Here is the formal definition: it is a function R^n -> {+1,-1}, computed by taking K different linear combinations of the n inputs according to the weight matrix W (this is the hidden layer), then taking the sign of each hidden value, and then taking the majority vote of all these signs. The results of the paper also apply to many related models: you are allowed an arbitrary function mapping the K hidden values to the final binary output.)  This paper studies the problem of learning the weights W under a natural random model. We are given m random examples (X,Y) where the input X (in R^n) is iid Gaussian and Y (in {+1,-1}) is the associated output of the network. The unknown weights W are iid from a known prior. The authors study the asymptotic regime where the dimension n goes to infinity with the ratio alpha = m/n held constant (recall m is the number of samples) and the number of hidden layers K held constant.  Since the above is a fully-specified random model, it is amenable to study via tools from statistical physics. Prior work has rigorously analyzed the simpler case of "generalized linear models" which includes single-layer neural networks (K=1). Prior work has also analyzed the two-layer case using non-rigorous heuristics from statistical physics (which are well-established and widely believed to be reliable). The first contribution of the present work is to rigorously analyze the two-layer case (showing that the heuristics are indeed correct). Specifically, they give an exact formula for the limiting value (as n goes to infinity) of the information-theoretically optimal generalization error as a function of the sample complexity (alpha = m/n). The proof uses the "adaptive interpolation method", a recent method that has been successful for related problems.  The second contribution of the present work is to give a polynomial-time algorithm for learning the weights W based on AMP (approximate message passing). They also give an exact formula for the generalization error (as n goes to infinity) that it achieves. AMP is a well-established framework and is widely conjectured to be optimal among all polynomial-time algorithms for this type of problem. By comparing the formulas for AMP's performance and the information-theoretically optimal performance, the authors identify regimes that appear to have inherent statistical-to-computational gaps. The authors give a thorough investigation of these gaps and of the various "phase transitions" occurring in the problem.  I think this is a strong paper that builds on a lot of deep and powerful machinery from statistical physics in order to give important and fundamental advances towards a rigorous understanding of neural networks. The paper is well-written. I would be interested if the authors could comment on whether there is any hope of extending the results to more than two layers.  EDIT: I have read the other reviews and the author feedback. My opinion of the paper remains the same -- I vote to accept it.