Porcupine Neural Networks: Approximating Neural Network Landscapes
Neural networks have been used prominently in several machine learning and statistics applications. In general, the underlying optimization of neural networks is non-convex which makes analyzing their performance challenging. In this paper, we take another approach to this problem by constraining the network such that the corresponding optimization landscape has good theoretical properties without significantly compromising performance. In particular, for two-layer neural networks we introduce Porcupine Neural Networks (PNNs) whose weight vectors are constrained to lie over a finite set of lines. We show that most local optima of PNN optimizations are global while we have a characterization of regions where bad local optimizers may exist. Moreover, our theoretical and empirical results suggest that an unconstrained neural network can be approximated using a polynomially-large PNN.
conditions for a local minimum to be global in the case of having regularization on the network weights.
We review these prior work in SM 1 Section 2. In this paper, we study a key question: can an unconstrained neural network be approximated with a constrained one whose optimization landscape has good theoretical properties? For two-layer neural networks, we provide an affirmative answer to this question by introducing a family of constrained neural networks which we refer to as Porcupine Neural Networks (PNNs) (Figure 1-a). In PNNs, incoming weight vectors to hidden neurons are constrained to lie on a fixed set of lines. For example, a neural network with multiple inputs and multiple neurons where each neuron is connected to one input is a PNN since input weight vectors to neurons lie on lines parallel to the standard axes (SM Section 3).
Designing new objective functions with good landscape properties for non-convex problems has been explored previously in [19,15,24]. Our work provides an alternative procedure to obtain a new objective function for the underlying non-convex optimization. Our framework can be extended naturally to deeper networks as well even though in our analysis in this paper, we focus on two-layer networks. Moreover, a PNN can be viewed as a neural network whose feature vectors (i.e., input weight vectors to neurons) are fixed up to scalings due to the PNN optimization. This view can relate a random PNN (i.e., a PNN whose lines are random) to the application of random features in kernel machines [25,26,27,28,29,30,6]. Although our results in Sections 4 and 5 are for general PNNs, we study them for random PNNs in Section 6 as well.
We analyze population risk landscapes of two-layer PNNs with jointly Gaussian inputs and rectified linear unit (ReLU) activation functions at hidden neurons. We show that under some modeling assumptions, most local optima of PNN optimizations are also global optimizers. Moreover, we characterize the parameter regions where bad local optima (i.e., local optimizers that are not global) may exist.
Next, we study whether one can approximate an unconstrained (fully-connected) two-layer neural network function with a PNN whose number of neurons is polynomially-large in dimension. Our empirical results offer an affirmative answer to this question. For example, suppose the output data is generated using an unconstrained two-layer neural network with d = 15 inputs and k * = 20 hidden neurons. Using this data, we train a random two-layer PNN with k hidden neurons. We evaluate the PNN approximation error as the mean-squared error (MSE) normalized by the L 2 norm of the output samples in a two-fold cross validation setup. As depicted in Figure 1-b, by increasing the number of PNN hidden neurons k, the PNN approximation error decreases. Notably, to obtain a relatively small approximation error, PNN's number of hidden neurons does not need to be exponentially large in dimension. We explain details of this experiment in Section 7.
In Section 6, we study a characterization of the PNN approximation error with respect to the input dimension and the complexity of the unconstrained neural network function. We show that under some modeling assumptions, the PNN approximation error can be bounded by the spectral norm of the generalized Schur complement of a kernel matrix. We analyze this bound for random PNNs in the high-dimensional regime when the ground-truth data is generated using an unconstrained neural network with random weights. For the case where the dimension of input and the number of hidden neurons increase with the same rate, we compute the asymptotic limit. Finally, in Section 8, we discuss how the proposed PNN framework can potentially be used to explain the success of local search methods such as gradient descent in solving the unconstrained neural network optimization.
In summary, PNNs provide three main advantageous. The first is that its loss landscape has nice properties with provably few bad locals. Thus, we can have some guarantees on the performance of gradient descent. Second, its approximation power is good, i.e. our experimental and theoretical results suggest that one can approximate unconstrained networks arbitrarily closely with polynomiallylarge PNNs. Third, PNN uses fewer parameters than unconstrained neural networks, thus leading to better generalization and also benefits in terms of storage. Finally, we discuss using the PNN framework in general deep neural networks in Section 8.
This paper presents a new type of neural network where the weights are constrained to lie over a fixed set of lines (called PPN lines). Although this reduces the capacity of the network, it still yields a high-capacity network and more importantly, it “simplifies” the energy landscape making it easier for an optimizer to find the optimum weights. I do have some comments, mostly regarding the relation to previous work or some technical part in the analysis. I will consider raising my scores if these questions are addressed in the rebuttal.   1) Comparison to existing approaches I think the approach presented by the authors is interesting as it offers some improvements over past approaches. As pointed out by the authors, most existing work assume the number of neurons is less than the dimensions. However, there are exceptions and I think it’s worth pointing this out. For instance, Soltanolkotabi et al. (2017) already focused on the (more reasonable) over-parametrized regime where n < O(d).  2) Comparison to other architectures such as resnet or densenet The authors essentially suggest a way to reduce the capacity of the network. There are already many ways to modify the architecture of a neural network to achieve similar effects such as using residual or skip connections. See e.g. https://arxiv.org/pdf/1712.09913.pdf for a discussion of the effect of these connections on the landscape of a neural network.  3) Characterization global minima Vidal et al. (https://arxiv.org/pdf/1712.04741.pdf) gave conditions for global optimality of a neural network (related to the number of zero weights). This seems to relate to the conditions given in Theorem 2, can you comment on this?  4) Choice of PNN lines How important is the choice of PNN lines on which your project the weights? For instance, would you get similar guarantees with random projections? Would this simply affect the eigenvalues of the kernel function?  5) Gaussian input “We analyze population risk landscapes of two-layer PNNs with jointly Gaussian inputs” It’s not clear to me where in the analysis you require the Gaussian input? Can you please comment on this?  6) Choice of activation function The analysis relies on the a rectified linear unit (ReLU) activation function. How important is this choice? Can you generalize it to other types of activation functions?  7) Extension to >2 hidden layers Can the authors comment on extending their approach to more than 2 layers?  8) Proof of Theorem 2 for characterization of global optima You seem to only consider the fact that the norm of the gradient should be zero. Aren’t you only consider first-order critical points? What about the eigenvalues of the Hessian? In theorem 3, you said the Hessian is positive semidefinite which does not rule out zero-eigenvalues (i.e. degenerate points).  9) Global optimizer The authors seem to assume the existence of one global minimizer in all their theorems, even in the case where every local optimizer it not a global optimizer. For instance, take Theorem 1 in the case where the conditions of Theorem 2 are not satisfied.  10) Effect of variables d (input dimension) and r (number of PNN lines) As pointed out by the authors, Theorem 5 does not characterize the rate of the decrease as a function of d (and r). I wonder if the authors have conducted some experiments to see the empirical effect of d and r. From a theory point of view, only the regime d->infty is analyzed in the paper. From an practitioner point of view, I think it would really be interesting to understand how large r should be to reach a good level of performance.  11) MNIST experiments: The authors only use a subset of MNIST (they only consider images of 1’s and 2’s). Can you elaborate on this choice? Where is the difficulty in extending this to >2 classes?  12) Appendix The numbering scheme used in the appendix is different from the main paper, sometime making it difficult to find the proof of a theorem. a) For instance, where is the proof of Theorem 1 in the appendix? The closest seems to be theorem 4 where the form of the Kernel C is much simpler that given in the main paper. Can you explain how you derive the form of the kernel giving in Theorem 1 in the main paper? b) Similar remarks for other theorems. I think you should consider re-organizing the supplementary material to make it easier for the reader to find the proofs of the theorems cited in the paper.