On the Computational Efficiency of Training Neural Networks
It is well-known that neural networks are computationally hard to train. On the other hand, in practice, modern day neural networks are trained efficiently using SGD and a variety of tricks that include different activation functions (e.g. ReLU), over-specification (i.e., train networks which are larger than needed), and regularization. In this paper we revisit the computational complexity of training neural networks from a modern perspective. We provide both positive and negative results, some of them yield new provably efficient and practical algorithms for training certain types of neural networks.
The paper considers computational and theoretical questions around training multi-layer neural networks. In particular, the authors consider, in some restricted settings, when and how efficiently good solutions may be obtained. Both positive and negative results are presented. The identification of appropriate settings under which a useful analysis can be undertaken is itself a result.Overall, the paper is clear and well-structured. Of the theory literature around neural networks, this effort is among the more grounded in reality, and more accessible. The paper includes several results which “sort of” flow together, giving the slightest hint of a manuscript that collected partially related pieces together into a whole. But, on balance the work is coherent enough. This reviewer did not check proofs in the supplementary material, however the general reasoning and flow of ideas presented in the paper itself appear to be sound. The theoretical results do seem interesting on the whole, although Thm 2 and Cor. 1 may be subsumed in existing results, or may be incremental in nature. The authors spend a good deal of time studying networks induced by the quadratic activation function, which is non-standard, however they crucially try to understand how networks with this nonlinearity might be compared to networks defined from more widely adopted nonlinearities (such as the sigmoid). Experimental validation is lacking: the authors ran out of space, and experiments are more of an afterthought. In a sub-field that is primarily driven by practical applications and experimental successes, experiments should probably feature prominently. One of the main contributions and selling-points of the paper is the “efficient algorithms” for training certain depth-2 and -3 networks, but it is hard to evaluate these methods. Limited experimental detail is provided, and only depth-2 networks are explored in a narrow context. The evidence given, however, appears positive. Weaknesses of the algorithms are not discussed in sufficient detail. More information about the experimental comparisons should be given, even if in the supplementary material, so that an interested reader can understand whether the baseline networks are realistic.In the absence of more information and a sharpened impact message/result, the “practical” algorithms suggested by the theory may be of  limited use. Inertia in this field seems strong, and many researchers might very well happily continue to use networks defined by sigmoid and rectification nonlinearities trained with vanilla  SGD, at any depth they prefer. In this sense, the theory results may be viewed as the dominant contributions by which the paper may be evaluated.Specific Comments:- Section 2: What are your assumptions on the network, and the basic properties of the map f?   The general mathematical setting lacks definition.- The assumption that  f(V) is continuous is also somewhat strong. Why do you need this?- Theorem 2 says that the net has to have at least as many hidden units as there are training points, opening the door to obvious memorization of the data. What is the practical usefulness of this theorem? Combined with the (brief) experiments in Sec 5, is this saying that we’re making it easier to memorize the data?- pg. 5, line 220: The classes of nnets defined by sigmoid and rectification nonlinearities are (proper) supersets of the \sigma_{0,1} class, so concluding from Corollary 1 that the former aren’t efficiently learnable doesn’t sound entirely helpful. Isn’t it precisely the networks that *aren’t* in \sigma_{0,1} which make the other two classes interesting? What can we say about, for instance, N_{\sigma_{sig}} \setdiff N_{\sigma_{0,1}} ?- pg. 5, line 227: In what sense is N_{\sigma_{sig}, L} less “expressive” than N_{\sigma_{0,1}} ? In the sense of universal approximation networks, or the narrower sense where the former is supposed to include the latter as L\to\infty?- Theorem 6: Does the result hold for any, unbounded n ? Don’t you need an asymptotic constraint on n dependent on d ? - Sec 4.3 seems reminiscent of Oja’s algorithm (Oja flows). How can it be compared to that literature?- More discussion about the computational complexity of the “modified” GECO algorithm is needed. In particular, the trace maximization required at each iteration, seems to potentially itself require many iterations (especially when there are many data points, or the problem lacks conditioning).- Experiments Section: The second experiment (overspecification) is lacking. Does the plot show the training error (MSE)? If so, it doesn’t seem surprising that a network with more units is able to reduce the training error faster -- but perhaps this experiment is meant to confirm the expected? One could argue that it is easier for the network to memorize the data. More importantly, overspecification typically goes hand-in-hand with some form of regularization, as the goal is not to minimize train set error as quickly as possible, but to quickly reach a solution with good generalization performance. It would help if the experiment could address this key question as it’s central goal, with confirmation that the train error drops suitably fast as more of a corollary.