Published as a conference paper at ICLR 2018 GLOBAL OPTIMALITY CONDITIONS FOR DEEP NEURAL NETWORKS
We study the error landscape of deep linear and nonlinear neural networks with the squared error loss. Minimizing the loss of a deep linear neural network is a nonconvex problem, and despite recent progress, our understanding of this loss surface is still incomplete. For deep linear networks, we present necessary and sufficient conditions for a critical point of the risk function to be a global minimum. Surprisingly, our conditions provide an efficiently checkable test for global optimality, while such tests are typically intractable in nonconvex optimization. We further extend these results to deep nonlinear neural networks and prove similar sufficient conditions for global optimality, albeit in a more limited function space setting.
Summary:
The paper gives theoretical results regarding the existence of local minima in the objective function of deep neural networks. In particular:
- in the case of deep linear networks, they characterize whether a critical point is a global optimum or a saddle point by a simple criterion. This improves over recent work by Kawaguchi who showed that each critical point is either a global minimum or a saddle point (i.e., none is a local minimum), by relaxing some hypotheses and adding a simple criterion to know in which case we are.
- in the case of nonlinear network, they provide a sufficient condition for a solution to be a global optimum, using a function space approach.
Quality:
The quality is very good. The paper is technically correct and nontrivial. All proofs are provided and easy to follow.
Clarity:
The paper is very clear. Related work is clearly cited, and the novelty of the paper well explained. The technical proofs of the paper are in appendices, making the main text very smooth.
Originality:
The originality is weak. It extends a series of recent papers correctly cited. There is some originality in the proof which differs from recent related papers.
Significance:
The result is not completely surprising, but it is significant given the lack of theory and understanding of deep learning. Although the model is not really relevant for deep networks used in practice, the main result closes a question about characterization of critical points in simplified models if neural network, which is certainly interesting for many people.