Beyond Convexity: Stochastic Quasi-Convex Optimization
Stochastic convex optimization is a basic and well studied primitive in machine learning. It is well known that convex and Lipschitz functions can be minimized efficiently using Stochastic Gradient Descent (SGD). The Normalized Gradient Descent (NGD) algorithm, is an adaptation of Gradient Descent, which updates according to the direction of the gradients, rather than the gradients themselves. In this paper we analyze a stochastic version of NGD and prove its convergence to a global minimum for a wider class of functions: we require the functions to be quasi-convex and locally-Lipschitz. Quasi-convexity broadens the concept of unimodality to multidimensions and allows for certain types of saddle points, which are a known hurdle for first-order optimization methods such as gradient descent. Locally-Lipschitz functions are only required to be Lipschitz in a small region around the optimum. This assumption circumvents gradient explosion, which is another known hurdle for gradient descent variants. Interestingly, unlike the vanilla SGD algorithm, the stochastic normalized gradient descent algorithm provably requires a minimal minibatch size.
Summary: This paper analyzes the stochastic version of normalized version of normalized gradient descent (NGD), which is the first effort to explore the efficacy and property of stochastic normalized gradient descent (SNGD). In order to verify the benefits of NGD in training non-convex optimization problems, this paper introduces a new property, local-quasi-convexity, to prove its convergence to a global minimum. Particularly, they prove that NGD finds an \epsilon-optimal minimum for local quasi convex functions within O(1/ \epsilon^2) iterations. In addition, this paper introduces a new setup: stochastic optimization of locally-quasi convex functions, in which the gradient is estimated using a minibatch of examples. Empirically, this paper reports experimental results by training deep neural networks by comparing with the-state-of-the-arts methods, minibatch SGD and Nesterov's accelerated gradient method.Quality: This paper is technically sound without obvious mistakes. The theoretical analyses are easy to follow and achieve the expected outcome. The empirical study shows the improvement of the proposed algorithm than the existing methods. This is the first effort for the provable gradient-based algorithm for stochastic quasi-convex optimization. Nevertheless, its training model is relatively small network, that is, a single hidden layer of 100 units. This setup is very likely to be too easy to show the advantage of the algorithm, not to mention the real application of training deep neural network on large datasets. Clarity:This paper is easy to follow and well structured. Several typos do exist, however. For example, line 37: 'SGD is guaranteed find' should be 'SGD is guaranteed to find'; line 134: 'Inforamlly' should be 'Informally'; line 185: 'its is' should be 'it is'; line 250: 'and and' should be 'and', etc. The draft should be better checked if accepted.  Originality: In keeping with NIPS objective of rewarding new unexplored areas and research directions (more than incremental advancement of the state of the art), this paper studies an important problem for non-convex optimization and tries the first effort on provable results, which should be scored a high score for originality. It is of significance to move the focus from stochastic convex optimization to stochastic non-convex optimization. Significance: This work studies an important research problem and is in a good research direction since stochastic non-convex optimization has shown its importance for several years and needs to be solved with solid theoretical guarantee. They define the property, local-quasi-convexity, to extend quasi-convexity and capture the unimodal functions which are not quasi-convex, and then leads to the proof the convergence to the global optimal. However, the empirical studies shown in the paper are not comprehensive or convincing to show that the proposed algorithm could work for training deep neural network since only a single layer model is presented.Other comments:The key strength of this paper is to introduce the property, local-quasi-convexity, to extend quasi-convexity and capture the unimodal functions that are not quasi-convex. They propose solutions for locally-quasi-convex optimization with NGD and strictly-locally-quasi-convex optimization with stochastic NGD with both theoretical proofs. The main concern about this paper lies in the empirical studies. I believe that the proposed algorithm aims to solve the problem of training deep neural network finally. However, in this paper, the model for training is too simple, which is just a single layer with 100 hidden units. Therefore, whether the proposed algorithm could really work for training deep neural network in practice beating SGD is yet to be verified.