On the Local Minima of the Empirical Risk
Population risk is always of primary interest in machine learning; however, learning algorithms only have access to the empirical risk. Even for applications with nonconvex nonsmooth losses (such as modern deep networks), the population risk is generally significantly more well-behaved from an optimization point of view than the empirical risk. In particular, sampling can create many spurious local minima. We consider a general framework which aims to optimize a smooth nonconvex function F (population risk) given only access to an approximation f (empirical risk) that is pointwise close to F (i.e., F − f ∞ ≤ ν). Our objective is to find the -approximate local minima of the underlying function F while avoiding the shallow local minima-arising because of the tolerance ν-which exist only in f . We propose a simple algorithm based on stochastic gradient descent (SGD) on a smoothed version of f that is guaranteed to achieve our goal as long as ν ≤ O( 1.5 /d). We also provide an almost matching lower bound showing that our algorithm achieves optimal error tolerance ν among all algorithms making a polynomial number of queries of f . As a concrete example, we show that our results can be directly used to give sample complexities for learning a ReLU unit.
This paper considers the problem of minimizing a non-convex smooth population risk function, where one has access to a 0-th order oracle that can evaluate the empirical risk. With this goal, the authors propose an algorithm (ZPSGD) which is essentially an instance of PSGD [Ge COLT 2015] on a Gaussian smoothing of the empirical loss. Under the standard regularity conditions of the population risk (boundedness, gradient, and Hessian Lipschitzness), and additional pointwise closeness of the empirical risk to the population risk, this paper shows the following results: 1. if empirical and population risk are (eps^1.5)/d pointwise close, then ZPSGD returns an eps-approximate second order stationary point of the population risk with a number of 0-th order queries polynomial in the problem parameters. 2. the (eps^1.5)/d pointwise closeness in optimal up to polylogarithmic factors for all algorithms that make a polynomial number of 0-th order queries. 3. if we let an exponential number of 0-th order queries, there is an algorithm for finding eps-approximate second order stationary point if the empirical and population risk are eps^1.5 pointwise, and this dependence is optimal.  General Evaluation: The paper considers an important problem at the intersection of machine learning and non-convex optimization. The paper is well-written, and the contributions are clear. The mathematical derivations and the proofs seem correct (I have not checked all the proofs). My main concern is about the novelty of the paper. ZPSGD is essentially SPGD on a Gaussian smoothing of the empirical loss. The fact that unbiased estimates of the gradient of the Gaussian smoothing of a function can be obtained by function evaluations on a batch is not novel [Duchi ToIT 2015]. Convergence of PSGD when unbiased stochastic gradients have sub-Gaussian tails is also established in [Jin PC 2018]. So the main novelty of the paper is in setting up the smoothing radius such that together with the other parameters, guarantees of PSGD can be carried over to the Gaussian smoothing of the empirical loss, and that the population risk and its gradients and Hessians are close to those of the Gaussian smoothing of the empirical risk. Despite this, I still believe this paper makes important contributions that makes it a good fit for NIPS.   =============================  I have carefully read the author response. It is not clear to me why a zeroth order oracle is interesting. Authors fail to motivate it from a computational perspective, and zeroth order oracles are not typical in machine learning. More importantly, if one does have access to a first-order oracle, then the results presented here are either well known or follow as simple corollaries to the existing results. So what is the significance of the work here? Secondly, somewhat equally important is the fact that the authors analyze the ZPSGD as an inexact gradient method. The proofs are rather straightforward, and again it is not clear what are the contributions from a technical perspective (no new tools/techniques being contributed). I still think it is a good paper and agree that the lower bound results are novel.