Algorithms and matching lower bounds for approximately-convex optimization
In recent years, a rapidly increasing number of applications in practice requires optimizing non-convex objectives, like training neural networks, learning graphical models, maximum likelihood estimation. Though simple heuristics such as gradient descent with very few modifications tend to work well, theoretical understanding is very weak. We consider possibly the most natural class of non-convex functions where one could hope to obtain provable guarantees: functions that are "approximately convex", i.e. functionsf : R d → R for which there exists a convex function f such that for all x, |f (x) − f (x)| ≤ ∆ for a fixed value ∆. We then want to minimizẽ f , i.e. output a pointx such thatf (x) ≤ min xf (x) + . It is quite natural to conjecture that for fixed , the problem gets harder for larger ∆, however, the exact dependency of and ∆ is not known. In this paper, we significantly improve the known lower bound on ∆ as a function of and an algorithm matching this lower bound for a natural class of convex bodies. More precisely, we identify a function T : R + → R + such that when ∆ = O(T ( )), we can give an algorithm that outputs a pointx such thatf (x) ≤ min xf (x) + within time poly d, 1 . On the other hand, when ∆ = Ω(T ( )), we also prove an information theoretic lower bound that any algorithm that outputs such ax must use super polynomial number of evaluations off .
Trivially, one can solve the problem by constructing a -net and search through all the net points. However, such an algorithm requires Ω 1 d evaluations off , which is highly inefficient in high dimension. In this paper, we are interested in efficient algorithms: algorithms that run in time poly d, 1 (in particular, this implies the algorithm makes poly d, 1 evaluations off ).
One extreme case of the problem is ∆ = 0, which is just standard convex optimization, where algorithms exist to solve it in polynomial time for every > 0. However, even when ∆ is any quantity > 0, none of these algorithms extend without modification. (Indeed, we are not imposing any structure onf − f like stochasticity.) Of course, when ∆ = +∞, the problem includes any non-convex optimization, where we cannot hope for an efficient solution for any finite . Therefore, the crucial quantity to study is the optimal tradeoff of and ∆: For which , ∆ the problem can be solved in polynomial time, and for which it can not.
In this paper, we study the rate of ∆ as a function of : We identify a function T : R + → R + such that when ∆ = O(T ( )), we can give an algorithm that outputs a pointx such thatf (x) ≤ min xf (x) + within time poly d, 1 over a natural class of well-conditioned convex bodies. On the other hand, when ∆ =Ω(T ( )) 1 , we also prove an information theoretic lower bound that any algorithm outputs suchx must use super polynomial number of evaluations off . Our result can be summarized as the following two theorems:
Theorem (Algorithmic upper bound, informal). There exists an algorithm A that for any functionf over a well-conditioned convex set in R d of diameter 1 which is ∆ close to an 1-Lipschitz convex function 2 f , and
A finds a pointx such thatf (x) ≤ min xf (x) + within time poly d,
This paper studies the problem of minimizing approximately convex functions by zero order methods. This problem adds one extra layer of difficulty to the traditional exact minimization methods, given by the (uniform) distance ∆ of the objective to a convex 1-Lipschitz function. The main contribution is providing tight upper and lower bounds on the complexity of this problem for all regimes of inexactness ∆, accuracy ε, and dimension d. To the best of my knowledge, all lower bounds (except for Case 3, which is somewhat simpler) are new, and the same is true for the algorithm proposed for the regime ε/d ≼ ∆ ≼ ε^2/\sqrt{d}. This paper considerably extends the understanding of approximate convex optimization by some novel ideas: • The lower bound consists on hiding a linear function on a random slab. The improvement achieved in the paper relies on constructing a convex function outside the slab, which is key on the new results (previous extensions were only linear). • The upper bound is based on stochastic smoothing and gradient methods. By the divergence theorem, one can obtain an approximate gradient by estimating finite differences on random unit vectors, and this approximation is robust to the inexactness introduced by ∆. Although this idea is not entirely new (e.g., the cited work of Shamir), the analysis itself seems to be original, and it is studied in the inexact setup, which is certainly new. Overall, I believe this paper is very interesting, and therefore I strongly recommend its publication at NIPS. However, I have to mention some very mild criticism, particularly on the related literature and the writing style.  
First I would like to justify my ratings for this paper. I believe the quality of the paper is indeed high: it resolves open questions from the literature, it brings new ideas in the construction of hard instances for convex programming, and can hopefully impact related areas. However, there are some aspects that deserve more attention, particularly I think writing requires some extra effort.   I would like to describe some high-level criticism for the work:  1. The paper emphasizes the differences between approximately convex minimization and various oracle models; however, it is not mentioned that approximately convex minimization is equivalent to minimization of convex functions with an inexact zero order method. This is somehow a more general setup (as the oracle does not need to be consistent with information among iterations) and it is arguably simpler to grasp.  3. The paper needs some writing revision. Aside from correcting typos, it would be nice to give some insights on the intriguing new regime ∆ = O( ε^2 / sqrt{d}), and perhaps expand the motivating discussions on proof techniques (Section 3.1) and the informal discussion on the lower bound, at the beginning of Section 4.  Below are some minor comments:  1. The Abstract and first lines of the Introduction stress the relevance of optimization with nonconvex objectives, mentioning applications such as neural networks, graphical models, clustering, etc. Algorithms for minimizing approximately convex functions are not applicable to these problems, so it seems inappropriate to mention these applications.  2. In the paper it should be mentioned the very first reference on inexact zero order methods from the Nemirovsky and Yudin monograph (Exercise 1 in page 360). This example provides the same superpolynomial lower bound when ∆ = Ω(ε/ sqrt{d}) (in line 77) for the unit Euclidean ball (instead of the hypercube), and it fits precisely within the model of the paper.  3. Line 145. The definition of the lower convex envelope is somewhat confusing. First, some form of differentiability is required so that the formula makes sense. Second, the remark: “When the input function F is convex, LCE(F) can be considered an extension of F to the entire R^d” is used incorrectly, as it is later used for a function defined on a nonconvex set (first line of proof of Lemma 4.2). I suggest to prove “by hand” that LCE and the function coincide in the proof, as it is easy and does not lead to ambiguities.  4. Line 282. “Thus, for every point \tilde u_1, there exists[..]”. Please add more details on this, as it is an important technical step in the proof.