Constant Nullspace Strong Convexity and Fast Convergence of Proximal Methods under High-Dimensional Settings
State of the art statistical estimators for high-dimensional problems take the form of regularized, and hence non-smooth, convex programs. A key facet of these statistical estimation problems is that these are typically not strongly convex under a high-dimensional sampling regime when the Hessian matrix becomes rankdeficient. Under vanilla convexity however, proximal optimization methods attain only a sublinear rate. In this paper, we investigate a novel variant of strong convexity, which we call Constant Nullspace Strong Convexity (CNSC), where we require that the objective function be strongly convex only over a constant subspace. As we show, the CNSC condition is naturally satisfied by high-dimensional statistical estimators. We then analyze the behavior of proximal methods under this CNSC condition: we show global linear convergence of Proximal Gradient and local quadratic convergence of Proximal Newton Method, when the regularization function comprising the statistical estimator is decomposable. We corroborate our theory via numerical experiments, and show a qualitative difference in the convergence rates of the proximal algorithms when the loss function does satisfy the CNSC condition.
This paper studies constant nullspace strong convexity, strongly convex only over a constant subspace, that can result in a linear convergence rate for the proximal gradient method and quadratic convergence rate for the proximal Newton method, even if the objective function is not strongly convex on the whole space. This work is an important extension of recent convergence rate analysis on sparse and low-rank minimization problems, where the objective functions are usually l1 norm or nuclear norm and the linear operators in the constraint usually have some nullspace property. The authors also extend the analysis to general decomposable norms.However, the paper has the following issues:1. The proximal Newton method does not seem practical in reality. The subproblem (10) is usually not easily solvable, unless h(x) is a quadratic function. That all subproblems should be easily solvable is a basic requirement on a practical algorithm. Otherwise, may I simply assume that (1) is easily solvable, whatever it is? Indeed, the authors have used randomized coordinate descent to solve (10), which hardly has any guarantee on accuracy of the solution to (10). So the subsequent quadratic convergence does not make sense at all. This algorithm is also awkward as it requires an extra linear search. And the authors do not specify when to stop line search (only wrote "sufficient decrease"). So I don't think analysis on the proximal Newton method adds much contribution.2. The authors overclaim the contribution a bit. Actually, this paper only analyzes the case of h(x) being a decomposable norm, not a general convex function. So the title, abstract and introduction should all be fixed.3. There may be some errors in the proofs that need clarification from the authors:a. In (22), it is unclear why the first inequality holds.b. In the third line below (22), it is unclear why the l2-norms of a_j and \bar{a}_j are both 1. Note that in the definition decomposable norm, the dual norms of a_j and \bar{a}_j are both 1.4. In the second line below (29), "\epsilon" and "L\epsilon" should switch. In experiments, where is 'x' in the loss functions? And why the minimal objective function values are both zeros? Finally, the authors assume that f(x) is twice continuous differentiable but I don't think that the hinge loss satisfies this condition.5. In Fig. 1, I don't think that it is obvious that the upper curve corresponds to linear convergence. And the convergence is surprisingly slow -- at the order of millions of iterations! Similarly, in Fig. 2 that the upper curve is in quadratic convergence is not obvious either.