Analysis of Krylov Subspace Solutions of Regularized Nonconvex Quadratic Problems
We provide convergence rates for Krylov subspace solutions to the trust-region and cubic-regularized (nonconvex) quadratic problems. Such solutions may be efficiently computed by the Lanczos method and have long been used in practice. We prove error bounds of the form 1/t 2 and e 4t/ p  , where  is a condition number for the problem, and t is the Krylov subspace order (number of Lanczos iterations). We also provide lower bounds showing that our analysis is sharp.
Summary The authors consider optimization problems where one tries to minimize a (possibly non-convex) quadratic function in two different settings: - when the optimization domain is a ball with fixed radius, - when a cubic regularizer is added to the quadratic function. These two settings are motivated by their importance for second-order optimization. The goal of the article is to describe the convergence rate, in these settings, of Krylov subspace solutions (the Krylov subspace solution, at order t, is the solution when the problem is restricted to a t-dimensional subspace naturally associated with the quadratic function). The authors establish two convergence rates: one that is exponential in t, with a rate that depends of the (properly defined) condition number of the problem, and another one that scales like t^(-2), with a constant that depends logarithmically on the scalar product between two specific vectors (and that scales at worst like log^2(d) when a random perturbation is added). They show that the two bounds are essentially tight.   Main remarks I am unfortunately not very familiar with Krylov subspace methods and second-order optimization, so it is difficult for me to evaluate the position of this article in the literature, and its importance to the domain. Nevertheless, as a non-expert, I can say that I sincerely enjoyed reading this article, and found it very interesting; I hope it will be published. In more detail,  - Krylov subspace methods are an extremely important family of optimization algorithms. I therefore think that it is very valuable to precisely understand which convergence guarantees they can offer in all important use cases, and to develop new tools for proving such guarantees (as well as to improve the previous tools). The motivations from trust-regions and cubic-regularized Newton methods are also important.  - The article is very well-written, especially the technical parts. The proofs rely on polynomial approximation results that are relatively classical; the technical novelty lies in how these results are used, and combined with previous literature, to obtain the two convergence rates. The classical results are presented in full detail in the appendix; with this respect, I appreciated the effort that has been put into making the article self-contained. The rest is very clearly explained in the main part of the article, with all the necessary details, but also enough discussion so that the reader can get a good intuition of the principles behind the proofs.  - The two convergence rates, and the fact that they are optimal, are essentially the same as the optimal convergence rates for solving convex problems with first-order methods in two situations : (i) when the convex function is assumed to have a Lipschitz gradient (in which case the rate is O(t^(-2))) (ii) when the convex function is strongly convex, and has a Lipschitz gradient (in which case the rate is exponential, with the same dependency in the condition number as in Equation (4) of this article). Maybe the link could be discussed and clarified in the article?   Minor remarks and typos - Line 38: I did not find the expression "the sub-optimality ... is bounded by" very clear. - Line 120: "is choose" -> "is to choose"? - Line 220: "and os it is" -> "and it is" - Section 4, first paragraph: This is a minor point but, since the gap is smaller in the case of the cubic-regularized problem, is it immediate how to deduce the trust-region result from the cubic-regularized one, including the replacement of the cubic gap by the trust-region one in the right-hand side of Equation (18)? - Line 238: tau has to be larger than 1. - Bibliography: Reference [14] contains some weird characters, and it seems to me that the link given for reference [32] is broken. - Line 402: "for all y>=0" and "for all z>=1" could be added. - Line 415: in the left-hand side of the double inequality, I think that sqrt(kappa) should be sqrt(kappa)-1. - Line 417, second line of the block: "n-1" should be "n+1". - Line 440: g must be convex. - Line 485: I think that the last lambda_max should be a lambda_min. - The equation after line 489 could be split into two lines. - Line 491: The definition of n is missing. - Line 496: "(t+1)-dimensional" -> "d-dimensional"? - Equation after line 496: The definition of rho is actually the definition of 1/rho (we want lambda_star = rho ||s_star||). - Equation after line 501: There is a Delta missing. - Equaiton after line 525, left-hand side: "q^2(0)(u^Tv)" is probably missing in the denominator. - Equation after line 554: I do not get why the first R is not a \tilde R. - Equation after line 562: I think there is a minus sign missing in the right-hand side. - Line 578: "guarantee" -> "guarantees"  Added after author feedback  Thanks to the authors for taking my minor remarks into account. I am naturally still in favor of accepting the article.