Adaptive Newton Method for Empirical Risk Minimization to Statistical Accuracy
We consider empirical risk minimization for large-scale datasets. We introduce Ada Newton as an adaptive algorithm that uses Newton's method with adaptive sample sizes. The main idea of Ada Newton is to increase the size of the training set by a factor larger than one in a way that the minimization variable for the current training set is in the local neighborhood of the optimal argument of the next training set. This allows to exploit the quadratic convergence property of Newton's method and reach the statistical accuracy of each training set with only one iteration of Newton's method. We show theoretically that we can iteratively increase the sample size while applying single Newton iterations without line search and staying within the statistical accuracy of the regularized empirical risk. In particular, we can double the size of the training set in each iteration when the number of samples is sufficiently large. Numerical experiments on various datasets confirm the possibility of increasing the sample size by factor 2 at each iteration which implies that Ada Newton achieves the statistical accuracy of the full training set with about two passes over the dataset. 
(i) Global convergence of Newton's method requires implementation of a line search subroutine and line searches in ERM require a complete pass over the dataset. (ii) The quadratic convergence advantage of Newton's method manifests close to the optimal solution but there is no point in solving ERM problems beyond their statistical accuracy. (iii) Newton's method works for strongly convex functions but loss functions are not strongly convex for many ERM problems of practical importance. (iv) Newton's method requires inversion of Hessians which is costly in large dimensional ERM.
Because stochastic Newton-like methods can't use line searches [cf. (i)], must work on problems that may be not strongly convex [cf. (iii)], and never operate very close to the optimal solution [cf (ii)], they never experience quadratic convergence. They do improve convergence constants and, if efforts are taken to mitigate the cost of inverting Hessians [cf. (iv)] as in [21,12,7,18] they result in faster convergence. But since they still converge at linear rates they do not enjoy the foremost benefits of Newton's method.
In this paper we attempt to circumvent (i)-(iv) with the Ada Newton algorithm that combines the use of Newton iterations with adaptive sample sizes [5]. Say the total number of available samples is N , consider subsets of n  N samples, and suppose the statistical accuracy of the ERM associated with n samples is V n (Section 2). In Ada Newton we add a quadratic regularization term of order V n to the empirical risk -so that the regularized risk also has statistical accuracy V n -and assume that for a certain initial sample size m 0 , the problem has been solved to its statistical accuracy V m0 . The sample size is then increased by a factor ↵ > 1 to n = ↵m 0 . We proceed to perform a single Newton iteration with unit stepsize and prove that the result of this update solves this extended ERM problem to its statistical accuracy (Section 3). This permits a second increase of the sample size by a factor ↵ and a second Newton iteration that is likewise guaranteed to solve the problem to its statistical accuracy. Overall, this permits minimizing the empirical risk in ↵/(↵ 1) passes over the dataset and inverting log ↵ N Hessians. Our theoretical results provide a characterization of the values of ↵ that are admissible with respect to different problem parameters (Theorem 1). In particular, we show that asymptotically on the number of samples n and with proper parameter selection we can set ↵ = 2 (Proposition 2). In such case we can optimize to within statistical accuracy in about 2 passes over the dataset and after inversion of about 3.32 log 10 N Hessians. Our numerical experiments verify that ↵ = 2 is a valid factor for increasing the size of the training set at each iteration while performing a single Newton iteration for each value of the sample size.
The authors study the ERM (Empirical Risk Minimization) problem and propose an Adaptive Newton algorithm for solving it to statistical accuracy. The idea is to consider a sequence of nested ERM problems of increasing sizes (each new ERM problem is described by all the examples of the previous ERM problem in the sequence and a number of fresh samples) up to a certain fixed size, and take a carefully designed Newton step (of unit stepsize) related to each ERM problem. The claim is that if this is done “right”, each ERM problem is solved up to statistical accuracy.
The paper contains a novel and interesting idea, and is well written. I think it should be accepted.    This is one of a very few papers which attempt to combine optimization and statistical considerations. Most papers suggesting algorithms for solving the ERM problem simply focus on solving the deterministic ERM problem, and ignore the fact that ERM objective is an approximation to the true loss which arises as an expectation of the loss over an unknown sample distribution; and hence is subject to an approximation error. This is of course known in the literature, but it is notoriously difficult to address both the optimization aspect and the approximation aspect in a meaningful way in a single work. This papers takes a highly novel route to this problem; and I think there is a potential in this line of work.   However, the paper is not flawless – below I will enumerate some of the issues with the paper. Some of these issues can be remedied by a revision, but some seem to be an intrinsic part of the proposed approach, and hard to get rid off.  1) The introduction enumerates 4 challenges left in the implementation of Newton-like methods for ERM. I agree that these issues remain with most of the approaches. However, there are approaches which circumvent these issues – but these are not mentioned by the authors. The paper would be stronger if a more balanced treatment was made. For instance, consider the SDNA algorithm of Qu, Richtarik, Takac and Fercoq (Proceedings of The 33rd International Conference on Machine Learning, pp. 1823–1832, 2016). For L2-reglarized ERM problems, this method takes Newton-like steps in random subspaces in the dual. i) The method has a global linear rate, better than SDCA, without any explicit line-search procedure.  ii) Since there is no point in looking at local convergence beyond statistical accuracy, the analysis of SDNA focuses on speedup from mini-batching, which is more important. This speedup is superlinear in the  mini-batch size, unlike all other stochastic first-order methods. iii) SDNA was analysed for strongly convex ERM problems – so this issue is present. iv) This issue is circumvented by inverting sufficiently small random principal submatrices of the Hessian in the dual (the Hessian does not change in the dual, which is the advantage of using the dual formulation). This is just one example of how *some* of these issues *are* addressed by existing research. Hence, the paper should be more precise in stating the current status of these challenges.  2) I do not see how the claim “Our theoretical and numerical analyses indicate that we  can make alpha = 2. In that case, we can can optimize to within statistical accuracy in about 2 passes over  the dataset and after inversion of about 3.32 log10 N Hessians. “ is supported in theory.  Also, it is not clear enough what exactly the above statement means. It is made in the abstract, on lines 49-51 and lines 143-145. The precise complexity depends on values values of c and alpha, which in turn depend on the values of V_n, which are unknown in practice. The sizes of the constants in the O(1/n), O(1/sqrt{n}) decay rates will matter, and these will influence the precise constant elsewhere, including the constant 2 claimed above. So, I think this conclusion is exaggerated and needs moderation.  3) The same applies to the claim that the result is supported by experiments. Conclusions can’t be drawn from a single experiment using a single dataset (The sentence “We have observed similar performances for other datasets such as A9A,  COVTYPE, and SUSY.  ” is much less satisfying than including the experiments). Another reason to moderate the statement and remove it from the abstract in order for the readers not to be potentially misled. 4) The authors should make a detailed complexity analysis of the method, including the cost of each iteration, and compare this to existing results. A the moment, no such comparison is given, and hence it is not straightforward how the results should be interpreted and why they should be appreciated. Are your complexity results better than those of existing methods, such as SAGA or SDCA or Quartz? 5) The algorithm seems to be conceptual in practice, and impractical (this is admitted on lines 126-127). This is because the method relies on the knowledge of parameters such as V_n for all n, but these are not known. Fine tuning these might be computationally difficult, and I would expect the actual solver to be slower than state of the art (SDCA, SAGA, Quartz, SAG, AdaSDCA, dfSDCA, SVRG, S2GD, Finito, …)  on most datasets for this reason. There are more parameters which need to be tuned, such as alpha_0, beta, c. How would these be chosen in practice? Will the method be robust to this choice of parameters across different datasets? How will it compare to off-the shelf implementation of SDCA or S2GD, say, which do not require any parameters to be tuned? If the authors believe the method to be practical, which would enhance the paper considerably, then considerations of this type need to be included in the paper, and extensive computational experiments performed.  6) Theorem 1 (appearing in Section 3) relies on assumptions which are not stated until Section 4. The reader is not alerted to this. Likewise, Proposition 2 relies on these assumptions, but this is not explicitly stated. Moreover, the term M appears in the result there, unexpectedly, as this is the first time this term appears in the paper. I later found tat it was defined in a later section. Some adjustment is needed to the text in order to rectify the flow of ideas a bit and make the paper a bit easier to read in these places (the paper is very well written already; this is a minor issue which can be easily addressed in a number of ways).    Small issues:  7) Line 50: cases -> case 8) Should not c be a parameter of Alg 1? 9) In several paces (e.g., lines 171, 189, …), you write “Assumption 1-3” instead of “Assumptions 1-3” 10) Line 178: that -> in which (might be best to break this sentence up into 2-3 shorter sentences) 11) Line 181: necessary -> necessarily 12) Line 264: The -> the    Bibliography: There are many issues with the bibliography. Here are some I found:  1) [1]: capitalize journal name 2) [3]: capitalize Newton 3) [4]: include NIPS volume 4) [6]: appeared already in Proceedings of The 33rd International Conference on Machine Learning, pp. 1463–1471, 2016 5) [7]: Saga -> SAGA; also include NIPS volume 6) [10]: Bfgs -> BFGS; appeared: Proceedings of The 33rd International Conference on Machine Learning, 1869-1878, 2016  7) [11]: newton -> Newton 8) [12]: include NIPS volume 9) [13]: l-bfgs -> L-BFGS; appeared: Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, pp. 249–258, 2016 10) [14] publisher?; volume I, not volume i 11) [15]: was published already (long time ago) 12) [18]: include NIPS volume 13) [19]: newton -> Newton 14) [23]: capitalize book title  --- post rebuttal comments ---  Thanks for the explanations; some of my concerns were addressed, but many were not (or not in the way I hoped). I still think highly of the paper (and think it should be accepted), but to be fair, I also believe the score I gave for clarity is higher than it really should be. I am decreasing the clarity score from 4 to 3.