Sub-sampled Newton Methods with Non-uniform Sampling
We consider the problem of finding the minimizer of a convex function F :
where a low-rank factorization of ∇ 2 f i (w) is readily available. We consider the regime where n d. We propose randomized Newton-type algorithms that exploit non-uniform sub-sampling of
, as well as inexact updates, as means to reduce the computational complexity, and are applicable to a wide range of problems in machine learning. Two non-uniform sampling distributions based on block norm squares and block partial leverage scores are considered. Under certain assumptions, we show that our algorithms inherit a linear-quadratic convergence rate in w and achieve a lower computational complexity compared to similar existing methods. In addition, we show that our algorithms exhibit more robustness and better dependence on problem specific quantities, such as the condition number. We empirically demonstrate that our methods are at least twice as fast as Newton's methods on several real datasets.
The focus of this paper is improving convergence times for Newton's method by approximating the Hessian during each iteration.  The proposed algorithm applies to smooth, strongly convex objectives subject to convex constraints.  An important assumption is that the Hessian is a sum of known low-rank matrices and one additional matrix.  The main idea is to approximate the Hessian by intelligently subsampling (and properly rescaling) the low rank matrices.  The Newton step is then solved approximately.  Prior works considered uniform sampling for approximating the Hessian.  This paper studies novel non-uniform sampling techniques based on the norm or standard leverage scores of the Hessian components.  In the theoretical results section, the paper builds off recent work on fast matrix approximation methods to derive sufficient asymptotic sample sizes for the algorithm to achieve a local linear convergence rate with high probability.  An important point about this result is that unlike uniform sampling, the sufficient sample size for the proposed method does not depend on the problem's total number of training examples.  The section also clearly compares the time complexities of the proposed and related algorithms.  The theory allows the different sampling schemes to be compared based on their dependence on the problem's condition number, which is interesting.  The paper only focuses on local convergence guarantees and only very briefly refers to other work regarding what to do in the initial phase of the algorithm.  In the empirical section, the paper considers logistic regression problems with l2 regularization.  The proposed Hessian sampling approaches are compared to uniform sampling, Newton's method, and LBFGS.  The experiments focus on obtaining highly accurate solutions for data sets with roughly 100 features and 100k examples.  The proposed methods seem to clearly outperform uniform sampling, while standard Newton is a closer competitor.  
There are some obvious criticisms for this paper.  First, the algorithm focuses on finding highly precise solutions to loss minimization problems with many more examples than features.  For most real world prediction tasks, it is well-known that highly precise solutions are not necessary.  Second, it is not clear from the experiments whether the algorithm significantly improves upon the conventional Newton's method.  It seems Newton's method consistently displays the same or faster local convergence speed compared to the proposed methods, and Newton's method is only slower in early stages.  This observation is not discussed, and there are not sufficient experiment details to understand why this is the case (for example whether the same subproblem solver is used for both methods, which steps of the algorithms dominate iteration times, what type of backtracking is used).  This is especially an issue, since the analysis focuses only on the local convergence phase, and it is not clearly stated what the algorithms do in the initial phase (see footnote 1 in the paper).  On a related note, the sampling size for each method is chosen to optimize convergence progress during the initial stage of the algorithm (footnote 5 in the paper), and the uniform sampling method looks competitive in this phase.  However, the algorithms are then evaluated based on their local convergence speed, calling into question the fairness of the parameter tuning.  Finally, compared to Newton's method, the proposed method has additional parameters to tune (sample size, when to use stale leverage scores), and the data set sizes are quite small.  For these reasons, I am not convinced the algorithm will be useful in practice.  As a whole, however, the paper has several appealing aspects.  Assuming that the conclusions of the empirical section are correct, I was surprised that the somewhat complicated partial leverage scores sampling scheme significantly outperforms uniform sampling.  I imagine these subsampling and matrix approximation ideas could be useful in other areas of optimization and machine learning, and this paper could serve as a template for applying this ideas.  The theory is also interesting in the sense that the different methods can easily be compared based on their dependence on the number of training examples as well as the problem's condition number.  The sections of the paper also complement each other fairly well, as the theory can be used to explain several of the empirical results.  Finally, the presentation is for the most part very clear.  Some suggestions/questions for the authors to potentially consider are: 1. Think about defining the objective as \sum_i f_i(a_i^T w) + R(w), so that the Hessian structure is more explicit.  Also, what is the benefit of defining the A_i's as matrices and not vectors (i.e. k_i >= 1 vs. k_i = 1)?  This "block" notion was consistently confusing to me, and it is not clear to me what additional objectives this allows. 2. Also consider removing the constraint from the problem formulation. The constraint does not seem to be important to the paper, and it causes some confusion when stating time complexities of the subproblem solvers. 3. Lemma 2 is presented without much discussion.  To make the theory section easier to follow, consider briefly discussing the implications of meeting conditions C1 and C2 immediately after the lemma. 4. The sample size (in)dependence on n is framed as an important point.  For me at least, it was not obvious how the sr(A) and \tau(A) quantities in the bounds related to n, if at all.  It may be helpful to make this more clear in the text. 5. Consider including empirical results (likely in an appendix) that demonstrate the trade-offs between the time required and approximation error for each way to construct the Hessian.  This would be helpful in understanding the convergence vs. time curves. 6. It seems to be that for row norm squares sampling in the logistic regression case, the norms of the feature vectors can be cached to perform the sampling in O(n) time.  If this is true, it might be worth mentioning. 7. Consider including empirical results that illustrate the sample size bound's independence on n. 8. Think about removing the last paragraph of Section 4 to add additional commentary elsewhere.