Newton-Stein Method: A Second Order Method for GLMs via Stein's Lemma
We consider the problem of efficiently computing the maximum likelihood estimator in Generalized Linear Models (GLMs) when the number of observations is much larger than the number of coefficients (n p 1). In this regime, optimization algorithms can immensely benefit from approximate second order information. We propose an alternative way of constructing the curvature information by formulating it as an estimation problem and applying a Stein-type lemma, which allows further improvements through sub-sampling and eigenvalue thresholding. Our algorithm enjoys fast convergence rates, resembling that of second order methods, with modest per-iteration cost. We provide its convergence analysis for the case where the rows of the design matrix are i.i.d. samples with bounded support. We show that the convergence has two phases, a quadratic phase followed by a linear phase. Finally, we empirically demonstrate that our algorithm achieves the highest performance compared to various algorithms on several datasets.
The authors uses an estimation approach to estimate the scaling matrix and replaces an O(np^+p^3) per iteration cost with an O(np+p^2) cost.The new optimization method applies to generalized linear models and is extremely well crafted.  The paper is extremely clear, the quality is superb, the originality is good and the significance only limited by the experiments and the importance of generalized linear models.The experiments are a bit weak.There are only two examples -- logistic regression and ordinary least squares.These are compared to general optimization techniques, but not to any methods tailored for these.There was for example a paper by Byrd, Oztoprak and Nocedal that used sampling estimates of the Hessian for logistic regression.This is quite closely related to what's done in this paper.There are also other papers using sampling techniques to estimate the Hessian.It is a bit disconcerting that the "appendix" is 3 times longer than the paper itself.In many ways the paper is just an advertising pamphlet for the appendix, but the main body of the paper is well crafted and most readers may not wish to dive into the most horrid derivations in the appendix.Here are some detailed comments: "Distribution of a random variable belongs to an..." --> "The distribution of a random variable belongs to an..."I expected to see the dependence of t shown on the RHS in the first equation in chapter 3.As it stands it's just an unnecessary repetition of 2.3."well-approximates": this is not a word.The "event" E defined in 4.1 is not properly defined.I imagine this would be a set over \beta^{t} in which the set should be written\{\beta^{t}: |\mu_2(\beta^{t})+...| > \tau,...\} The way it is written, it could also be a set over tau, beta_*, Sigma etc.  What's the difference between ||\beta^t-\beta_*|| and E[||\beta^t-\beta_*||] in Lemma 4.1, and what is the expectation over? Given that the proof is in the lengthy appendix, an explanation/discussion would be helpful.Could you define sub-gaussian in the main body of the paper?Figure 2 is a bit busy and I can't tell which line is which as some of the colors are very close and the legend is not sufficiently good to match the line up with the legend.The bibliography has not been done in the usual NIPS style.I think the authors need to conform with the usual standard NIPS template.