Efficient Second Order Online Learning by Sketching
We propose Sketched Online Newton (SON), an online second order learning algorithm that enjoys substantially improved regret guarantees for ill-conditioned data. SON is an enhanced version of the Online Newton Step, which, via sketching techniques enjoys a running time linear in the dimension and sketch size. We further develop sparse forms of the sketching methods (such as Oja's rule), making the computation linear in the sparsity of features. Together, the algorithm eliminates all computational obstacles in previous second order online learning approaches.
The authors strive to make second-order online learning efficient by approximating the scaling matrix A_t by a low-rank sketched version based on S_t^T S_t. They prove scale-invariant regret guarantees for this approach when the desired matrix A_t is well approximated in this way, and show that the algorithm can be implemented efficiently for sparse data.
The quadratic update time and space requirements of second-order online methods like Online Newton Step make such algorithms unsuitable for most practical problems. The present work takes a significant step in addressing this. The primary contribution of the paper are variations of Online Newton Step that remove this drawback using a sketching approximation to the scaling matrix and a clever implementation of sparse updates.  The primary theoretical contributions are the analysis of the RP and FD versions of the algorithm. For RP they show a regret bound which holds when the matrix G_T (the matrix of observed gradients) is actually low-rank. Given the structure of the loss functions assumed,  f_t(w) = \ell(< w, x_t >), gradients will always be in the direction of the examples x_t, and so I think this theorem only holds when the data is actually low-rank. But if that was the case you could always simply project the data onto a basis and run ONS on the lower dimensional space. Hence, this result feels somewhat limited. The FD result Thm 3 is thus stronger (and is the one presented in detail in the body of the paper) since it depends instead on the spectral decay of G_T^T G_T. This point should be clarified in the paper.  The authors emphasize the fact their results are scale-invariant, but this also comes at some cost. Most results in online convex optimization apply to arbitrary convex functions, possibly satisfying additional conditions like strong convexity or smoothness. This work assumes a much more restrictive class of functions where f_t(w) = \ell(< w, x_t >), essentially a generalized linear model. This appears necessary to introduce the concept of "invariant learning", but the importance of this approach in practice isn't clear to me. One can choose the fixed norm bound on the feasible set after scaling the features, and in practice one can often avoid projecting onto a bounded feasible set at all as long as the learning rate is set reasonably. Further, while the author's claim the bound in Theorem 1 is "qualitatively similar" to the standard setting, the explicit dependence of the dimension d is a significant difference in my mind.  The experimental section presents results on both synthetic and real-world datasets. The synthetic data nicely highlights the potential for improvements from this approach; only some of this is demonstrated on real-world datasets. The author's only present experiments on the Oja-SON variant, which is unfortunate since it lacks theoretical guarantees. The importance of the theory is somewhat called into question by this, since it implies the assumptions on A_t necessary for RP-SON and FD-SON to work well may not actually hold in practice (as discussed above, this seems quite likely for RP-SON). Further, in order to outperform AdaGrad (which is 10x faster and substantially easier to implement), a significant trick was needed (lines 254-259). Given that this approach was necessary to get good performance in practice, a more thorough discussion is warranted.