Proximal Quasi-Newton for Computationally Intensive 1 -regularized M -estimators
We consider the class of optimization problems arising from computationally intensive 1 -regularized M -estimators, where the function or gradient values are very expensive to compute. A particular instance of interest is the 1 -regularized MLE for learning Conditional Random Fields (CRFs), which are a popular class of statistical models for varied structured prediction problems such as sequence labeling, alignment, and classification with label taxonomy. 1 -regularized MLEs for CRFs are particularly expensive to optimize since computing the gradient values requires an expensive inference step. In this work, we propose the use of a carefully constructed proximal quasi-Newton algorithm for such computationally intensive M -estimation problems, where we employ an aggressive active set selection technique. In a key contribution of the paper, we show that the proximal quasi-Newton method is provably super-linearly convergent, even in the absence of strong convexity, by leveraging a restricted variant of strong convexity. In our experiments, the proposed algorithm converges considerably faster than current state-of-the-art on the problems of sequence labeling and hierarchical classification.
## SummaryThe paper introduces a proximal quasi Newton algorithm for the solution ofcomputationally intensive l1-regularized m-estimators. CRFs are an example ofsuch a setting (it is relatively slow to compute the function value and/orgradient of an instance). The main contribution is a superlinear convergenceproof of the algorithm under the assumption of strong convexity in a subspace ofthe original optimization.## Specific Points- it's Lipschitz, not Lipchitz!- 134: one-dimensional problem- 183: of the inner problem- 201: include a pointer to the appendix in the caption?- 241: q-superlinear without q appearing in the formula...- 249: state assumptions onn \ell(w) again?- 259: CRF problems- 321: one-dimensional- 330: only in very few blocks- 354: I'd be instered to see how performance is for different \lambda and  possibly also different feature functions (say non-quadratic unary features)  to get an understanding of whether the algorithm improves performance across  the board.- 374: why no cross-validation? Or at least show results for several \lambda- 408: how do you choose \lambda=2 and \eta_0=0.1? Does performance change if  you alter \lambda?- 427: I feel it is rather unfair to include the time for the whole function  evaluation in SGD, can't you just *not* include the time required to compute the  objective for plotting purposes? Or show plots with epochs, rather than time  on the x-axis.- 429: rephrase second part of sentence- [430: short discussion/summary is always appreciated]## QualityThe experiments should be improved (fix SGD and include some other settings of\lambda). Also, it is not obvious to me whether the convergence rate still holdfor the actual implementation of Prox-QN with the inexact inner problem solution /shrinking strategy.## ClarityThe paper is well written and clear in most parts. I would appreciate some moredetails in the main paper about the shrinking strategy. It is also not clear tome how the shrinking strategy comes into play with the convergence proof?## OriginalityAs far as I can say the work is novel. Overlap with [1] might be something toworry about, because there might the CNSC part is probably present in bothworks.## SignificanceThe experiments seem to support the superlinear convergence, which is quiteimpressive (a plot with test error might though still be interesting). Efficienttraining of l1-regularized CRFs is an important problem and such substantialspeedups make the work significant.