Rest-Katyusha: Exploiting the Solution's Structure via Scheduled Restart Schemes
We propose a structure-adaptive variant of a state-of-the-art stochastic variancereduced gradient algorithm Katyusha for regularized empirical risk minimization. The proposed method is able to exploit the intrinsic low-dimensional structure of the solution, such as sparsity or low rank which is enforced by a non-smooth regularization, to achieve even faster convergence rate. This provable algorithmic improvement is done by restarting the Katyusha algorithm according to restricted strong-convexity (RSC) constants. We also propose an adaptive-restart variant which is able to estimate the RSC on the fly and adjust the restart period automatically. We demonstrate the effectiveness of our approach via numerical experiments.
I have read the response. The authors have addressed my question to satisfaction. ============================================== This paper presents an optimization algorithm for regularized ERM when the solution has low-dimensional structure (in the current paper, the specific structure is sparsity). The algorithm periodically applies Katyusha to the regularized ERM, while initializing each round of Katyusha from the approximate solution of previous round. When the strong convexity parameter is not known, the authors propose a line-search type of method for estimating it. The algorithm is demonstrated to have good performance on several datasets for solving the Lasso problem.     While I could parse the results fine, I am not very familiar with the detailed techniques in this area, so my confidence is not high.  My understanding is that Proposition 3.2 characterizes how close is the regularized ERM solution from the population optimum, thus optimizing the regularized ERM well means being close to the population solution. On the other hand, if one can establish the effective RSC for the regularized ERM, the objective becomes effectively strongly convex, even if the original objective is not globally strongly convex. It appears these two results were established by [3] and [36] previously. And the authors can then apply state-of-the-art algorithms for smooth and strongly-convex objectives, with Katyusha being a particular choice.  From this perspective (if it is correct), the theoretical result presented here may not be very technically challenging to obtain. On the other hand, the method for locating the strong convexity parameter can be a practical contribution.  Another question: In fact, if the RSC parameter mu_c is known and fixed, do we gain anything from restarting Katyusha, than simply running it once?