Local Smoothness in Variance Reduced Optimization
We propose a family of non-uniform sampling strategies to provably speed up a class of stochastic optimization algorithms with linear convergence including Stochastic Variance Reduced Gradient (SVRG) and Stochastic Dual Coordinate Ascent (SDCA). For a large family of penalized empirical risk minimization problems, our methods exploit data dependent local smoothness of the loss functions near the optimum, while maintaining convergence guarantees. Our bounds are the first to quantify the advantage gained from local smoothness which are significant for some problems significantly better. Empirically, we provide thorough numerical results to back up our theory. Additionally we present algorithms exploiting local smoothness in more aggressive ways, which perform even better in practice.
All variance reducing stochasting algorithms use a sampling distribution over the training points. Though the first works used a uniform distribution, it has quickly been extended to distributions based on the Lipschitz constants of the gradients of each individual function. These constants, however, were computed once for the entirety of the optimization.Orthogonally, the optimal stepsize for these methods relies on the maximum of these constants. Schmidt et al proposed a line-search for the SAG algorithm which adapted this stepsize throughout the optimization.The current paper merges these two techniques in some way by adapting the sampling distribution along the optimization. Though obvious in retrospect, I do not know of any work implementing this idea and making it work.I think Algorithms 1 and 2 are of very limited interest since they apply in very contrived cases. In particular, I think the authors should make clearer than, in most cases, these algorithms will not bring anything over the classical ones (a brief mention is made at the bottom of page 5).I am however impressed by the results obtained by empirical SVRG and this enough makes it worth publicizing. It is unfortunate that these results end up in the appendix.Overall, my main comment would be to rewrite this paper as a practical implementation with promising results and expand the experimental section, while shrinking the more theoretical part which is obvious.Other comments: - The beginning of section 2 with the SGD (typo) and SVRG updates is quite confusing. Please reword it. - I did not see mentioned the cost of drawing from a non-uniform distribution. It seems to be that, to be efficient, you need to build a tree of the probabilities. Do you need to rebuild the tree everytime a local L changes?In conclusion, I'm leaning for acceptance because of the impressive results but I think it should definitely be recast as an empirical paper.