Stochastic Gradient Descent, Weighted Sampling, and the Randomized Kaczmarz algorithm
We improve a recent guarantee of Bach and Moulines on the linear convergence of SGD for smooth and strongly convex objectives, reducing a quadratic dependence on the strong convexity to a linear dependence. Furthermore, we show how reweighting the sampling distribution (i.e. importance sampling) is necessary in order to further improve convergence, and obtain a linear dependence on average smoothness, dominating previous results, and more broadly discus how importance sampling for SGD can improve convergence also in other scenarios. Our results are based on a connection between SGD and the randomized Kaczmarz algorithm, which allows us to transfer ideas between the separate bodies of literature studying each of the two methods.
This paper considers stochastic gradient descent, for the problem of minimizing F(x) = \E[f_i(x)]. The paper considers several variants of the problem, depending on whether the functions f_i are smooth, strongly convex, Lipschitz continuous gradient, or just Lipschitz continuous. We discuss these individually more below.In the introduction, the main point of the paper seems to be the improvement of the rate for linear convergence. Indeed, in the special setting where the optimizer of F is also the optimizer of each individual setting (the authors call this the “realizable setting” which is not very applicable in most problems in machine learning, save for a set of linear equations that are all satisfiable) there is linear convergence. But the paper is nice and has a consistent theme even for less exciting convergence regimes. The authors might consider outlining this more in the introduction, since before I got to the second half of the paper, my impression was that most of the results concerned this very limited regime.In the first short part of the paper, the authors show that by using a co-coercivity lemma (that gives a bound on the squared difference of the gradient, based on the Lipschitz constant) the dependence in the optimality guarantees in Bach-Moulines on the square of the Lipschitz constants, can be reduced to dependence on the supremum of the Lipschitz constant. While this is a simple result to obtain, using a known result (the co-coercivity lemma) it is a nice observation, and is the starting point for the rest of the paper. In particular, it shows that if the supremum of the Lipschitz constants is not much different than their average, then the dependence on the average Lipschitz constant is reduced from the square appearing in Bach-Moulines, to linear dependence. The supremum, of course, could be significantly bigger than the average. But the main point of this result is that the authors then show that (a) the dependence is real and not a shortcoming of the analysis (it cannot be improved), and that (b) this dependence on sup L_i indicates a natural algorithmic manner of correcting/improving it: importance sampling. Accordingly, the rest of the paper is about this modification: rather than sample the gradients uniformly, one can sample them according to a different distribution. The authors consider several settings here, including extensions to the non-smooth case, non-strongly convex case, and also the most common For the smooth and strongly convex case, in the realizable setting, the authors show that the optimal choice is to twist with a distribution proportional to the ratio of Lipschitz constant of the individual function, to the average Lipschitz constant, which is essentially what one would expect from the failure to avoid dependency on the supremum value in the ordinary SGD setting. When, as in any regression setting with noise, the problem is not realizable, this choice comes at a cost. The authors then show that one gets the best of both worlds by doing an averaged sampling, weighting by a mixture of the distribution mentioned above, and the uniform distribution. Using this scheme, the modified SGD algorithm dominates ordinary SGD (and the analysis in Bach-Moulines). Using essentially similar analysis, the authors also consider the non-smooth case, the non-strongly convex case, and the non-realizable case, in each showing that the modified “importance sampling” SGD gives some improvements. Perhaps the only catch here is that doing this modified sampling may come at a cost (at the very least a cost of sample complexity) depending on how we have access to the samples we draw. There is a short discussion here. Is there something more to say about the additional cost?Overall, this is a nice paper. The initial focus on the realizable case doesn’t seem to be too compelling. I don’t know of too many cases when we have overdetermined systems where every single equality can be simultaneously satisfied. But there is plenty of interesting material in the rest of the paper.The paper is generally well written. There are a few very minor typos I found:46: select —> selects91: seems like it should be epsilon_0/epsilon instead of the other way around.119: we the —> we use the226: get a —> we get a264: a reduction of in the number ofpp 7 2nd paragraph: then —> than330: weighing —> weighting383: rewighting —> reweighting